grimace face

I might be missing context here, but this kind of problem speaks more to a company’s inability to create useful observability, or worse, their lack of conviction around solving noisy alerts (which upon investigation might not even be “just” noise)! Your product is welcome and we can certainly use more competition in this space, but this aspect of it is basically enabling bad cultural practices and I wouldn’t highlight it as a main selling point.

Anyway in lot of places management see IT as a necessary evil. Like a service center similar to HR but less popular because management genuinely don’t understand it and most IT departments lack HRs political shrewdness and communication abilities. At the same time it’s not uncommon for users to be unable to tell support if they’re on an Android or iOS device (yes, I’m serious). Sometimes employees won’t even differentiate between their professional and personal IT issues on their work devices. Which means that sometimes they’ll raise hell over things that might not warrant full alert systems for on-site support.

What might be challenging here is that you’ll still need someone to actually use the authors tool correctly. Though that is probably going to be a lot easier than making any sort of change management to was organisations relationship with IT.

In similar mindset, I've seen attempts to "fix" flaky test suites by retrying failing tests 5 times until they pass. What happens: You just set a new baseline of shit allowed. This allows even more noise to enter the system and you have to rerun them even more or need increasingly more advanced tools to filter out the noise.

Once the new baseline is anchored, you become dependent on the filter. Now every tool that interacts with the metrics need to be aware of the additional filter, there may be more than just the slack messages. Should your dashboard show the raw or the filtered metrics?

Devils advocate: consider an alert with 99/100 false positives. The LLM may be good at classifying it as noisy but will it do a better job than a human to react to the 1 true positive? Maybe, but at the same time it allows more such noise to accumulate in the system, in effect a net negative. It's better to remove such an alert instead. Even if the numbers were turned around in favor, that's a lot of complexity added.

The additional context this product provides may of course still be useful and i applaud the effort. This product space does have a lot of potential for growth and is a real pain for operators. Be careful with using it as a substitute for proper alert hygiene and culture.

You are right, but there's ideal state and there's the real world. When on call most of my time is spent trying to make on call better. Reducing noise, providing more context when the alert and logs are lacking, and of course fixing the real issues that alerts have identified. That said there is a period of time in between receiving non-actionable alerts and classifying them as such, and more context without using brain power is always welcome. I think I'll give it a shot.

My company (like so many) is struggling a bit with culture around noisy alarm. Not only is noise tolerated, but when some closes an alarm because it's "known to be noise" and I prod them, it turns out that there is a very real impact on the user, it's just that nobody bothered to look into it. The alarm rings, the on-call hopes that it closes itself soon enough, it does so they consider it "false positive and noisy" even though there was impact on the user during these few minutes.

The only way to fight that is a zero-tolerance culture on alarms, which means no false positive is ever tolerated: fix it.

If we completely shut off the alarm, one time it's actually an issue, I'll get dinged for shutting down monitoring.

So yea, I mark it's priority as low so it doesn't wake me up in PagerDuty and move on.

However, installing a tool to specifically reign in those garbage noisy alerts is a potentially easy, significant win for the time and mental health of on-call engineers.

I mean, it sounds like you can then afterwards go in and identify the alerts that are just noise, and having that data means you can take action. Maybe contact the teams that are writing the noisiest alerts, or prepare some data-driven engineering standards for the company, whatever. But that still falls into "fix the culture", which is famously hard to do by fiat.

Iterating on your alarms is super informative about the underlying product. It'll point to how you might improve your KPI measurements, or find bugs you didn't know were there.

Consider a not-that-hypothetical example: "host computer is unreachable" alerts that page oncall when they arrive for members of a fleet of critical database servers or replicas.

The alerts have proven their usefulness (they tend to arrive several minutes before application-level error spikes when a database is e.g. so overloaded the monitoring agent can't function or a replica is gone so changelogs are overflowing) ... when they're genuine. However, they're mostly not genuine: alerting agents crash and automatic-restart-service init scripts bug out or give up; per-database-owner customizations in hosts' available file descriptor numbers are propagated incorrectly to non-database services and prevent the alerting agent from running, databases that serve infrequent-but-critical on-demand reporting loads are subjected to tens-of-minutes-long load spikes during which the host is doing what it's supposed to but so pegged that the alerting agent won't work, and so on.

What do you do with those alerts?

"Just fix the problems causing the false positives!" Fine, but that takes a lot of time and coordinated effort, even if the oncall folks are empowered to prioritize the work getting done (which is far from a given at many companies, for reasons both good and bad): auto-restart-agent scripts can be replaced with better scripts (time, effort, debugging of some hokey bash that needs to run on a wide variety of environments) or systemd (time, effort, maintenance windows, and approval/retraining to update ancient linux distributions running critical databases). File descriptor/per-database tunings can be unified and continually audited for/invalidated before configs are pushed (developer effort, coordination with teams writing configs). Reporting databases can be upsized (money, maitenance windows) or the database processes can be moved into a cgroup to leave some resources to spare (effort, distro upgrades, maintenance windows).

That's going to take awhile, if it ever happens to completion.

Meanwhile, this "host unreachable" alert is useless 90% of the time and very useful (as in: it can be leveraged to prevent downtime for customers entirely) the remaining 10% of the time.

Like, sure, some of those issues are stupid. But none are hypothetical, all are younger than 5y, and I bet this kind of struggle is common and representative even at companies who are invested in operations and operations staff.

That's not an "inability to create useful observability", that's a genuinely hard problem resulting in noisy, spurious alerts that, depending on the rate-of-change/regulatory space of the company, might persist for months or years. What's more, alert management is an ongoing process. Even if one family of noisy alerts is addressed, another one will emerge as new behaviors and technologies are adopted.

I guess this is all to say that I don't think tools like Opslane (which I have not used) are "enabling bad cultural practices". Organizations that don't give a shit about operations will continue to suck at operations no matter what tools they use. But products Opslane are valuable even (especially?) in capable, operations-focused organizations as well.

This way there is data to make a case that certain alerts are noisy (for various reasons) and we should strive to reduce the time spent dealing with these alerts. Fixing some of them might be as easy as deleting them but for others might need dedicated time working on them.

The unintended consequence of forcing alerts down ops' throat is them gradually caring less about pages, because there's a very good chance that each one is unactionable. I've worked places that do this, I've seen it happen first-hand more than once.

It starts with frustration and ops being less helpful to devs, and ends in a jaded acceptance where ops people start telling each other "just close it and see if it happens a second or third time, that alert never means anything". At that point, the system may as well not emit the errors anyways because no one is looking at the alerts anyways.

That's true, but if the error says "PANIC! EVERYTHING IS DOWN" when it's not true, then it's asking for an action that's outsized to the problem. Error messages are fine, but they just need to be classified and responded to correctly, and noisy alerts are typically the ones that are misclassified and demanding attention they (probably) don't deserve.

If the error is not-actionable, why wake someone up in the middle of the night because of it?

I don't think anyone is rejecting the observability of these errors, but just that there's no point in having it alert/wake someone up unnecessarily.

At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, "Was this alert useful?". Then we would send the alert owners reports about what percent of people found their alert useful.

It really let us narrow in on which alerts were most useful so that others could subscribe the them, and which were noise, so they could be tuned or shut off.

That one button alone made a huge difference in people's happiness with being on call.

I spent about 3 months, each day trying to triage into buckets based on activity and dealing with whatever was causing the most alerts each day. Some came down to just tamping out classes of 4xx errors that should never have been in the email/alert system to begin with. Others came down to indexes to reduce load/locking/contention on some db tables. Others still were much harder to dig into.

Will say at the end of the 3 months, there was only a trickle of emails a day and the notifications were taken much more seriously after not being so overwhelming as to being ignored altogether.

edit: This was just the first thing I did each day was deal with one problem, then moving to new feature work... It wasn't assigned, as the company would always prioritize new feature work, it was just something I did for my own sanity.

We use that to generate a report so that teams have visibility into which alerts are causing the most amount of noise.

Many of these lend themselves well to dashboards instead of alerts, but not everything is "dashboardable". Sometimes it's good to have a set of low-priority alerts that are treated differently than others.

E.g. "we're not receiving any data/requests". Sometimes that's just a lull in activity. Maybe a holiday. Sometimes it's because everything _else_ is broken and nothing is getting in (e.g. DNS issues).

With that said, I do think that classification should be made manually and not automatically.

1: https://5x9s.svix.com/p/evolution-of-incident-management

My rationale for flagging the alert was to help prioritization for the on-call (lets say there are multiple alerts going off at the same time)

For very specific values of "people problem", "fix", and "tech". In reality, a more true (and relevant) assertion is "appropriate tools can make virtually any problem more tractable."

For example, it takes an annoyed engineer to notice that the same flaky alert keeps going off and is noise. Then it takes non-trivial skill on their part to communicate the need to disable that alert. They will meet non-trivial resistance, because disabling alerts is dangerous. However, if the tools they are using say "This is a noisy alert, it hasn't been useful for 6 months," disabling that alert becomes more of a best practice for the organization.

Writing it out makes me laugh because that's like something from Douglas Adams stories. Automated Ignoring System along with Infinite Improbability Drive.

We deployed a new system and had one week on call for each of five team members. The first couple of rotations were hell. Almost every night ended up with at least one wake up call. As we learned how to solve each type of outage, we then taught the first-line staff how to reboot the right components so we didn’t get as many wake-ups, while we spent our days fixing the bugs. And eventually the system stopped crashing.

The on-call pay was really good (nearly double for that week) and it was a pretty sweet reward to be able to rake that in as calls stopped coming. We broke out a bottle of champagne when the first week of no calls had passed.

Eventually on-call was cancelled.

Imagine how this story would have ended if management had incentivized us differently, for example if you only got the extra pay for the nights where you got pages.

You should be building resilient architectures, not being on firewatch duty.

Developers don't want to ever be paged because they don't want to be bothered, but the business might be perfectly happy to pay you to be on firewatch duty.

Consider a "low traffic" alert, how can you tell the difference between a slow period at 3am on a holiday vs a true outage? You can't without someone getting up and testing if the site is still up. (Maybe you can automate that check but there's always edge-cases you can't automate).

OP seemed to suggest it's better to disable the alarm than to just suffer the false alarm every now and then. I doubt very much that the people paying you for the on-call service would agree though.

This is a very reductive statement.

Developers have experienced their best colleagues burning out and leaving jobs because of on-call being completely overwhelming.

Developers want to behave intelligently.

Developers want the system to work.

Developers don’t want to burn their lifespan for false alarms that are being sent because someone didn’t spend 30 seconds thinking about whether a human being needs to be woken up in the middle of the night for whatever widget they’re slapping together.

Is that not also reductive then? Or maybe my statement pretty accurately captures that sentiment without 4 sentences of explanation.

But no, instead of engaging with the meat of my argument you just reductively attack one sentence.

I get it, I'm oncall right now for my job. I don't like it when alarms go off. I also understand that if I were to tune the alarms so I "NEVER get called" I'd be out of a job soon enough because the business would go under.

How would your interpretation change if the article said this instead?

> The goal for oncall should be to continuously tune the system toward having no outages and no false alarms.

FWIW, I did only attack one sentence. This was not exactly intended to be dismissive. It was my reaction to, in my eyes, the weakest part of your argument.

Are they paying you to answer false alarms, or are they paying you to make sure the site is available and performant to keep customers happy? Nobody with half a brain wants to answer a bunch of false alarms. Are there are people that will happily get paid to ACK yet another noisy alarm just to collect a paycheck? Certainly; but these are button pushers, not problem solvers.

Your low traffic alert scenario simply requires synthetic requests. This is you you test anything with low usage, but requires high reliability.

Granted, neural networks were not generally applicable to this problem at the time, but this whole idea seems like the same problem being solved again.

Telecoms and IT used to supervise their networks using Alarms, in either a Network Management System (NMS) or something more ad-hoc like Nagios. There, you got structured alarms over a network, like SNMP traps, that got stored as records in a database. It’s fairly easy to program filters using simple counting or more complex heuristics against a database.

Now, for some reason, alerting has shifted to Slack. Naturally since the data is now unstructured text, the solution involves an LLM! You build complexity into the filtering solution because you have an alarm infrastructure that’s too simple.

If it were an alarm database, an operator would be able to 1. Acknowledge the alarm 2. Manually clear an alarm that was issued in error.

Without those mechanisms, alarm handling becomes really difficult for an ops team, because now all you have is either a string of emails or a chat log.

We're building a tool in the same space but opted out of using LLMs. We've received a lot of positive feedback from our users who explicitly didn't want critical alerts to be dependent on a possibly opaque LLM. While I understand that some teams might choose to go this route, I agree with some commentators here that AI can help with symptoms but doesn't address the root cause, which is often poor observability and processes.

In your particular organization. Slack is one of many instant messaging platforms. Tightly coupling your tool to Slack instead of making it platform agnostic immediately restricts where it can be used.

Other comment threads are already discussing the broader issues with using IM for this job, so I won't go into it here.

Regardless, well done for making something.

However, the design is pretty flexible and we don't want to tie ourselves to a single platform either.

Just fix the original problem, don't layer an LLM into it.

We wanted to provide that awareness because a lot of teams arent fully aware how bad the problem might be (on-calls change weekly, there might be a bunch of other issues)

More importantly, and not mentioned by StackOverflow, those devs are among the 85% of businesses using M365, meaning they have "Sign in with Microsoft" and are on teams that will pay. The rest have Google and/or Github.

This means despite being a high value hacking target (accounts and passwords of people who operate infrastructure, like the person owned from Snowflake last quarter) you don't have to store passwords therefore can't end up on Have I Been Pwned.

It suddenly becomes a much higher priority to get alerting in order.

I’ve never seen an issue that’s conclusively a false alarm without investigating at all. Just delete the alarm? An LLM will never find something like another team is accidentally stress testing my service but it does happen

Another perfect example is when the queen died and it looked like an outage for UK users. Can your LLM read the news? ChatGPT doesn’t even know if she’s alive

I expect you will need AGI before large companies will trust your product.

- We have a weekday (2 shifts) / weekend (1 slightly longer shift including friday morning to allow people to take long weekends) oncall rotation as well as a group-combined oncall schedule which gets finnicky.

- When people join or leave the rotation, making sure nothing shifts before a certain date or swapping one person with another without changing the rest and other things are a massive pain in the butt

- Combine this with a company holiday list - usually there's different policies and expectations during those. - Allow custom shift change times for people in different timezones.

- We have "oncall training" / shadowing for newbies, automate the process of substituting them in gradually, first with a shared daytime rotation and then on their own etc.

- Make oncall trades (if you can't make your shift simpler)

Gripes with PD:

- Pagerduty keeps insisting I'm "always on call" because I'm on level N of a fallback pager chain which makes their "when oncall next" box useless - just let me pick.

- Similarly, pagerduty's google calendar export will just jam in every service you're remotely related to and won't let you pick when exporting, even though it will in their UI. So I can't just have my oncall schedule in google calendar without polluting it to all hell.

We are working on a variant being used more by investigative teams than IT ops - so think IR, fraud, misinfo, etc - which has similarities but also domain differences. If of interest to someone with an operational infosec background (hunt, IR, secops) , and esp US-based, the Louie.AI team is hiring an SE + principal here.

You can tune your monitoring!

Noisy alert that tends to be a false positive but not always? Tune alert message to only send if the issue continues for more than a minute, or if the check fails 3 times in a row. Theres hundreds of ways to tweak a monitor to match your environment.

Best of all? It takes 30 seconds at most. Find the trigger, adjust slightly, and after maybe 1-2 tries, youll be getting 1 false positive sometimes, and actual alerts when they happen, compared to 99% false alerts, all the time.

Oh and did you know any monitoring solution worth its salt can execute things automatically on alerts, and then can alert you if that thing fails?

Also, Slack is not a defacto anything. Its a chat tool in a world of chat tools

Running on-call well is a culture problem. You need management to prioritize observability (you can't fix what you can't show as being broken), then you need management to build a no-broken-windows culture (feature development stops if anything is broken).

Technical tools cannot fix culture problems!

edit: management not talking to engineers, or being aware of problems and deciding not to prioritize fixing them, are both culture problems. The way you fix culture problems, as someone who is not in management, is to either turn your brain off and accept that life is imperfect (i.e. fix yourself instead of the root cause), or to find a different job (i.e. if the culture problem is so bad that it's leading to burnout). In any event, cultural problems cannot be solved with technical tools.

The problem are not the alerts — the alerts actually are catching real problems — the problem is the following:

1. The team is understaffed so sometimes spending a few days root causing an alert is not prioritized 2. When alerts are root caused sometimes the work to fix the root cause is not prioritized 3. A culture on the team which allows alerts to go untriaged due to desensitization.

Our headcount got reduced by ~40% and — surprise surprise — reliability and on-call got much worse. Senior leadership has made the decision that the cost cuts are worth the decreased reliability so nothing is going to change.

The job market is rough so people put up with this for now.

I don’t mean to imply there is some kind of failure magnitude competition, I just want to reinforce that software “engineering” already has a huge problem with abject neglect of the learnings that other sign-and-stamp engineering fields have already learned from and fixed. Us code slingers are not in uncharted territory, we just need to learn from our predecessors and peers that build literal bridges and towers and force management to treat our field in the same way.

I was lucky enough to join a company where management does this. The managers were made to do this by experienced engineers who explained to them in no uncertain terms that stuff was broken and nothing was being shipped until things stopped being broken. Unless you have good managers this won’t happen without a fight and it’s a fight I think we as engineers need to take.

Some managers in other teams played the “oh it’s not super high impact it’s not prioritized” game, and those teams now own a bunch of broken stuff and make very slow progress because their developers are tiptoeing around broken glass, and end up building even more broken stuff because nothing they own is robust. Those managers played themselves.

Communication with management is bidirectional, sometimes they need a lot of persuasion.

Sounds like managing up, i.e. doing IC workload and the manager's job. Hard pass.

I doubt very much that my experience was unique. In my new position we have the same problems with reliability but I don’t get involved in the political side of trying to argue about it, just turn up and do my 9-5. I’m a lot less stressed now!

The existing tools mentioned can show the metrics. Management needs an education - and that is part of the engineering job.

Isn’t that bizarre? In all my years as an engineer I can count the number of managers that went to learn about engineering by themselves, on one hand.

It’s literally their job, but somehow they feel they can do it without understanding it.

I think engineer brings the numbers to management to decide course.

I prefer the situation where the CTO has no MBA and worked their way up - but that is uncommon IME.

So, in many orgs, engineer puts their comms hat on an presents a solid case.

The engineer who can communicate well, and show the metrics is typically the one who can get promoted to the decision maker role. First from the bottom up, then as a great leader

That part is fine. What I do not understand is why there is so little interest in learning what makes engineering different from running a widgets factory.

“Tell me why it won’t work” is a fine question, but it’d be nice if I didn’t have to force all their education on them.

E.g. how many managers ignore that oft repeated adage that 9 women cannot have a baby in a month, and just spam more people on a project in the hope it’ll go faster.

The number of people who started pushing code with subtle bugs so they could create a ticket for it, fix their own bug, and get closer to that $100 gift card was shocking to me.

I can’t imagine the chaos that would occur if something came with a $10K bonus attached. Some people will bend over backward to get even tiny rewards. Dangling a $10K reward would get the wheels turning in their heads immediately.

In my experience, it’s not a net win. They’ve budgeted the same amount for compensation either way, so you’re probably getting lower base comp if they’re allocating some of it for on-call.

It also creates an atmosphere where on-call becomes more normalized, because you’re getting paid extra to do it. Some people, usually young single people, will try to milk the overtime for as much as they can, dragging out the hours spent doing on-call work because every extra hour spent on the problem makes their paycheck bigger.

However, one of the things that I noticed in my previous companies was that my management chain wasn't even aware that the problem was this bad.

We also wanted to add better reporting (like the alert analytics) so that people have more visibility into the state of alerts + on-call load on engineers.

What strategies have worked well for you when it comes to management prioritizing these problems?

"We're paying down our technical debt"

Isn't that a cultural problem?

That said, I like that you're 'saying out loud' with this. Slack and other similar comm tooling has always been advertised as a productivity booster due to their 'async' nature. Nobody actually believes this anymore and coupling it with the oncall notifications really closes the lid on that thing.

good to see more options in this space! especially OS. I think de-noising is a good feature given alert fatigue is one of the repeating complaints of on-callers.

> Opslane is a tool that helps (make) the on-call experience less stressful.

Meanwhile they either have a global workforce so don't actually need to have anyone on-call or only have customers in countries they have employees in.

It's bullshit.

Either companies should be up front about this when hiring or it should be optional and paid.

Or, they can use their engineering talent, just like telecoms companies have been doing for ever, to engineer their products to be more resilient and automate failure cases so proper remediation can wait until working hours.

Did you search tools that are cheaper than DD?

I guess alerts to messengers are OK as long it's only a couple manually created ones, and there should be a graphical dashboard to learn the rest of problems

We had a rule in my team (before a management change that blew it all to shit) that we don’t use email or messaging for monitoring. Everything goes into the SOR. Once it’s in the SOR, if people want emails, texts, or whatever, it let them know there is work to do, that’s up to the team. Others would make dashboards… lots of options once it’s in the system, and nothing gets lost.

For example, I went from a team that looked at tickets all day to one that mainly worked on user stories in Jira. Because no one was looking at the incidents in the SOR, things were getting missed. I wrote something to check for incident tickets assigned to our team every hour, and it would post them in our team chat so people knew there was work to do. Then once per day, it would post everything still unassigned, so if something was lost on that hourly post, it would annoy everyone once per day until it was assigned/resolved. It worked out decently well. If there was a lot of stuff, it would post a message to have someone actually login to the SOR and look at all our tickets. I would sometimes use the standup to assign stuff out and get some attention on it, if things were getting bad.

But, if alarms are configured in a clean way, ideally your team is getting some warnings and such there and then if there's an alert that needs to actually page, it sends that to PagerDuty or whatever platform you use along with another message to Slack.

What strategies have you seen work well?

I have no idea what's the best setup for small companies

We are based in Europe and have the problem that some of us sometimes just forget we're on call or are afraid that we'll miss OpsGenie notifications.

We're desparately looking for a hardware solution. I'd like something similar to the pagers of the past but at least here in Germany they don't really seem to exist anymore. Ideally I'd have a Bluetooth dongle that alerts me on configurable notifications on my phone. Carrying this dongle for the week would be a physical reminder I'm on call.

Does anyone know anything?

You can already enable silence/focus time bypass modes for apps like PagerDuty and such…

If you can’t develop some sense of responsibility to check if you’re on-call, frankly you have no business being in an on-call role.

No hardware will make you or your engineers more diligent. The only reason pagers made more sense than phones is/was because of protocol reasons NOT because it’s some separate device.

I realize it's a "me" problem and therefore I'm looking for a solution. Others in the company have the same problem. That said: This is my very own company and I have a great sense of responsibility but I also have a shit-ton of other things in my head and I'm not the only one with this issue here.

The silence etc. bypass doesn't always work (I commented in another thread).

I have no affiliation with OpsGenie outside of using at work.

And we have one user with another brand that also locks down the notification/alert settings and kills apps in attempt "to save battery" which can't be controlled.

I don't understand this. Either the issue is important and requires immediate human action -- or the issue can potentially resolve itself and should only ever send an alert if it doesn't after a set grace period.

The way you're trying to resolve this (with increasing alert volumes) is the worst approach to both of the above, and improves nothing.

I once worked for a string of businesses that would just send everything to on call unless engineers threatened to quit. Promised automated late night customer sign ups? Haven't actually invested in the website so that it can do that? Just make the on call engineer do it. Too lazy to hire off shore L1 technical support? Just send residential internet support calls to the the On Call engineer! Sell a service that doesn't work in the rain? Just send the on call guy to site every time it rains so he can reconfirm yes, the service sucks. Basic usability questions that could have been resolved during business hours? Does your contract say 24/7 support? Damn, guess thats going to On Call.

Shit even in contracting gigs where I have agreed to be "On Call" for severity 1 emergencies, small business owners will send you things like service turn ups or slow speed issues.

You don’t need that Times Square ad, only 8-10 people will look up. If you just want the footage of your conspicuous consumotion, you can easily photoshop it for decades already.

Similarly, chat causes anxiety and lack of productivity. Threaded forums like HN are better. Having a system to prevent problems and the rare emergency is better than having everyone glued to their phones 24/7. And frankly, threads keep information better localized AND give people a chance to THINK about the response and iterate before posting in a hurry. When producers of content take their time, this creates efficiencies for EVERY INTERACTION WITH that content later, and effects downstream. (eg my caps lock gaffe above, I wont go back and fix it, will jjst keesp typing 111!1!!!)

Anyway people, so now we come to today’s culture. Growing up I had people call and wish happy birthday. Then they posted it on FB. Then FB automated the wishes so you just press a button. Then people automated the thanks by pressing likes. And you can probably make a bot to automate that. What once was a thoughtful gesture has become commoditized with bots talking to bots.

Similar things occurred with resumes and job applications etc.

So I say, you want to know my feedback? Add an AI agent that replies back with basic assurances and questions to whoever “summoned you”, have the AI fill out a form, and send you that. The equivalent of front-line call center workers asking “Have you tried turning it on and off again” and “I understand it doesn’t work, but how can we replicate it.”

That repetitive stuff should he done by AI and build up an FAQ Knowledge Base for bozos and then only bother you if it came across a novel problem it hasn’t solved yet, like an emergency because, say, there’s a windows BSOD spreading and systems don’t boot up. Make the AI do triage and tell the differencd.

But I would'nt use LLM for it due to hallucinations

You don't have to be a missile defense system to require a stable system where devs respond quickly...

Any outage that costs more than paying an engineer to be on-call is worth it. It's not that complex. If an outage blocks 10000 people from doing their work, it's obviously worth it to wake someone up to try to resolve it half a day sooner. (Someone you've been paying specifically for this purpose!)

> rob people of their personal time

Being paid to be on-call is not your personal time.

https://betterstack.com/community/guides/incident-management...

Also helpful is having management that is responsive to bad on-call situations and recognizes when capable, full-time around-the-clock staffing is really needed. It seems too few well-paid tech VPs understand what a 7-Eleven management trainee does, i.e., you shouldn't rely on 1st shift workers to handle all the problems that pop up on 2nd and 3rd shift!

The only way to make on-call less stressful is to do the boring work of preparing for incidents, and the boring work of cleaning up after incidents. No magic software will do it for you.