AIOps is software that watches your systems the way an on-call engineer watches them, and does it all the time without getting tired. That is the whole idea. Your services already produce the signals: numbers like CPU and error rate, words like log lines, and alerts from every tool you own. On a bad night those signals arrive by the thousands, and a human cannot read them all. AIOps reads them instead. It learns what normal looks like for your system. When something drifts away from normal, it groups the noise into one incident, ranks it, and either fixes it or wakes a human with the story already written. Here is one concrete night. A deploy goes out at 2 AM. Error rate on the checkout service climbs from 0.1 percent to 4 percent. The old way, your phone buzzes forty times in five minutes, once per alert, and you spend the first ten minutes just working out that all forty alerts are the same problem. The AIOps way, the system sees forty alerts from one service in one five-minute window, calls it one incident, points at the deploy that just landed, and drafts a rollback. You get one page, with the cause attached. That is the job. Less noise, faster cause, and a fix that is ready before you are fully awake. This lesson follows one anomaly through the whole pipeline, from the first metric to the final decision about who acts.
Every hop charges rent in milliseconds. The bar below is a worked example for one incident travelling through the pipeline, not a measurement from a live system, but the shape is honest: the expensive hops are the thinking hops, drafting a fix and deciding whether to trust it, not the intake hops. Metrics and logs are cheap. Deciding what to do about them is the work. Step through the lesson or switch the scenario and watch the bar change. The storm makes the middle hops expensive, because volume taxes every queue. The false positive makes the last hop expensive, because the gate has to assemble evidence and wait for a human. The total is not the story. Which hop owns the total is the story.
A second picture of the same system. I start with one box and build to the full Master picture. The dot is one event, becoming one incident.
Legend: orange dot = one event, becoming one incident · hexagon = a service or model · shaded box = a store, log, or file · dashed line = the wire between boxes. Progression: Starts from scratch with one metric and one log line arriving — no AIOps vocabulary assumed. Adds one stage at a time until the trust gate and its two exits appear. Ends at the Master-level picture: seasonal baselines, a false-positive budget, and trust tiers that earn the system autonomy one safe fix at a time. · Traffic: orange = request forward, green = response back.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the flagged alerts, the incidents, the false positives, and the pages move. The numbers are a teaching example, not live measurements. The shape is the lesson: the threshold and the correlation window decide how much noise becomes a page, and auto-remediation only helps when the model is right.
How many raw events the pipeline takes in every minute: metrics, log lines, traces, alerts, all of it. A small service is near 100. A big platform is near 10,000. Everything downstream scales with this number.
How far from normal an event must be before the model flags it. Lower means the model catches more, and cries wolf more. Higher means it stays quiet, and misses more real problems. Three sigma is the usual starting point.
How long the pipeline waits to group related events into one incident. A short window splits one incident into three. A long window merges three incidents into one. Five minutes is the usual starting point.
Off means every page-worthy incident goes to a human. On means the trust gate auto-handles 70 percent of page-worthy incidents that pass it, and pages a human for the rest. On only helps if the model is right. If it is wrong, On automates the mistakes.
The pager is busy. Some of these are real, some are noise, and the engineer is starting to skim.
Try this: (1) Drop the threshold to 1 at the defaults. Flagged alerts jump from 30 to 50 a minute, false positives jump from 6 to 20, and pages jump from 60 to 100 an hour. You caught more. Most of what you caught is noise. (2) Now put the threshold back to 3 and turn auto-remediation on. Pages fall from 60 to 18 an hour. That only helps if the model is right. If it is wrong, you just automated 42 bad fixes an hour.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above walks one anomaly through nine hops, from the first metric to the trust gate. This section covers the same pipeline four times, from the first page I read to the level where the model earns the right to act on its own.
Definition: AIOps reads the signals I already watch and guesses what they mean. Metrics are numbers with a time. Logs are sentences about what happened. The model scores them against the past and says whether the present looks normal. Observability is the record. AIOps is an opinion about that record. I keep the two apart. A wrong opinion is fine. A lost record is not.
Types:
Uses: I use this level on every page. I open the incident, not the alert list. I look for the deploy attached to it, because the deploy is usually the cause. If the incident has no cause attached, I read the logs before I trust the score.
Dimensions:
Definition: Correlation groups events by service, region, and time window into one incident. Suppression drops the patterns the team already decided do not matter. Ranking puts the incident with the most impact first. These three hops are where the noise becomes a story a human can hold in their head.
Types:
Uses: I use this level when the pager is noisy. I check the grouping keys first, then the suppression list. If a suppression never expires, it is a blind spot with a permanent address, and I fix that before I touch the model.
Dimensions:
Definition: Systems have seasons. Traffic is high at noon and low at 3 AM, high on weekdays and low on weekends, high on sale days and low after them. A model trained on weekday data flags a quiet Sunday as broken. Systems also drift. A migration moves a service, latency shifts by 40 ms overnight, and the model calls the new normal an anomaly for weeks. I retrain on a schedule and I keep a human in the loop on the labels.
Types:
Uses: I use this level when the model pages for a deploy or a sale-day peak. I check what the training window included. If it never saw a deploy, the fix is training data, not a bigger disk or a lower threshold.
Dimensions:
Alert correlation is grouping by blast radius, not by time. A hundred alerts in one minute from one service is one incident, and the correlation rule groups them by the service and the shared dependency. A hundred alerts in one minute from a hundred services is a platform problem, and the rule groups them by the shared platform. The test is the incident count: after correlation, the number of incidents a human sees should be close to the number of real problems, not the number of alerts. I tune the rule by reading a real incident and checking whether the alerts that belonged together were grouped, and whether the alerts that were separate stayed separate.
A model that never learns from its mistakes keeps making them. I feed the analyst labels back into the training data: every alert an analyst marked as noise becomes a negative example, and every incident the model missed becomes a positive one. The retrain runs on a schedule, and the eval compares the new model with the old one on the same labeled set before it ships. The failure mode is the label drift: analysts who mark everything as noise to clear the queue teach the model that nothing matters. I check the label distribution once a month, and if one analyst labels ten times faster than the rest, I read their labels before I trust them.
Definition: I never let a model act on day one. Trust is earned in tiers. Tier one, the model only watches. Tier two, it suggests and a human approves. Tier three, it runs safe fixes alone. Tier four, it runs risky fixes, and only after months of tier three with measured precision above 95 percent. Every tier has a blast radius cap and a cooldown. A tier is a privilege. It is earned with evidence and lost with one bad night.
Types:
Uses: I use this level on the night the system hurts itself. I read the gate policy first, then the audit log. If restarts are in the auto-action list and the model is wrong about a memory climb, the gate is the only thing standing between a wrong meaning and a real action.
Dimensions:
Auto-remediation is a privilege with a blast radius cap. The model may restart a service, but it may not restart ten services at once. It may clear a cache, but it may not delete data. Every automated action has a cooldown, so a wrong action cannot repeat itself every minute, and every action is written to an audit log with the reason the model gave. The kill switch is a single flag that turns all automation off, and I test it by flipping it during a drill and confirming nothing automated runs. A model that acts without a kill switch is a model I do not deploy.
When the model is wrong, the first thing I read is the gate policy, not the model output. The policy says which actions are allowed, at which tier, and with what evidence. If the model restarted a service it should not have, the question is whether the action was in the allowed list, and if it was, whether the list is wrong. I treat a wrong automated action like a bad deploy: I roll back the action, I fix the policy, and I add the case to the eval set so the next version is tested against it. The model is not blamed. The gate is fixed, because the gate is the thing that was supposed to stop it.
The lesson above is one anomaly through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pipeline is running and the pager is quiet.
Observability and AIOps are not the same job, and confusing them is the most common mistake I see. Observability is the eyes. It is the metrics, logs, and traces from Lesson 25, collected and stored so a human can ask what happened. AIOps is the pattern matching on top. It reads what observability collected and guesses what it means. The order matters. If the model is wrong, the raw data is still there and a human can still read it. If you let the model replace the raw data, a wrong model leaves you blind. I keep them separate on purpose. Observability is the record of truth. AIOps is an opinion about that record. Opinions can be wrong. Records cannot be argued with.
Training a model on operations data is harder than it looks, because the labels are missing. A label is a human saying 'this was a real incident' or 'this was noise', and most teams never wrote those labels down. So the model learns from raw history with no teacher, and raw history is full of traps. Systems have seasons: traffic is high at noon and low at 3 AM, high on weekdays and low on weekends, high on sale days and low after them. A model trained on weekday data will flag a quiet Sunday as broken. Systems also drift. A migration moves a service to a new region, latency changes by 40 ms overnight, and the model calls the new normal an anomaly for weeks. I retrain on a schedule, I keep a human in the loop on the labels, and I never trust a model that has not seen a full season of my system's life.
A false positive is not free. It costs about twenty minutes of a human's attention: read the page, open the dashboard, check the service, decide it is nothing, go back to sleep. Ten false pages a day is more than three hours of an engineer's day, and the real cost is worse than the hours. It is trust. After the tenth false page, the engineer stops reading the eleventh carefully. The eleventh is the real one. That is how alert fatigue becomes an outage. I measure false positives like I measure uptime, and I set a budget for them: if more than one in five pages is false, the model is too sensitive and the threshold goes up, even if it means missing something small. A missed small problem is cheaper than a team that ignores the pager.
I never let a model act on day one. Trust is earned in tiers. Tier one, the model only watches and reports. It finds anomalies and a human decides everything. Tier two, the model suggests fixes and a human approves them. Tier three, the model runs safe fixes on its own: clear a cache, scale up one replica, restart a stateless pod. Tier four, the model runs risky fixes, and only after months of tier three with a measured precision above 95 percent. Every tier has a blast radius cap: one pod at a time, one region at a time, never everything at once. And every tier has a cooldown: after an automatic fix, the model waits before acting again, so one wrong idea cannot restart the same service ten times in a minute. A tier is a privilege. It is earned with evidence and lost with one bad night.
At 10x the event volume, the model itself becomes the bottleneck. Scoring has a latency budget like any other hop, and a model that takes two seconds to score is a model that pages late. I move scoring to a stream, score as events arrive, and keep the model small enough to run in the time budget. At 100x, the question changes from the model to the data. Storing every log line for a year costs more than the model saves, so I keep high-cardinality metrics for days, aggregated metrics for months, and raw logs only for the incidents that mattered. I also stop using one global model. One model for the whole platform learns the average, and the average is nobody's normal. Per-service models cost more to run and are right far more often.
This is the config shape I would actually write: one policy file, three jobs. It groups alerts into incidents, suppresses known noise, and sets which automatic actions the trust gate allows.
# aiops-policy.yaml — one policy file, three jobs: group, quiet, gate.
correlation:
window_minutes: 5
group_by: [service, region]
link_deploy_events: true # an incident that starts right after a deploy names the deploy
suppression:
- name: nightly-backup-disk-spike
match: {service: billing, metric: disk_used_pct}
window: "02:00-02:30"
reason: "Backup fills the disk for 20 minutes, then cleans up. Seen 40 times, never real."
expires: "2027-03-01"
auto_action:
allowed:
- clear_cache
- scale_up_one_replica
needs_human_approval:
- restart_pod
- rollback_deploy
never_automatic:
- database_failover
- delete_data
limits:
max_one_pod_at_a_time: true
cooldown_after_action_seconds: 300
Three jobs in one file, and I can read it in a minute. Correlation says what counts as one incident: same service, same region, inside five minutes, and if a deploy just landed, name it. Suppression says what is known noise: the nightly backup disk spike, with the reason written down and an expiry date, so it cannot become a permanent blind spot. Auto-action says what the gate may do alone, what needs a human, and what never runs on its own. The limits at the bottom are the safety net: one pod at a time, and five minutes of quiet after any automatic fix, so one wrong idea cannot cascade. When something goes wrong with AIOps, I read this file first. The model is a black box. This file is not.
Precision and recall pull in opposite directions, and the paging threshold is where I choose. Precision is the share of pages that were real. Recall is the share of real incidents that got paged. A threshold that pages on everything has perfect recall and terrible precision, and the team stops reading pages. A threshold that pages on nothing has perfect precision and terrible recall, and the incident nobody saw is the one that hurts. I pick the threshold from the cost: a missed incident costs more than a false page, so I accept more false pages, and I say that out loud in the policy, so nobody tunes it quietly later.
tp, fp, fn = 42, 18, 8
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(f"precision {precision:.2f} (42 real out of 60 pages), recall {recall:.2f} (42 caught out of 50 real incidents)")
print("18 false pages a week is the noise to cut first")
Expected output: precision 0.70 with 42 real pages out of 60, recall 0.84 with 42 caught out of 50 real incidents. The 18 false pages a week is the noise to cut first. Cutting the false pages raises precision without touching recall, which is the cheapest improvement in the whole system.
This one stayed with me because the system hurt itself on a schedule, and every automatic fix made the outage worse instead of better.
An auto-remediation loop restarted a healthy service over and over during a traffic spike, and made the spike worse. The model saw memory climbing on the checkout pods and called it a leak. The gate had restarts in the auto-action list, because restarts had been safe for months. So the system restarted a pod. The new pod came up cold: empty cache, cold connection pool, and it had to warm up while traffic was peaking. Memory climbed again on the next pod, because memory always climbs when a cold pod takes peak traffic. The model called it a leak again. It restarted the next pod. By the third restart, checkout latency had doubled, and the on-call engineer woke up to a system that was hurting itself on a schedule.
Four things. First, restarts moved out of the auto-action list and into needs-human-approval, where they stay until the model earns them back with three months of clean suggestions. Second, the model was retrained on data that includes sale-day peaks, so load no longer looks like a leak. Third, a cooldown was added: after any automatic action, the system waits five minutes before acting again on the same service, so one wrong idea cannot fire ten times. Fourth, the gate now checks the traffic trend before approving a restart. If traffic is climbing, memory climbing with it is expected, and the restart is held. The lesson I took from that night is simple. The model was not wrong about the numbers. It was wrong about what the numbers meant, and the gate let a wrong meaning become a real action.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real AIOps rollout. Take a minute with it before you read my approach.
Design AIOps for a platform with 200 services running on Kubernetes, producing 50,000 events a minute. The on-call rotation has 4 engineers. Today they get about 300 pages a week and half of them are noise. The team wants the pager quieter, the cause attached to every page, and auto-remediation for safe fixes only. Sketch the pipeline, the trust tiers, and the numbers you would watch to know it is working.
I would ask five things before I sized anything. What counts as an event here: every log line, or only the ones that already look like trouble? How much history do we have to train on, and does it include a full season: deploys, sale days, quiet Sundays, and at least one real outage? What is a page worth: which incidents must wake a human, and which can wait until morning? Which fixes are safe to automate, and who signs the list? And what is the failure we design for: the model going quiet and missing a real outage, or the model going loud and paging the team all night?
Start with the volume. 50,000 events a minute is about 833 a second. At 3 sigma the model flags 3 percent, so 1,500 alerts a minute. With a 5 minute correlation window, 10 alerts group into one incident, so 150 incidents a minute, and one in three is severe enough to page, so 50 pages a minute before auto-remediation. That is unusable, so the first job is not the model, it is suppression: the known noise patterns have to be written down and dropped before scoring, or the pager never quiets. Suppose suppression and better grouping cut the page-worthy incidents to 5 a minute, which is 300 an hour, still too many. Auto-remediation on, at 70 percent of page-worthy incidents handled, leaves 90 pages an hour. Still too many for 4 engineers. So the real target is the false-positive share: at 20 percent false, one page in five is noise, and 300 pages a week becomes 60 real ones plus the noise you still have to pay for. The sizing answer is not a bigger model. It is a quieter pipeline: suppress the known noise, group aggressively, and set the threshold so the false-positive share stays under 20 percent. Little's Law for the scoring hop: 833 events a second, each scored in 40 ms, means about 34 scoring jobs inside the system at once, so I plan the scoring workers for 34 plus headroom, and I alert at 80 percent. Storage: 50,000 events a minute is 72 million a day. I keep raw events for 7 days, aggregated metrics for 90 days, and full detail only for the incidents that paged someone.
A lower threshold catches more real problems and pages more noise. A higher threshold is quieter and misses more. A longer correlation window groups better and merges unrelated incidents. A shorter window splits one incident into three pages. Per-service models are right more often and cost more to train and run. One global model is cheap and learns nobody's normal. Auto-remediation buys sleep and risks automating a mistake, so it starts at tier one, observe only, and earns each higher tier with measured precision. And the pager budget is a contract: if false positives pass one in five, the threshold goes up, even if it means missing something small. A team that trusts the pager is worth more than a model that catches everything.
Saved in this browser, so the Lessons index can show your progress across the course.