Platform Engineering · Lesson 77

AIOps

AIOps is software that watches your systems the way an on-call engineer watches them, and does it all the time without getting tired. That is the whole idea. Your services already produce the signals: numbers like CPU and error rate, words like log lines, and alerts from every tool you own. On a bad night those signals arrive by the thousands, and a human cannot read them all. AIOps reads them instead. It learns what normal looks like for your system. When something drifts away from normal, it groups the noise into one incident, ranks it, and either fixes it or wakes a human with the story already written. Here is one concrete night. A deploy goes out at 2 AM. Error rate on the checkout service climbs from 0.1 percent to 4 percent. The old way, your phone buzzes forty times in five minutes, once per alert, and you spend the first ten minutes just working out that all forty alerts are the same problem. The AIOps way, the system sees forty alerts from one service in one five-minute window, calls it one incident, points at the deploy that just landed, and drafts a rollback. You get one page, with the cause attached. That is the job. Less noise, faster cause, and a fix that is ready before you are fully awake. This lesson follows one anomaly through the whole pipeline, from the first metric to the final decision about who acts.

Difficulty: Advanced · Know this first: Lesson 48, Lesson 61
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one incident travelling through the pipeline, not a measurement from a live system, but the shape is honest: the expensive hops are the thinking hops, drafting a fix and deciding whether to trust it, not the intake hops. Metrics and logs are cheap. Deciding what to do about them is the work. Step through the lesson or switch the scenario and watch the bar change. The storm makes the middle hops expensive, because volume taxes every queue. The false positive makes the last hop expensive, because the gate has to assemble evidence and wait for a human. The total is not the story. Which hop owns the total is the story.

Extra diagram — A thousand signals, one incident — the AIOps machine

A second picture of the same system. I start with one box and build to the full Master picture. The dot is one event, becoming one incident.

11 stepsMoving: one event, becoming one incidentBasic to Master
Services1Sources converging:metrics,4 SOURCESNormalizer1 SHAPEAnomaly model4Correlator3 KEYSSuppression listDROPPEDRanker3 AT RISKFix draft8Trust gate9Two exits: autoaction vs human2 EXITSBuild-up: seasonalbaselines,FP BUDGETone event, becoming one inci
Extra diagram

    Legend: orange dot = one event, becoming one incident  ·  hexagon = a service or model  ·  shaded box = a store, log, or file  ·  dashed line = the wire between boxes. Progression: Starts from scratch with one metric and one log line arriving — no AIOps vocabulary assumed. Adds one stage at a time until the trust gate and its two exits appear. Ends at the Master-level picture: seasonal baselines, a false-positive budget, and trust tiers that earn the system autonomy one safe fix at a time.  ·  Traffic: orange = request forward, green = response back.

    Turn the knobs and watch the pager

    The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the flagged alerts, the incidents, the false positives, and the pages move. The numbers are a teaching example, not live measurements. The shape is the lesson: the threshold and the correlation window decide how much noise becomes a page, and auto-remediation only helps when the model is right.

    How many raw events the pipeline takes in every minute: metrics, log lines, traces, alerts, all of it. A small service is near 100. A big platform is near 10,000. Everything downstream scales with this number.

    How far from normal an event must be before the model flags it. Lower means the model catches more, and cries wolf more. Higher means it stays quiet, and misses more real problems. Three sigma is the usual starting point.

    How long the pipeline waits to group related events into one incident. A short window splits one incident into three. A long window merges three incidents into one. Five minutes is the usual starting point.

    Off means every page-worthy incident goes to a human. On means the trust gate auto-handles 70 percent of page-worthy incidents that pass it, and pages a human for the rest. On only helps if the model is right. If it is wrong, On automates the mistakes.

    Alerts flagged per minute30 / min1,000 events a minute, 3% flagged
    Incidents after correlation per minute3 / min30 flagged a minute grouped 10 to an incident
    False positives per minute6 / min20% of flagged alerts are false positives
    Human pages per hour60 / hour3 incidents a minute, 1 in 3 pages a human

    The pager is busy. Some of these are real, some are noise, and the engineer is starting to skim.

    Try this: (1) Drop the threshold to 1 at the defaults. Flagged alerts jump from 30 to 50 a minute, false positives jump from 6 to 20, and pages jump from 60 to 100 an hour. You caught more. Most of what you caught is noise. (2) Now put the threshold back to 3 and turn auto-remediation on. Pages fall from 60 to 18 an hour. That only helps if the model is right. If it is wrong, you just automated 42 bad fixes an hour.

    Check yourself

    Three questions. No score is saved anywhere — this is just you against the material.

    Basic to master

    The lesson above walks one anomaly through nine hops, from the first metric to the trust gate. This section covers the same pipeline four times, from the first page I read to the level where the model earns the right to act on its own.

    Basic — Read the incident, not the alerts

    Definition: AIOps reads the signals I already watch and guesses what they mean. Metrics are numbers with a time. Logs are sentences about what happened. The model scores them against the past and says whether the present looks normal. Observability is the record. AIOps is an opinion about that record. I keep the two apart. A wrong opinion is fine. A lost record is not.

    Types:

    • The metric — a number with a name and a time, arriving every few seconds.
    • The log line — the witness that was in the room, saying what happened.
    • The anomaly score — how far this event sits from what the system usually does.
    • The correlated incident — forty alerts grouped into one story, with the cause attached.

    Uses: I use this level on every page. I open the incident, not the alert list. I look for the deploy attached to it, because the deploy is usually the cause. If the incident has no cause attached, I read the logs before I trust the score.

    Dimensions:

    • Load: At one service the pipeline is quiet. At two hundred services the events arrive by the tens of thousands a minute, and the scoring hop has a latency budget like any other hop.
    • Breaks: It breaks when the collector dies and the metrics stop. Silence from a collector is data, not the absence of data. I notice it when the pipeline goes quiet on a night that should be noisy.
    • Cost: The cost is the storage for the signals and the compute to score them. Keeping every log line for a year costs more than the model saves.
    • Trade-off: I give up reading every alert myself to get a pipeline that reads them all night. I lose the habit of scanning the raw feed. I gain a night where one incident replaces four hundred alerts.

    Intermediate — Correlation and suppression

    Definition: Correlation groups events by service, region, and time window into one incident. Suppression drops the patterns the team already decided do not matter. Ranking puts the incident with the most impact first. These three hops are where the noise becomes a story a human can hold in their head.

    Types:

    • The correlation window — five minutes by default. Shorter splits one incident into three. Longer merges three into one.
    • The suppression rule — a known noise pattern, with the reason written down and an expiry date.
    • The severity rank — impact, not loudness. A quiet payment failure outranks a loud dashboard warning.
    • The deploy event — the link that names the cause when an incident starts right after a deploy.

    Uses: I use this level when the pager is noisy. I check the grouping keys first, then the suppression list. If a suppression never expires, it is a blind spot with a permanent address, and I fix that before I touch the model.

    Dimensions:

    • Load: At low volume correlation is instant. At high volume the grouping itself gets slow, and the queue behind it grows while the incidents wait.
    • Breaks: It breaks when the window is wrong. One incident split into three pages the team three times. Three incidents merged into one hides the real cause behind the loudest symptom.
    • Cost: The cost is the time to write the suppression rules and keep them honest. A rule without an expiry date is a debt that grows in the dark.
    • Trade-off: I give up seeing every alert to get one incident with a cause. The raw alerts are still there if I need them. I just stop reading them first.

    Advanced — Seasons, drift, and the false-positive budget

    Definition: Systems have seasons. Traffic is high at noon and low at 3 AM, high on weekdays and low on weekends, high on sale days and low after them. A model trained on weekday data flags a quiet Sunday as broken. Systems also drift. A migration moves a service, latency shifts by 40 ms overnight, and the model calls the new normal an anomaly for weeks. I retrain on a schedule and I keep a human in the loop on the labels.

    Types:

    • The season — the daily and weekly shape of normal, learned from a full cycle of the system's life.
    • The drift — the system changing after training, so the model scores against a past that no longer exists.
    • The threshold — three sigma by default. Lower catches more and cries wolf more. Higher stays quiet and misses more.
    • The false-positive budget — one page in five, at most. Past that, the threshold goes up even if it means missing something small.

    Uses: I use this level when the model pages for a deploy or a sale-day peak. I check what the training window included. If it never saw a deploy, the fix is training data, not a bigger disk or a lower threshold.

    Dimensions:

    • Load: At 10x the volume the model itself becomes the bottleneck. I move scoring to a stream and keep the model small enough to fit the time budget.
    • Breaks: It breaks when the model has never seen the pattern. A sale-day peak looks like a leak. A deploy looks like an anomaly. I notice it when the page repeats on a schedule.
    • Cost: The cost is retraining and the labels. A model with no human labels learns from raw history, and raw history is full of traps.
    • Trade-off: I give up the global model to get per-service models. One model for the whole platform learns the average, and the average is nobody's normal. Per-service models cost more and are right far more often.

    Alert correlation is grouping by blast radius, not by time. A hundred alerts in one minute from one service is one incident, and the correlation rule groups them by the service and the shared dependency. A hundred alerts in one minute from a hundred services is a platform problem, and the rule groups them by the shared platform. The test is the incident count: after correlation, the number of incidents a human sees should be close to the number of real problems, not the number of alerts. I tune the rule by reading a real incident and checking whether the alerts that belonged together were grouped, and whether the alerts that were separate stayed separate.

    A model that never learns from its mistakes keeps making them. I feed the analyst labels back into the training data: every alert an analyst marked as noise becomes a negative example, and every incident the model missed becomes a positive one. The retrain runs on a schedule, and the eval compares the new model with the old one on the same labeled set before it ships. The failure mode is the label drift: analysts who mark everything as noise to clear the queue teach the model that nothing matters. I check the label distribution once a month, and if one analyst labels ten times faster than the rest, I read their labels before I trust them.

    Master — The gate earns trust, one safe fix at a time

    Definition: I never let a model act on day one. Trust is earned in tiers. Tier one, the model only watches. Tier two, it suggests and a human approves. Tier three, it runs safe fixes alone. Tier four, it runs risky fixes, and only after months of tier three with measured precision above 95 percent. Every tier has a blast radius cap and a cooldown. A tier is a privilege. It is earned with evidence and lost with one bad night.

    Types:

    • The trust tier — the level of action the model has earned, written down in the gate policy.
    • The auto-action list — what the gate may do alone, what needs a human, and what never runs alone.
    • The blast radius cap — one pod at a time, one region at a time, never everything at once.
    • The audit log — every automatic action recorded, so a human can read later exactly what happened and why.

    Uses: I use this level on the night the system hurts itself. I read the gate policy first, then the audit log. If restarts are in the auto-action list and the model is wrong about a memory climb, the gate is the only thing standing between a wrong meaning and a real action.

    Dimensions:

    • Load: At 100x the question is the data, not the model. I keep high-cardinality metrics for days, aggregated metrics for months, and raw logs only for the incidents that mattered.
    • Breaks: It breaks when the gate approves a restart for a cold pod under peak traffic. The new pod warms up slow, memory climbs again, and the loop restarts the next pod. I notice it in the restart count, climbing in a loop.
    • Cost: The cost is the gate itself: the policy, the audit log, the cooldowns, and the human time to review the suggestions. A gate nobody reviews is a queue, not a safety net.
    • Trade-off: I give up speed to get safety. A fix that waits five minutes for a human is slower than a fix that runs now. It is also the fix that does not restart a healthy service three times in twenty minutes.

    Auto-remediation is a privilege with a blast radius cap. The model may restart a service, but it may not restart ten services at once. It may clear a cache, but it may not delete data. Every automated action has a cooldown, so a wrong action cannot repeat itself every minute, and every action is written to an audit log with the reason the model gave. The kill switch is a single flag that turns all automation off, and I test it by flipping it during a drill and confirming nothing automated runs. A model that acts without a kill switch is a model I do not deploy.

    When the model is wrong, the first thing I read is the gate policy, not the model output. The policy says which actions are allowed, at which tier, and with what evidence. If the model restarted a service it should not have, the question is whether the action was in the allowed list, and if it was, whether the list is wrong. I treat a wrong automated action like a bad deploy: I roll back the action, I fix the policy, and I add the case to the eval set so the next version is tested against it. The model is not blamed. The gate is fixed, because the gate is the thing that was supposed to stop it.

    Go deeper

    The lesson above is one anomaly through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pipeline is running and the pager is quiet.

    AIOps vs observability

    Observability and AIOps are not the same job, and confusing them is the most common mistake I see. Observability is the eyes. It is the metrics, logs, and traces from Lesson 25, collected and stored so a human can ask what happened. AIOps is the pattern matching on top. It reads what observability collected and guesses what it means. The order matters. If the model is wrong, the raw data is still there and a human can still read it. If you let the model replace the raw data, a wrong model leaves you blind. I keep them separate on purpose. Observability is the record of truth. AIOps is an opinion about that record. Opinions can be wrong. Records cannot be argued with.

    Training on ops data

    Training a model on operations data is harder than it looks, because the labels are missing. A label is a human saying 'this was a real incident' or 'this was noise', and most teams never wrote those labels down. So the model learns from raw history with no teacher, and raw history is full of traps. Systems have seasons: traffic is high at noon and low at 3 AM, high on weekdays and low on weekends, high on sale days and low after them. A model trained on weekday data will flag a quiet Sunday as broken. Systems also drift. A migration moves a service to a new region, latency changes by 40 ms overnight, and the model calls the new normal an anomaly for weeks. I retrain on a schedule, I keep a human in the loop on the labels, and I never trust a model that has not seen a full season of my system's life.

    The cost of a false positive

    A false positive is not free. It costs about twenty minutes of a human's attention: read the page, open the dashboard, check the service, decide it is nothing, go back to sleep. Ten false pages a day is more than three hours of an engineer's day, and the real cost is worse than the hours. It is trust. After the tenth false page, the engineer stops reading the eleventh carefully. The eleventh is the real one. That is how alert fatigue becomes an outage. I measure false positives like I measure uptime, and I set a budget for them: if more than one in five pages is false, the model is too sensitive and the threshold goes up, even if it means missing something small. A missed small problem is cheaper than a team that ignores the pager.

    Trust tiers

    I never let a model act on day one. Trust is earned in tiers. Tier one, the model only watches and reports. It finds anomalies and a human decides everything. Tier two, the model suggests fixes and a human approves them. Tier three, the model runs safe fixes on its own: clear a cache, scale up one replica, restart a stateless pod. Tier four, the model runs risky fixes, and only after months of tier three with a measured precision above 95 percent. Every tier has a blast radius cap: one pod at a time, one region at a time, never everything at once. And every tier has a cooldown: after an automatic fix, the model waits before acting again, so one wrong idea cannot restart the same service ten times in a minute. A tier is a privilege. It is earned with evidence and lost with one bad night.

    What changes at 10x / 100x

    At 10x the event volume, the model itself becomes the bottleneck. Scoring has a latency budget like any other hop, and a model that takes two seconds to score is a model that pages late. I move scoring to a stream, score as events arrive, and keep the model small enough to run in the time budget. At 100x, the question changes from the model to the data. Storing every log line for a year costs more than the model saves, so I keep high-cardinality metrics for days, aggregated metrics for months, and raw logs only for the incidents that mattered. I also stop using one global model. One model for the whole platform learns the average, and the average is nobody's normal. Per-service models cost more to run and are right far more often.

    A correlation, suppression, and auto-action policy

    This is the config shape I would actually write: one policy file, three jobs. It groups alerts into incidents, suppresses known noise, and sets which automatic actions the trust gate allows.

    # aiops-policy.yaml — one policy file, three jobs: group, quiet, gate.
    correlation:
      window_minutes: 5
      group_by: [service, region]
      link_deploy_events: true   # an incident that starts right after a deploy names the deploy
    
    suppression:
      - name: nightly-backup-disk-spike
        match: {service: billing, metric: disk_used_pct}
        window: "02:00-02:30"
        reason: "Backup fills the disk for 20 minutes, then cleans up. Seen 40 times, never real."
        expires: "2027-03-01"
    
    auto_action:
      allowed:
        - clear_cache
        - scale_up_one_replica
      needs_human_approval:
        - restart_pod
        - rollback_deploy
      never_automatic:
        - database_failover
        - delete_data
      limits:
        max_one_pod_at_a_time: true
        cooldown_after_action_seconds: 300

    Three jobs in one file, and I can read it in a minute. Correlation says what counts as one incident: same service, same region, inside five minutes, and if a deploy just landed, name it. Suppression says what is known noise: the nightly backup disk spike, with the reason written down and an expiry date, so it cannot become a permanent blind spot. Auto-action says what the gate may do alone, what needs a human, and what never runs on its own. The limits at the bottom are the safety net: one pod at a time, and five minutes of quiet after any automatic fix, so one wrong idea cannot cascade. When something goes wrong with AIOps, I read this file first. The model is a black box. This file is not.

    Precision and recall pull in opposite directions, and the paging threshold is where I choose. Precision is the share of pages that were real. Recall is the share of real incidents that got paged. A threshold that pages on everything has perfect recall and terrible precision, and the team stops reading pages. A threshold that pages on nothing has perfect precision and terrible recall, and the incident nobody saw is the one that hurts. I pick the threshold from the cost: a missed incident costs more than a false page, so I accept more false pages, and I say that out loud in the policy, so nobody tunes it quietly later.

    tp, fp, fn = 42, 18, 8
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    print(f"precision {precision:.2f} (42 real out of 60 pages), recall {recall:.2f} (42 caught out of 50 real incidents)")
    print("18 false pages a week is the noise to cut first")

    Expected output: precision 0.70 with 42 real pages out of 60, recall 0.84 with 42 caught out of 50 real incidents. The 18 false pages a week is the noise to cut first. Cutting the false pages raises precision without touching recall, which is the cheapest improvement in the whole system.

    One time it broke

    This one stayed with me because the system hurt itself on a schedule, and every automatic fix made the outage worse instead of better.

    What broke

    An auto-remediation loop restarted a healthy service over and over during a traffic spike, and made the spike worse. The model saw memory climbing on the checkout pods and called it a leak. The gate had restarts in the auto-action list, because restarts had been safe for months. So the system restarted a pod. The new pod came up cold: empty cache, cold connection pool, and it had to warm up while traffic was peaking. Memory climbed again on the next pod, because memory always climbs when a cold pod takes peak traffic. The model called it a leak again. It restarted the next pod. By the third restart, checkout latency had doubled, and the on-call engineer woke up to a system that was hurting itself on a schedule.

    Timeline

    • 01:40Traffic peaks after a sale goes live. Memory on the checkout pods climbs as expected under load.
    • 01:47The model flags the memory climb as a leak. It has never seen a sale-day peak in its training data.
    • 01:48The gate auto-approves a restart. Restarts have been safe for months, so they sit in the auto-action list.
    • 01:52The new pod comes up cold. Empty cache, cold pool. It takes the same traffic and its memory climbs the same way.
    • 01:55The model flags the second pod. The gate approves a second restart.
    • 02:05Third restart. Checkout latency has doubled. The restarts are now the outage, not the response to one.
    • 02:10The on-call engineer wakes up, reads the audit log, and sees three automatic restarts in twenty minutes. She disables auto-remediation and the system settles on its own.
    • 02:30Traffic stays high, memory stays high, checkout works. The memory climb was load, not a leak. It never was.

    What changed after

    Four things. First, restarts moved out of the auto-action list and into needs-human-approval, where they stay until the model earns them back with three months of clean suggestions. Second, the model was retrained on data that includes sale-day peaks, so load no longer looks like a leak. Third, a cooldown was added: after any automatic action, the system waits five minutes before acting again on the same service, so one wrong idea cannot fire ten times. Fourth, the gate now checks the traffic trend before approving a restart. If traffic is climbing, memory climbing with it is expected, and the restart is held. The lesson I took from that night is simple. The model was not wrong about the numbers. It was wrong about what the numbers meant, and the gate let a wrong meaning become a real action.

    If you remember nothing else

    Design exercise

    This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real AIOps rollout. Take a minute with it before you read my approach.

    The prompt

    Design AIOps for a platform with 200 services running on Kubernetes, producing 50,000 events a minute. The on-call rotation has 4 engineers. Today they get about 300 pages a week and half of them are noise. The team wants the pager quieter, the cause attached to every page, and auto-remediation for safe fixes only. Sketch the pipeline, the trust tiers, and the numbers you would watch to know it is working.

    Clarifying questions to ask first

    I would ask five things before I sized anything. What counts as an event here: every log line, or only the ones that already look like trouble? How much history do we have to train on, and does it include a full season: deploys, sale days, quiet Sundays, and at least one real outage? What is a page worth: which incidents must wake a human, and which can wait until morning? Which fixes are safe to automate, and who signs the list? And what is the failure we design for: the model going quiet and missing a real outage, or the model going loud and paging the team all night?

    Rough sizing, worked

    Start with the volume. 50,000 events a minute is about 833 a second. At 3 sigma the model flags 3 percent, so 1,500 alerts a minute. With a 5 minute correlation window, 10 alerts group into one incident, so 150 incidents a minute, and one in three is severe enough to page, so 50 pages a minute before auto-remediation. That is unusable, so the first job is not the model, it is suppression: the known noise patterns have to be written down and dropped before scoring, or the pager never quiets. Suppose suppression and better grouping cut the page-worthy incidents to 5 a minute, which is 300 an hour, still too many. Auto-remediation on, at 70 percent of page-worthy incidents handled, leaves 90 pages an hour. Still too many for 4 engineers. So the real target is the false-positive share: at 20 percent false, one page in five is noise, and 300 pages a week becomes 60 real ones plus the noise you still have to pay for. The sizing answer is not a bigger model. It is a quieter pipeline: suppress the known noise, group aggressively, and set the threshold so the false-positive share stays under 20 percent. Little's Law for the scoring hop: 833 events a second, each scored in 40 ms, means about 34 scoring jobs inside the system at once, so I plan the scoring workers for 34 plus headroom, and I alert at 80 percent. Storage: 50,000 events a minute is 72 million a day. I keep raw events for 7 days, aggregated metrics for 90 days, and full detail only for the incidents that paged someone.

    Trade-offs to name

    A lower threshold catches more real problems and pages more noise. A higher threshold is quieter and misses more. A longer correlation window groups better and merges unrelated incidents. A shorter window splits one incident into three pages. Per-service models are right more often and cost more to train and run. One global model is cheap and learns nobody's normal. Auto-remediation buys sleep and risks automating a mistake, so it starts at tier one, observe only, and earns each higher tier with measured precision. And the pager budget is a contract: if false positives pass one in five, the threshold goes up, even if it means missing something small. A team that trusts the pager is worth more than a model that catches everything.

    Saved in this browser, so the Lessons index can show your progress across the course.

    Next in this series
    Lesson 78 — NoOps

    AIOps watches the systems and decides what to do about them. NoOps asks a bigger question: what if the operations work disappeared altogether? The next lesson is about automating operations to the point where no dedicated ops team is needed, and where that idea works and where it falls over.