SecOps is the team and the system that watches your production systems for attackers and deals with them when they show up. Think of it this way. DevSecOps is about building security into the code before it ships. SecOps is about what happens after it ships: someone, somewhere, is always trying to get in, and SecOps is the watchtower that spots them. Here is one concrete example. A login happens at 2 AM from a country your users never log in from, using a service account that normally only talks to one other service. That is not a bug. That is a person, or a script, trying your door. SecOps is the pipeline that notices that login, decides it matters, wakes the right human, and locks the door before the attacker gets further. This lesson follows one suspicious event through that pipeline, hop by hop, from a raw log line to a contained incident.
Every hop charges rent in milliseconds, and these numbers are a worked teaching example for one event, not measurements from a live system. The shape is the honest part. On a clean catch, the pipeline is fast and the human steps are the slow part in real life, which is why the simulator below measures human time in minutes. Switch the scenario and watch the bar change. The alert storm nearly triples the trip without the attacker doing anything different, because the pipeline pays the same enrichment and investigation tax forty times. The quiet miss looks cheap until you count the three days it went unnoticed, and no bar can show that.
The budget above is one fixed event. This box is yours. Push the event volume up, add more rules, change the shift size, and watch what happens to the queue. Then ask the question I ask before every on-call handover: how many alerts can this team actually handle before the real one gets lost.
How many log events arrive every second. More events means slower parsing and more alerts an hour, because a fixed share of events trips a rule.
How many detection rules run against each event. More rules means more questions asked per event, and each question costs time.
How many analysts are working the queue. Each analyst clears about 20 alerts an hour. Fewer analysts means a longer wait for every alert.
On means the playbook disables the token and kills the session in about 1.5 minutes after the analyst approves it. Off means a human does every step by hand, about 45 minutes.
The team is keeping up. Alerts are triaged fast and containment lands before the attacker gets far.
Try this: (1) Push events per second to 20,000 with 3 analysts. Alerts jump to 120 an hour, triage wait hits 2 minutes, and time to contain climbs past 6 minutes. (2) Now drop analysts to 1 and turn auto-contain off. Time to contain jumps past 50 minutes. Same pipeline, same rules. Only the staffing and the automation changed.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times, from the first plain idea to the level where I run it in production and debug it at 2 AM. Each level adds one thing the level below cannot do.
Definition: SecOps is the team and the system that watches your production systems for attackers and deals with them when they show up. Think of it this way. DevSecOps is about building security into the code before it ships. SecOps is about what happens after it ships: someone, somewhere, is always trying to get in, and SecOps is the watchtower that spots them.
Types:
Uses: Use it the moment a service is reachable from the internet. The concrete example from this lesson is a login at 2 AM from a country your users never log in from, using a service account that normally only talks to one other service. That is not a bug. That is a person, or a script, trying your door. SecOps is the pipeline that notices that login, decides it matters, wakes the right human, and locks the door.
Dimensions: At low volume one analyst reads the queue. At 10x the events the pipeline breaks before the analysts do: parsing and enrichment get slow, the storage bill doubles, and the rules take longer to run. What breaks is a rule written in a hurry during an incident: it fires all night and teaches the whole team to ignore it. It costs the log pipeline, the storage, and the analysts' hours. You give up ignoring logs for a queue that never empties. The work is deciding what is worth a human's time.
Definition: A detection rule is code, so I treat it like code. It lives in a repo, it gets a pull request, it gets reviewed, and it gets tested before it ships. The test that matters most is the replay: run the new rule against last week's logs and count what it would have fired on.
Types:
Uses: Use it before a rule ships. If it fires two hundred times on old logs and every one is a false positive, the rule does not ship. A rule without a negative test is a rule that fires on your own deploys at 2 AM. And use the hunt for the attack no rule caught: start with a hypothesis, not a dashboard. If an attacker stole a token, they would use it at an odd hour, from an odd place, to touch data the service never touches. Then go look for exactly that.
Dimensions: At 10x the volume the storage bill doubles, so I split the pipeline: hot storage for the last seven days, cold storage for the rest, and the rules run against the hot store only. What breaks is the attack no rule caught, which is what the weekly hunt is for. Nothing found is still a result, because it tells you the hypothesis was wrong or the logs are missing. It costs storage, rule runtime, and analyst time per alert. You give up shipping rules fast for rules that earn their place in the queue. A rule that fires on noise costs trust.
Definition: SOAR is a word for a simple idea: when an alert fires, a playbook runs a fixed set of steps automatically. Disable the token. Kill the session. Block the IP. Open the ticket. Put the evidence in the ticket. The playbook does in seconds what a tired human does in forty-five minutes, and it does it the same way every time.
Types:
Uses: Use it when the alert is real and the clock is running. The line I draw is this: if the action can break a real user, a human approves it. If the action can only hurt the attacker, the playbook runs on its own. I automate the containment and I keep the human on the approval for anything that touches production traffic.
Dimensions: Three analysts, each clearing about 20 alerts an hour, clear 60 an hour. If the pipeline produces 30 alerts an hour, the team is comfortable and the queue stays short. If a noisy rule pushes that to 120 an hour, the team clears half of what arrives, the queue grows by 60 an hour, and by the end of an eight-hour shift there are 480 unhandled alerts. Nobody reads alert number 481. What breaks is a playbook that isolates the wrong pod: an outage I automated, so I test playbooks on a staging namespace first. It costs the playbook platform, the enrichment, and the testing. You give up human judgment on containment for speed. You keep the human on anything irreversible.
Definition: At this level alert fatigue is not a feeling. It is a math problem. The fix is never "hire more analysts" first. The fix is to cut the noise at the source: tune the rule, raise the threshold, add a condition, or turn the rule into a hunt instead of an alert.
Types:
Uses: Use it when the queue grows faster than the team clears it. Every alert that reaches a human should be worth a human's time. If it is not, it should be a dashboard, not a page. And use it to remember the split: DevSecOps makes the attack harder, SecOps catches the attack that gets through anyway. A team that only does one of them is half a security team.
Dimensions: At 100x detection becomes a sampling and streaming problem. I stop trying to run every rule against every event in one place, and I push the cheap rules down to the collector. What breaks is a team with perfect shift-left scanning getting breached by a stolen token, because a scanned image does not stop a valid login from a bad place. It costs the pipeline, the central store, the analysts, and the cost of the alerts nobody reads. You give up catching everything for catching what matters: fewer alerts, each one worth a human.
The lesson above is one event through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pipeline is running and the queue is staffed.
A detection rule is code, so I treat it like code. It lives in a repo, it gets a pull request, it gets reviewed, and it gets tested before it ships. The test that matters most is the replay: run the new rule against last week's logs and count what it would have fired on. If it fires two hundred times on old logs and every one is a false positive, the rule does not ship. I learned that the hard way, after a rule I wrote in a hurry during an incident fired all night and taught the whole team to ignore it. Here is what a rule looks like in our repo.
# detections/svc-account-external-login.yaml
name: service-account-login-outside-cluster
description: A service account logged in from outside the cluster.
Service accounts should never log in from outside. Ever.
severity: high
schedule: every 5 minutes
lookback: 10 minutes
query: |
event.action == "login"
and user.type == "service-account"
and source.network != "cluster-internal"
and source.country != user.home_country
test:
- name: real attack
expect: fires
log: "svc-billing logged in from a new country at 02:14"
- name: normal deploy
expect: no fire
log: "svc-billing authenticated inside the cluster"
owner: secops-oncall
What the snippet means: the rule has a name, a plain description, a severity, and a schedule. It runs every five minutes and looks back ten. The query is four conditions joined by and: a login, by a service account, from outside the cluster, from a country that is not the user's home country. All four have to be true. The test block is the part teams skip, and it is the part that saves you. Two cases, one that must fire and one that must not. A rule without a negative test is a rule that fires on your own deploys at 2 AM.
SOAR is a word for a simple idea: when an alert fires, a playbook runs a fixed set of steps automatically. Disable the token. Kill the session. Block the IP. Open the ticket. Put the evidence in the ticket. The playbook does in seconds what a tired human does in forty-five minutes, and it does it the same way every time. I automate the containment and I keep the human on the approval for anything that touches production traffic. The line I draw is this: if the action can break a real user, a human approves it. If the action can only hurt the attacker, the playbook runs on its own. A playbook that isolates the wrong pod is an outage I automated, so I test playbooks on a staging namespace first, with the same care I test a deploy.
Threat hunting is looking for the attack that no rule caught. It starts with a hypothesis, not a dashboard. "If an attacker stole a token, they would use it at an odd hour, from an odd place, to touch data the service never touches." Then you go look for exactly that in the logs. A hunt is not a random search. It is a question with a time box: two hours, one hypothesis, and a written answer at the end, even if the answer is "nothing found." Nothing found is still a result, because it tells you the hypothesis was wrong or the logs are missing. I schedule one hunt a week, and the hunts that find nothing are the ones that keep the rules honest.
People mix these two up, so here is the plain split. DevSecOps is the shift-left work: secure code, scanned images, secrets out of the repo, least privilege on the service account. It happens before the deploy, and it is mostly done by the platform and app teams with security's help. SecOps is the watchtower work: the logs, the rules, the queue, the 2 AM call. It happens after the deploy, and it never ends, because the attacker never stops. They are the same mission at different points in time. DevSecOps makes the attack harder. SecOps catches the attack that gets through anyway. A team that only does one of them is half a security team. I have watched a team with perfect shift-left scanning get breached by a stolen token, because a scanned image does not stop a valid login from a bad place.
Alert fatigue is not a feeling. It is a math problem. A team of three analysts, each clearing about 20 alerts an hour, clears 60 an hour. If the pipeline produces 30 alerts an hour, the team is comfortable and the queue stays short. If a noisy rule pushes that to 120 an hour, the team clears half of what arrives, the queue grows by 60 an hour, and by the end of an eight-hour shift there are 480 unhandled alerts. Nobody reads alert number 481. The fix is never "hire more analysts" first. The fix is to cut the noise at the source: tune the rule, raise the threshold, add a condition, or turn the rule into a hunt instead of an alert. Every alert that reaches a human should be worth a human's time. If it is not, it should be a dashboard, not a page.
At 10x the event volume, the pipeline breaks before the analysts do. Parsing and enrichment get slow, the SIEM storage bill doubles, and the rules take longer to run. I split the pipeline: hot storage for the last seven days, cold storage for the rest, and the rules run against the hot store only. At 100x, detection becomes a sampling and streaming problem. I stop trying to run every rule against every event in one place, and I push the cheap rules down to the collector, so the obvious noise never reaches the SIEM at all. The queue does not scale by adding analysts. It scales by making each alert worth more: better enrichment, better scoring, and fewer alerts that are not worth a human's time. The budget stops being a target and becomes a contract, and every hop owns its share of it.
This one stayed with me because the system worked exactly as designed, and the attacker still got in. Every dashboard was green. The queue was empty. The breach was three days old before anyone knew. Treat this as an illustrative scenario, a composite of a pattern I have seen more than once, not one company's incident report.
A service account token was printed into a build log during a deploy. The log went to a shared log store that more people could read than anyone remembered. Someone, or something, copied the token. Three days later it was used to log in from a normal-looking location, during business hours, to read data the service reads every day. Nothing about the event was unusual enough to trip a rule. The login was valid. The token was valid. The location was plausible. The pipeline did its job and caught nothing, because no rule asked the right question.
Four things. First, the deploy script stopped printing secrets, and a CI check now fails the build if a token-shaped string appears in the logs. That is Lesson 33 doing security work. Second, the hunt that found it became a scheduled weekly hunt, not a one-off. Third, a new detection rule went in: a service account login from outside the cluster fires a high-severity alert, tested against a week of old logs, shipped through CI/CD. Fourth, the log store permissions were fixed, and an audit now runs monthly to check who can read the shared logs. The breach was not a clever hack. It was a token in a log and a rule that did not exist. Both were fixable, and both are fixed now.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real SOC. Take a minute with it before you read my approach.
Design the security operations setup for a mid-size SaaS company. Fifty services on Kubernetes, 5,000 events a second across the fleet, three analysts on shift, and a promise to the business that a real intrusion is contained within one hour. Sketch the pipeline, the rules, the queue, and the numbers that prove the hour is real.
I would ask five things before I sized anything. What are the crown jewels: which data, if stolen, actually hurts the business, because that decides what the rules protect first. Where do the logs live today, and what is not being collected at all, because a blind spot decides the design before any rule does. What does "contained within one hour" mean: from the first bad event, or from the alert firing, because the answer changes the budget by the detection time. How many alerts does the team get today, and how many are false positives, because a team drowning in noise cannot take on a new promise. And who is on call at 2 AM: is it a dedicated analyst, or an SRE woken up by a page they did not ask for.
Start with the numbers from this lesson. At 5,000 events a second, about 30 alerts an hour reach the queue. Three analysts clearing about 20 alerts an hour each clear 60 an hour, so the team has twice the capacity it needs, and triage wait is about half a minute. The pipeline at the defaults costs about 85 ms per event. The hour breaks down like this: detection in under a minute from the first bad event, triage in under a minute, investigation in about 15 minutes, containment in about 1.5 minutes with the SOAR playbook. That is under 20 minutes end to end, with 40 minutes of headroom. The headroom is the point. On a bad night, when a noisy rule doubles the alerts to 60 an hour, the team is exactly at capacity and triage wait grows. So I add one rule to the design: no rule ships without a replay test against a week of old logs, and any rule that fires more than 10 times a day gets tuned or turned into a hunt. I also size the SIEM: 5,000 events a second is about 430 million events a day, so I keep seven days hot and the rest cold, and the rules run against the hot store only. Little's Law for the queue: 30 alerts an hour arriving, each taking about 3 minutes of analyst time, means about 1.5 alerts being worked at any moment, so three analysts is comfortable and one analyst on a quiet night is survivable.
Auto-containment buys speed and costs control. I automate the token disable and the session kill, and I keep a human on anything that isolates a pod or blocks an IP range, because an automated containment that hits the wrong target is an outage I caused. Centralizing all logs in one SIEM buys one place to search and costs a single bill and a single point of failure, so I keep a cold copy in cheap storage. A broad rule buys coverage and costs noise, and a narrow rule buys quiet and costs misses, so I run the broad version as a weekly hunt and the narrow version as a live alert. And the one-hour promise is measured from the first bad event, not from the alert, because a promise that starts when the alarm rings is a promise that ignores the three days nobody heard anything.
Saved in this browser, so the Lessons index can show your progress across the course.