Platform Engineering · Lesson 32

Common Attacks and Defenses

I have watched a bad request walk straight through a system that looked secure on a diagram. It was not one big failure. It was a small gap at every layer. This lesson follows one malicious request through the same path a real checkout takes. You will see where it gets stopped, where it costs us time even when it fails, and why the defense that matters is the one that fails closed.

Difficulty: Advanced · Know this first: Lesson 18, Lesson 29

Run it as:

When it breaks, I check

One thing to remember

Extra diagram · Architecture map

The fence, layer by layer

One request, one fence, the WAF. I add the rate limiter, validation, and auth, then the last line and the closed back door.

Layers: WAF, limit, validation, authLast line: parameterized queryBypass: direct path closed
logslogsclosedAttacker / usergood or craftedWAFfirst fenceRate limitertoo many, too fastInput validationgateway + serviceAuth servicewho are youAppdoes the workDatabaseparameterized queryAudit log sinkfed by every layerDirect-to-servicepathclosed
the request
the requestblockedpassed the layeraudit sinkrequest · flowingresponse · the way back

    Extra diagram · Second trip

    The stuffing burst

    One bad login is a 401. A burst from one IP is caught by the limit. I follow the burst as it spreads across hundreds of IPs.

    Pairs: thousands of triesPer-IP limit: blind to the spreadFix: per-account limit + bot checks
    Stolen pairs triedfrom one IPTries spreadacross hundreds ofIPsPer-IP limit nevertripsAuth CPU climbsslow hash on everytryReal users cannotlog inAccount lockoutlocks the realaccountsPer-account limit +bot checksstop the burst
    the stolen pairs
    stolen pairsburst / lockoutdefence holdsrequest · flowingresponse · the way back

      Where the time goes

      Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system. The shape is the lesson. In the normal checkout, auth and the database do the real work. In the injection attempt, the trip ends early and cheap. In the stuffing burst, auth and logging eat the budget while the app sits idle.

      Try the burst yourself

      Move the sliders and watch what a credential-stuffing burst does to one login endpoint. The numbers are a teaching model, not live measurements. The shape is what matters: a small change in block rate moves thousands of requests.

      How hard the botnet knocks, every second.
      Share of attack requests the WAF stops at the edge.
      Requests per minute allowed through to auth.
      60,000
      Attack requests per minute
      57,000
      Blocked by WAF per minute
      3,000
      Passed by WAF per minute
      2,900
      Blocked by rate limit per minute
      100
      Requests reaching auth per minute
      1,140
      False positives per minute — real users caught by the WAF
      120 ms
      Auth time per request under this load
      40 ms
      Audit log time per request under this load
      244 ms
      Added latency per request in a burst

      At the defaults, the added latency is 244 ms — the same total as the credential-stuffing scenario above. Switch to that scenario and move the sliders: the auth and audit hops, the budget bar, and the totals follow.

      Check yourself

      Three questions. No score is saved anywhere — this is just you against the material.

      Basic to master

      The same topic, four times. One malicious request, from the first fence to the level where I can run a login endpoint through a burst at 2 AM and trust it.

      Basic — The bad request and the fence

      Definition: An attack is usually an ordinary request with malicious intent, not a break-in through a back door. A bad request walks the same path a real checkout takes. Defense in depth means several checks in a row, each one catching what the last one missed, so a single gap does not open the whole system.

      Types:

      • SQL injection, where input is glued into a query and becomes code instead of data.
      • Credential stuffing, where stolen username and password pairs from an old breach at another site are tried here.
      • Bad input in general, values that are too long, the wrong shape, or aimed at a field that was never meant to take them.

      Uses: I use this level on any public login or checkout endpoint. The first question I ask is where the request gets stopped, and the second is what it costs me even when it fails, because a blocked attack still costs log volume and attention.

      Dimensions: As load grows, a web application firewall at the edge blocks most of the noise before it reaches the app. What breaks is the pattern matcher: it stops the attacks it recognizes and misses a new one, or one split across two fields, or encoded twice. What it costs is false positives, real users blocked by a strict rule. What I give up is the idea of one perfect filter; there is no setting that gives me zero false positives and zero missed attacks.

      Intermediate — Defense at the service, not just the edge

      Definition: The real defense sits deeper than the fence. Parameterized queries keep input as data, never as SQL, even when the firewall misses. Validation at the service stops a bad value even when the gateway is bypassed. Auth fails closed, so a login is refused when the counter store is down, and I accept that real users get blocked in that minute, because the alternative is an open door.

      Types:

      • Parameterized queries, with placeholders instead of string-built SQL.
      • Input validation and length checks at the service, before the database sees the value.
      • Rate limiting per IP and per account, both, because one count alone lies.
      • Identical error messages for a missing user and a wrong password, so the answer never leaks which one it was.

      Uses: I use this level on the login endpoint in this lesson. A per-IP limit alone fails when the attacker spreads tries across hundreds of IPs. A per-account limit alone fails when the attacker sprays one try across many accounts. Both together catch both shapes.

      Dimensions: As load grows, the rate limiter becomes a system of its own, and I shard counters by IP hash and accept that a count can be slightly wrong at the edge of a window. What breaks is the counter store going down, and then the fail-closed choice blocks real users. What it costs is auth CPU and audit log volume. What I give up is the friendly login that never says no; the safe answer when a check is down is no, not maybe.

      Advanced — The burst as a system problem

      Definition: At this level an attack is load, and I treat it like load. The slow password hash sits behind the rate limit, so a blocked try never costs a hash. The audit log sits behind a queue with sampling for routine failures, and I keep every failure on a targeted account. The auth service gets a circuit breaker, so a flood degrades to a plain 429 instead of taking login down for everyone.

      Types:

      • A CAPTCHA or a short delay after a few failures on one account, not a hard lockout.
      • An auth circuit breaker that sheds login load before it sheds real users.
      • Sampled audit logging for routine events, with every blocked attack and every failed login on a targeted account kept.
      • Failed-login alerts grouped by account, so one targeted account stands out from the noise.

      Uses: I use this level in the incident in this lesson, a credential-stuffing burst at 2 AM. The rate limit was per IP, the attacker spread tries across hundreds of IPs, auth CPU climbed while the app sat idle, and then the account lockout kicked in and locked the real accounts the attacker had guessed, which locked out the exact users who needed help most. The damage was availability and noise, not a breach.

      Dimensions: As load grows, the audit log becomes the bottleneck, and I sample routine events but never the events that prove what an attacker did. What breaks is the lockout rule itself: a hard lockout hands the attacker a way to lock out any user they name. What it costs is log volume, CAPTCHA friction, and on-call attention. What I give up is the clean story where blocking the attack ends the incident; the noise keeps coming after the block.

      Master — Running defense when the attacker adapts

      Definition: At this level I assume the attacker adapts. New payloads, distributed IPs, patterns the firewall has never seen. Defense is measured, drilled, and reviewed. A lockout triggers a reset flow for the real owner instead of a silent block. The runbook has the line I wish I had the first night: if auth CPU is high and the app is idle, look at login tries first.

      Types:

      • Detection that groups failed logins by account, not just by volume.
      • Runbooks for the burst shape: what to tighten, in what order, and what not to touch.
      • Lockout as a reset flow, so the real owner gets back in instead of being punished for being targeted.
      • Post-incident follow-ups with names and dates, like fixing the lockout rule the same week.

      Uses: I use this level when the burst is still running and I have to choose under pressure. Tighten the login rate limit per account, not just per IP. Add a short delay after three failures on one account. Confirm no account was taken over before I call it a win. The trade-off I name out loud is false positives against missed attacks, and I pick the setting that fails in the direction I can see: a blocked real user files a ticket and I learn about it, and a missed attack files nothing.

      Dimensions: As load grows, defense spend grows with attack volume, not user volume, and the budget for it is attention as much as money. What breaks is the miss I never see, because a missed attack files nothing and a blocked user files a ticket. What it costs is the on-call time to watch the noise, the tickets from false positives, and the review after every burst. What I give up is zero-friction login, and I get it back as accounts that stay safe while the burst is still knocking.

      Go deeper

      A WAF is a pattern matcher. It stops the attacks it recognizes. It does not stop a new attack, and it does not stop an attacker who splits a payload across two fields or encodes it twice. I treat the WAF as the outer fence. The real defense is deeper. Parameterized queries stop injection even when the WAF misses. Validation at the service stops a bad value even when the gateway is bypassed. Auth that fails closed stops a login when the counter store is down, and I accept that real users get blocked in that minute, because the alternative is an open door.

      At 10x scale, the rate limiter becomes a system of its own. One counter store will not hold. I shard counters by IP hash and I accept that a count can be slightly wrong at the edge of a window. A slightly wrong count is fine. A down limiter is not. At 100x scale, the audit log becomes the bottleneck. I sample routine events and I keep every blocked attack and every failed login. I never sample the events that prove what an attacker did.

      The trade-off I name out loud is false positives against missed attacks. A strict WAF blocks real users. A loose WAF lets attacks through. There is no setting that gives me zero of both. I pick the setting that fails in the direction I can see. A blocked real user files a ticket and I learn about it. A missed attack files nothing.

      # Bad pattern, never do this: build SQL by gluing user input into the text.
      # query = "SELECT * FROM users WHERE email = '" + email + "'"
      
      from flask import request, jsonify
      from flask_limiter import Limiter
      from flask_limiter.util import get_remote_address
      import psycopg2
      
      limiter = Limiter(key_func=get_remote_address, default_limits=[])
      
      @limiter.limit("5 per minute")
      def login():
          email = request.form.get("email", "")
          password = request.form.get("password", "")
          if not email or len(email) > 254 or not password:
              return jsonify({"error": "Login failed."}), 401
          conn = psycopg2.connect("dbname=shop user=app")
          cur = conn.cursor()
          cur.execute("SELECT id, password_hash FROM users WHERE email = %s", (email,))
          row = cur.fetchone()
          cur.close()
          conn.close()
          if row is None:
              return jsonify({"error": "Login failed."}), 401
          if not check_password(password, row[1]):
              return jsonify({"error": "Login failed."}), 401
          return jsonify({"ok": True, "user_id": row[0]})

      The snippet shows three defenses in one place. The rate limit decorator caps tries per IP. The length check rejects absurd input before the database sees it. The %s placeholder keeps the email as data, never as SQL. The error message is identical for a missing user and a wrong password.

      One time it broke

      A public login endpoint got hit by a credential-stuffing burst at 2 AM. The attacker tried stolen username and password pairs from an old breach at another site. The rate limit was per IP, and the attacker spread the tries across hundreds of IPs, so no single IP tripped it. Auth CPU climbed. Real users could not log in. Then the account lockout kicked in, and it locked the real accounts the attacker had guessed correctly, which locked out the exact users who needed help most.

      Timeline

      • 02:04Failed logins climb from a normal handful per minute to over 2,000 per minute. The alert fires on volume, not on a breach.
      • 02:11I open the dashboard. Auth CPU is high. The app is idle. The database is warm. The problem is login tries, not checkouts.
      • 02:19I check failed logins per account. A few hundred accounts have ten or more failures each. Those are the targets.
      • 02:26I tighten the login rate limit per account, not just per IP, and I add a short delay after three failures on one account.
      • 02:41The burst continues, but tries reaching auth fall sharply. Real logins start working again.
      • 03:05I confirm no account was taken over. The attacker never had a valid pair for our system that night. The damage was availability and noise, not a breach.
      • 09:30I write the follow-up. The lockout rule gets a fix the same week.

      What changed after

      The login rate limit is now per account and per IP, both. A lockout now triggers a reset flow for the real owner instead of a silent block. Failed login alerts group by account so one targeted account stands out from the noise. The auth service has a circuit breaker so a flood degrades to a plain 429 instead of taking down login for everyone. I also added a runbook line I wish I had that night: if auth CPU is high and the app is idle, look at login tries first.

      If you remember nothing else

      Design exercise

      The prompt: Defend a public login endpoint against credential stuffing without locking out real users. The endpoint serves a shop with 200,000 users. Attackers try stolen username and password pairs from other breaches. Real users log in from shared office networks, so many of them share one IP.

      Clarifying questions to ask

      • How many login tries per minute is normal at peak, and how many would count as a burst?
      • Are the stolen pairs aimed at a few accounts or spread across many?
      • What does a real user do after three failed tries: retry, reset the password, or call support?
      • Can we add a second factor, a CAPTCHA, or a delay, and where does the product team draw the line on friction?
      • What happens when the counter store is down: fail open or fail closed?

      Worked approach

      Start with the shape of the traffic. Count tries per account and per IP separately, because one count alone lies. A per-IP limit alone fails when the attacker spreads across IPs. A per-account limit alone fails when the attacker sprays one try across many accounts. Use both. Set the per-account limit low, like 5 failures in 10 minutes, and make the penalty a short delay plus a CAPTCHA, not a hard lockout. A hard lockout hands the attacker a way to lock out any user they name. Set the per-IP limit higher, to catch a single source without punishing a shared office network.

      Put the slow password hash behind the rate limit, so a blocked try never costs a hash. Put the audit log behind a queue with sampling for routine failures, and keep every failure on a targeted account. Fail closed when the counter store is down: answer 429 and log it, instead of letting the flood through. Size it roughly: if normal peak is 200 logins per minute and a burst is 2,000 per minute, the rate limit has to shed 90 percent of tries before auth, and auth needs headroom for the 200 real ones plus the few tries that pass the gate.

      Trade-offs to name

      A CAPTCHA stops bots and annoys real users, so use it only after failures, not on the first try. Per-account limits protect accounts and risk locking out a real user whose name is being tried, so pair the limit with an easy reset flow. Failing closed protects the system and blocks real users during an outage, so say that choice out loud and get the product team to agree to it before the incident, not during it.

      Saved in this browser, so the Lessons index can show your progress.
      Next in this series
      Lesson 33 — From Commit to Running Code