Every database I have run has the same shape of problem. A few questions get asked thousands of times a minute. The rest barely get asked at all. A cache is a shelf next to the app where I keep the answers to the questions people keep asking. It is fast because it is just memory, and it is dangerous for the same reason: memory forgets, memory goes stale, and memory can empty itself at the worst possible minute. This lesson follows one read and one write through that shelf. The incidents in it are mine. One of them took a service down.
Difficulty: Advanced · Know this first: Lesson 01, Lesson 02
The bar below is a worked example for this lesson's scenario — not a measurement from a live system. The shape is the honest part: in the normal case the reads are nearly free and the write does the heavy lifting, and in a stampede the database hop eats almost the whole bar. Switch the scenario and watch the bar change.
Extra diagram · from scratch to master
Start with the app talking straight to the database and no shelf at all. Add the shelf, then the layers, then the second writer.
Extra diagram · from scratch to master
Start from the calm shelf with a single reader missing alone. Then the hot key expires with ten readers waiting, and they all run at once.
The scenario buttons show you three fixed days. These sliders let you make your own day. Move one dial and watch what the database has to do.
The write steps stay at their normal values here. This simulator is about the read stampede. The full stampede scenario slows the write too.
Three questions. No score is saved anywhere — this is just you against the material.
I take the same shelf four times here. First I learn what a hit and a miss cost. Then I learn when an answer is allowed to be old. Then I learn what happens when the shelf empties at once. Last I learn to run the shelf as a stack.
Definition: A few questions get asked thousands of times a minute. The rest barely get asked at all. A cache is a shelf next to the app where I keep the answers to the questions people keep asking. It is fast because it is just memory. A hit costs milliseconds. A miss costs a database trip. The cache is a shield for the database, not a fix for slow queries.
Types:
Uses: I use this when the database is doing the same work over and over. I put the hot answers on the shelf and the database only sees the misses. A product page that reads the same product a thousand times a minute is the first candidate.
Dimensions: At low traffic the shelf is quiet and the hit ratio sits above 90 percent. What breaks is the first hot key expiring with a crowd waiting: every reader misses at once and the database takes the whole crowd. I notice it in the hit ratio falling off a cliff and database connections climbing in the same minute. It costs memory and one more thing to watch. The trade is that the shelf buys speed and costs freshness: the answer on the shelf is always a little old.
Definition: A TTL is a promise that an answer goes stale on a schedule. Invalidation is a promise that an answer goes away the moment the truth changes. I use TTLs for data that can be a little old, like a product description. I use invalidation for data that cannot be wrong, like a price. When I need both, I set a short TTL and invalidate on write, so the TTL is the safety net for the invalidation that fails quietly.
Types:
Uses: I use this when I design a product page. I key by product id, one key for price and stock together so they cannot disagree. I serve stock from the cache but I re-check stock at checkout against the database, because stock is the number that changes fastest and the one that causes overselling.
Dimensions: As traffic grows, the hot set stays small: if the top 1,000 products take 80 percent of traffic, that is 4,000 reads a second on 1,000 keys. What breaks is invalidation that deletes the wrong key, so readers keep getting the old answer and nobody notices for a while. I notice it when the key I deleted is not the exact key the readers use. It costs care in key design and a rule: never let raw user input become a key without normalising it first. The trade is that write-through is slower and safe, and write-behind is faster and a bet: if the cache box dies before the flush, the write is gone.
Definition: The failure modes are not exotic. A hot key expires with a crowd waiting and the crowd stampedes the database. A deploy flushes the whole cache and every page misses at once. A cache that fills up evicts the warm data that was about to be hot again. A cache that returns an error is not the same as a cache that returns a miss: if my app treats an error like a miss, every request falls through to the database at once, which is a stampede with a different trigger.
Types:
Uses: I use this after a deploy flushed the whole cache at noon and ten thousand reads ran to the database in the same minute. The hit ratio fell from 96 percent to zero in under a minute. The fix was time: let the cache warm back up on its own instead of flushing again. Now a deploy flushes only the keys its change actually touches, and most deploys flush nothing at all.
Dimensions: At ten times the traffic one cache box stops being enough, so I shard the keys across boxes and replicate the hottest ones. What breaks is the flush nobody reviewed, and I notice it in two alerts: one on hit ratio dropping fast, and one on database connections climbing fast. Both page now. It costs locks, jitter, and a runbook line that says if the hit ratio fell off a cliff, ask what flushed the cache before touching anything else. The trade is that each protection costs a little latency or complexity, and it buys a database that survives the worst minute.
Definition: At a hundred times the traffic the cache stops being one layer and becomes a stack. The browser caches the page. The edge caches the region. The app cache holds the hot set. Each layer has its own TTL and its own invalidation, and a miss at one layer is a hit at the next. The database is the last layer, and it should almost never be reached.
Types:
Uses: I use this when I design for a catalogue of 200,000 products at 5,000 page views a second. The full catalogue is about 400 MB and the hot set is about 2 MB, so one cache box holds it easily, and the database only needs to survive a few hundred queries a second, not 5,000. I put a single-flight refill on the top 100 keys and I keep the cache read failing open.
Dimensions: At this scale a stampede at one layer becomes a stampede at the next, so every layer needs its own protection and its own alert. What breaks is a layer nobody owns, usually the browser cache with a TTL set once and forgotten, and I notice it when users see a fix everywhere except one city, or one browser. It costs a stack to operate instead of a box, and every layer needs its own key design. The trade is that the stack costs the most to reason about, and it buys a database that stays asleep while the site is busy.
I use cache-aside most of the time because the app sees the miss and decides what to do. Read-through hides the miss inside the cache, and that is fine until the cache is the thing that is slow and nobody can see why. Write-through is what I use for money. It costs time on every write and it keeps the database and the cache in agreement. Write-behind is faster and it is a bet. If the cache box dies before the flush, the write is gone. I only take that bet for data I can rebuild.
TTL and invalidation are two different tools and I do not mix them up. A TTL is a promise that an answer goes stale on a schedule. Invalidation is a promise that an answer goes away the moment the truth changes. I use TTLs for data that can be a little old, like a product description. I use invalidation for data that cannot be wrong, like a price. When I need both, I set a short TTL and invalidate on write. The TTL is the safety net for the invalidation that fails quietly.
Stale data has a middle option I reach for a lot: serve the stale copy while one request refills it in the background. The user gets a fast answer that is a minute old, and the next user gets a fresh one. That is stale-while-revalidate. It only works when a minute-old answer is acceptable. For a price at checkout, it is not.
The failure modes I watch for are not exotic. A hot key is one key that every request wants, and one cache box ends up holding all of it. A stampede is a hot key expiring with a crowd waiting. A cache that fills up evicts the wrong things, usually the warm data that was about to be hot again. A key that is built from user input can be poisoned, so I never let raw input become a key without normalising it first. And a cache that returns an error is not the same as a cache that returns a miss. If my app treats an error like a miss, every request falls through to the database at once, which is a stampede with a different trigger.
At 10x scale, one cache box stops being enough. I shard the keys across boxes so no single box holds the hot key alone, and I replicate the hottest keys so a read can be served from more than one place. I add a lock or a single-flight refill on the hottest keys, so only one request goes to the database while the rest wait for the answer. I add jitter to every TTL. I set an alert on hit ratio and on database connections, because those two graphs tell me about a stampede before the users do.
At 100x scale, the cache stops being one layer and becomes a stack. The browser caches the page. The edge caches the region. The app cache holds the hot answers. The database has its own buffer. Each layer has its own TTL and its own invalidation path, and a change has to clear the right layers in the right order. A deploy at that scale never flushes a whole layer. It rolls keys out in waves and watches the hit ratio after each wave. If the ratio drops, the rollout stops.
First, the headers I set so a browser or edge cache knows how long an answer is good for. Second, a single-flight refill sketch, so only one request rebuilds an expired hot key.
Cache headers
Cache-Control: public, max-age=300, stale-while-revalidate=60
Single-flight refill
// Only one request rebuilds the hot key. The rest wait for its answer.
function getProduct(id) {
const key = "product:" + id + ":price";
const hit = cache.get(key);
if (hit) return hit;
return singleFlight(key, () => {
const row = db.query("SELECT price FROM products WHERE id = ?", id);
cache.set(key, row, { ttlSeconds: 300 + jitter(30) });
return row;
});
}A deploy flushed the whole cache at noon. Not one key. The whole shelf. Every product page missed at once, ten thousand reads ran to the database in the same minute, and the database stopped answering in time. The site stayed up. The pages just stopped loading before the timeout. The deploy itself was fine. The code was fine. The flush was the outage.
Full cache flushes on deploy were banned. A deploy now flushes only the keys its change actually touches, and most deploys flush nothing at all. TTLs got jitter so hot keys stop expiring together. The hottest keys got a single-flight refill, so one request rebuilds the answer while the rest wait. Two alerts were added: one on cache hit ratio dropping fast, and one on database connections climbing fast. Both page now. The runbook got one line at the top: if the hit ratio fell off a cliff, ask what flushed the cache before you touch anything else.
“Design the caching for a product page. The catalogue has 200,000 products. Peak traffic is 5,000 page views a second. Prices change a few hundred times a day. Stock changes constantly. A user must never pay a different price from the one they saw on the page. Walk me through your design.”
How fresh must the price be at checkout, and how fresh must stock be on the page? What share of traffic hits the top 1,000 products? How often do prices really change, and who changes them? What happens if the cache is down: do pages fail, or do they go slow to the database? Is there a logged-in price or a discount that changes the answer per user?
5,000 reads a second against 200,000 products means the hot set is small. If the top 1,000 products take 80 percent of traffic, that is 4,000 reads a second on 1,000 keys. One product record is roughly 2 KB with price, stock, name, and image links. The full catalogue is about 400 MB. The hot set is about 2 MB. That fits in one cache box easily, and the full set fits too. The database only needs to survive the misses, so size the database for a few hundred queries a second, not 5,000.
Put the app cache in front of the database with cache-aside. Key it by product id, one key for price and stock together so they cannot disagree with each other. Set a TTL of 300 seconds with 30 seconds of jitter. Invalidate the key on every price write, and write the price through to the database first, then the cache. Serve stock from the cache but re-check stock at checkout against the database, because stock is the number that changes fastest and the one that causes overselling. Put a single-flight refill on the top 100 keys. Add a per-key lock so a hot key expiring under load is rebuilt once, not a thousand times. Keep the cache read from failing open: if the cache errors, go to the database and count it, so a cache outage is slow pages, not dead pages.
TTL against invalidation: a long TTL is calm and stale, a short TTL is fresh and busy. Price freshness against speed: checking the database at checkout costs time and buys correctness, and for money that is the right trade. One key per product against one key per field: one key keeps price and stock together, separate keys let you invalidate stock without touching price. Cache-aside against read-through: cache-aside lets the app see the miss, read-through is simpler and hides it. A single cache box against sharded boxes: one box is enough at this size, sharding only matters when the hot key itself becomes the bottleneck. Name the stampede explicitly: if the top key expires at peak, 5,000 readers miss at once unless the refill is single-flight.