I have watched a database grow past what one machine can hold. The fix was not a bigger machine. It was cutting the data into pieces and teaching every request how to find its piece. This lesson follows one request to the right shard, and one query that has to ask every shard, so you can see what sharding buys and what it charges.
By Mohankrishna PodilePlatform Engineering & SRE · 13+ yearsReading time · ~8 minutes, interactive
Difficulty: Intermediate · Know this first: Lesson 09, Lesson 12
Run it as:
your request
When it breaks, I check
One thing to remember
Where the time goes
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in the normal case the fan-out and the merge cost more than every single-shard hop combined, and in the bad scenarios one hop eats almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The lesson is not that sharding is fast. It is that a query with its shard key is fast, and a query without one pays for every shard it touches.
Move the dials, watch the budget
The numbers above are one worked example. Move the dials and the example moves with you. This is the fastest way I know to learn the trade-off. More shards make one lookup faster and one fan-out slower, and the budget bar shows you which one wins.
More shards mean smaller indexes, so a single-shard hop gets faster. A fan-out has more shards to ask and more answers to merge.
Up to 50% the hot shard copes. Above that, every extra point adds 12 ms to the read and 11 ms to the write.
Every scatter query per minute adds 2 ms to the fan-out hop and 1 ms to the merge hop.
32 shards, other dials at default: read ~5 ms, write ~6 ms, fan-out 1,180 ms, merge 180 ms, total ~1,398 ms. The single-shard hops get faster. The fan-out eats the saving and then some.
2 shards, other dials at default: read 72 ms, write 88 ms, fan-out 130 ms, merge 30 ms, total ~347 ms. Fewer shards, cheaper fan-out, slower single-shard hops.
Hot-key share 90%, other dials at default: read 498 ms, write 462 ms, total ~1,387 ms. One shard doing the work of eight costs almost three normal trips.
Scatter queries 200 per minute, other dials at default: fan-out 740 ms, merge 260 ms, total ~1,067 ms. The dashboard is now the whole budget.
Check yourself
Three questions. No score is saved anywhere — this is just you against the material.
Go deeper
The steps above are the happy shape. Production is where the shape bends. Here is what I watch once sharding is real.
The trade-offs I actually weigh
The shard key is a one-way door. You can add shards. You can split a range. You cannot change the key without moving every row to a new home. I have seen that migration take a quarter. Pick the key like it is permanent, because it nearly is.
Cross-shard joins do not exist. If a query needs rows from two shards, the application does the join in memory, or the data is copied somewhere it can be joined. Both are work you pay for on every query. Design the key so the common queries never need a join across shards.
Cross-shard transactions are worse. A transaction inside one shard is a normal database transaction. A transaction across two shards is a distributed transaction, and that is where my on-call nights come from. If the business rule truly needs two shards at once, change the design or use a saga with an outbox. Do not reach for two-phase commit because it is there.
Every shard is a database to operate. Eight shards means eight backups, eight schema migrations, eight upgrades, eight sets of alerts. The operational cost scales with the shard count, not with the data size. I count that cost before I count the performance win.
Rebalancing is a migration with live traffic. It runs in the background, it moves real rows while real users are writing, and the shard map flips only when the copy is complete. Plan the split at 70 percent disk. At 95 percent you are not planning. You are reacting.
Failure modes that have bitten me
A stale shard map. Two routers hold different versions of the map. Writes land on the old shard, reads look on the new one, and the user sees their own order vanish. The fix is boring: version the map, compare versions across routers after every rebalance, and alert on a mismatch.
A hot shard. Covered in the lesson. The fix is splitting the range, not adding hardware. Hardware on one shard just moves the bill.
The coordinator runs out of memory. A scatter query returns a million rows to show ten. The merge holds all of it. Push the filter, the sort, and the limit down to the shards first.
A shard goes down. Its slice is unavailable. Sharding splits data. It does not save it. That is what replication is for (Lesson 12). In production you usually need both: shards for room, replicas for safety.
What changes at 10x and 100x
At 10x (8 shards to 80): the shard map becomes a service of its own, not a config file. Every shard gets its own dashboard for disk, latency, and growth rate. Rebalancing stops being a manual job and becomes a scheduled one. Schema migrations get a rollout plan, because migrating 80 shards by hand is not a plan.
At 100x (80 shards to 800): adding a shard must not move half the data. That is where consistent hashing comes in (Lesson 14). Secondary access patterns get lookup tables instead of scatter-gather. The reporting dashboards stop fanning out entirely and read from a pre-aggregated store built by a background job. And the hot keys get special handling: a hot key range is split on its own, or the hot keys are cached in front of the shards.
Code: a shard-key routing sketch
def shard_for(user_id, shard_count=8):
# Hash the key, take the remainder, look up the shard.
# The app never hard-codes a shard. It asks this function.
return hash(user_id) % shard_count
def get_orders(user_id):
shard = shard_for(user_id)
return db[shard].query(
"SELECT * FROM orders WHERE user_id = ?", user_id)
def total_orders_today():
# No shard key in this query, so it fans out.
# Log every call here. If this fires often, the dashboard
# needs a key it can route on, or a pre-aggregated table.
log("scatter-gather: no shard key")
partials = [
db[s].query("SELECT COUNT(*) FROM orders WHERE day = TODAY")
for s in range(8)
]
return sum(partials)
One time it broke
What broke
One Friday evening, a bulk import job started loading a single customer's backlog. Every row carried the same user id. Every write landed on shard 3. Disk on shard 3 went from 70 percent to 95 percent in under an hour. Reads for every customer on that shard slowed to half a second. The other seven shards sat idle. Nothing had failed. One shard was simply doing the work of eight, and the import job was the reason.
Timeline
6:12 PMThe disk alert fires on shard 3 at 90 percent. The alert goes to the on-call phone. I am already off the clock.
6:40 PMp99 read latency on shard 3 crosses 500 ms. Customer lookups start timing out. Clients retry, and the retries double the write load.
7:05 PMThe retry storm is now the biggest source of writes on the shard. The import job is still running. Nobody has connected the two yet.
7:20 PMI open the per-shard dashboard. One shard is red. Seven are green. I kill the import job first, then I look at the data.
7:50 PMThe import is stopped, the retries drain, latency falls back under 50 ms. The shard is still at 95 percent disk. The load is gone. The data is not.
Next Tuesday afternoonWe split the hot range and move half of it to a new shard, in daylight, with the runbook, while traffic is normal.
What changed after
Per-range growth alerts now fire at 70 percent disk, not 90. Ninety percent is a deadline, not a warning.
Bulk import jobs must carry a spread-out shard key and a rate limit, and they run in the afternoon, never on a Friday evening.
The rebalance runbook has a named owner, and we test it once a quarter on a quiet shard, so the first time we run it is not the time we need it.
The dashboard now shows disk and latency per shard on one screen, not per database. One red shard next to seven green ones is a picture you cannot miss.
If you remember nothing else
The shard key is the whole design. Pick it by the queries you actually run — high cardinality, in nearly every query, never changing.
A query with the shard key touches one shard. A query without it touches all of them, and you pay for the slowest one plus the merge.
A hot shard is a distribution failure, not a capacity failure. Adding hardware to one shard just moves the bill.
Rebalancing is a migration with live traffic. Plan the split before the shard fills up, not after it hits 95 percent disk.
Sharding buys scale and charges complexity: cross-shard queries, cross-shard transactions, and rebalancing are the price. Pay it only when one database genuinely cannot hold the data.
Design exercise
Prompt: You are designing storage for a photo-sharing app. It has 100,000 users today and will reach 10 million in two years. Users upload photos, view their own photos, and follow other users. The feed shows photos from people you follow, newest first. Design the storage. How do you shard it?
Clarifying questions to ask
What does a read look like: one user's photos, or a feed across many users? Those two reads want different shard keys.
How many photos does a user upload per day, and how big is a photo?
What is the read to write ratio? Feeds are read-heavy. Uploads are not.
Do likes and comments live with the photo, or in their own store?
How fresh must the feed be? Seconds, or is a minute fine?
What happens when one user goes viral and gains five million followers overnight?
See a worked approach
Photos themselves go to object storage, not the database. The database holds metadata only: photo id, owner, timestamp, caption, a pointer to the object.
Shard the metadata by owner user id. Every upload and every profile read is single-shard, because the owner is the key and the query always carries it.
Rough sizing: 10 million users, 100 photos each over two years, 2 MB per photo. That is 1 billion photos and about 2 PB of photo bytes in object storage. Metadata is 1 billion rows at roughly 500 bytes each, about 500 GB. Across 16 shards that is about 31 GB of metadata per shard. The database is not the big number. The object store is.
The feed is the hard part, because it crosses shards: every followee lives somewhere else. Do not fan out on every feed load. Fan out on write instead. When a user posts, push the photo id into each follower's feed cache. A feed read then hits one cache, not sixteen shards.
The hot key is the celebrity with five million followers. Their post fans out to five million feed caches. Cap it: accounts above a follower threshold get a pull feed instead of a push feed. Their followers read the celebrity's shard directly when they open the app.
Trade-offs to name: shard key user id versus photo id (user id wins, because the reads are by owner); push versus pull for the feed (push is fast to read and expensive to write, pull is the reverse); rebalancing when a shard fills (split the range, move half, same as this lesson); global search and trending cannot scatter-gather on every request, so they get a separate search index, not a fan-out.