I have watched a database restart after a crash and I have watched reads go slow for no obvious reason. Both times the answer was in the storage layer, not in the app. This lesson follows one write from the moment it arrives to the moment it is safe on disk, then follows a read back to find it. Once you see the write-ahead log, the memtable, the flush, and compaction as one path, the failure modes stop looking mysterious.
Difficulty: Advanced · Know this first: Lesson 09, Lesson 02
The other family, kept here as a note and not a hop: a B-tree updates in place instead. It stores keys in a balanced tree of pages on disk, and a read walks the tree to one page in about 6 ms, with no pile of files to search. The trade is on the write side: an in-place update means a random disk write, and a full page can split in two. Fast reads, costlier writes, no compaction queue. LSM is write optimized. B-tree is read optimized. The access pattern picks the family.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest. One thing to read carefully: the total here is the sum of the lesson hops, not the latency of one request. The write path itself is only a few milliseconds. Compaction is background work, so its 120 ms sits in the total as the cost of keeping the files tidy, not as time the user waits. Step through the lesson or switch the scenario and watch the bar change. In the crash scenario the extra time is the WAL replay. In the compaction scenario the extra time is read amplification plus a compaction job that cannot catch up.
Extra diagram · from scratch to master
Start with the simplest durable design imaginable, a write straight to a file. Add one layer at a time until the full engine is drawn.
Extra diagram · from scratch to master
Start from the healthy engine with fast reads. Then a bulk import starts, and each step adds to the pile until reads are twenty times slower.
The budget above is one fixed load. Move these three and the load changes. Watch what happens to the flush hop, the read hop, and compaction. The numbers are a teaching model, not a benchmark.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one write from arrival to safe on disk, then follows a read back to find it. This section covers the same path four times, from the first WAL entry to the level where I run sharded LSM trees and keep compaction ahead of the writes.
Definition: A write lands in the write-ahead log first. The log is written and fsynced, and only then does the ack go out. The memtable is the fast copy in memory, and memory forgets. It is a fast copy, not the safe copy. When the memtable fills up it flushes to disk as an SSTable, and a read checks the memtable first, then the files.
Types:
Uses: I use this level every time a write goes missing after a crash. I check the WAL first. If the write never reached the log, it was never durable. If it reached the log, replay brings it back.
Dimensions:
Definition: LSM makes writes cheap by appending, and compaction pays that debt back later. A B-tree updates in place instead. It stores keys in a balanced tree of pages, and a read walks the tree to one page in about 6 ms, with no pile of files to search. LSM is write optimized. B-tree is read optimized. The access pattern picks the family.
Types:
Uses: I use this level when I pick an engine for a new workload. I pick LSM when writes dominate and reads can tolerate a little extra work. I pick B-tree when reads dominate and the keys are looked up one at a time.
Dimensions:
Definition: Memtable size is a real tuning knob, not a magic number. I start at 64 MB and I change it only when the flush metrics tell me to. A small memtable flushes often and makes many small files. A big memtable flushes rarely and makes fewer, bigger files, but it eats RAM and replay after a crash takes longer. Compaction has its own choice. Leveled keeps the file count low. Tiered rewrites less and leaves more files lying around. Size-tiered is the middle ground I reach for first.
Types:
Uses: I use this level when the flush rate or the file count moves. If flushes land every few seconds, the memtable is too small or the burst is too sharp. If replay after a restart takes minutes, the memtable was too big or the WAL was never trimmed.
Dimensions:
Definition: At 100x, one node stops being the answer. I shard by key range or by hash, and each shard gets its own LSM tree, its own WAL, its own compaction queue. Compaction lag on one shard stops being a whole-system outage and becomes a shard problem. That is the point of sharding here. It is not about total size. It is about keeping each compaction queue small enough to keep up.
Types:
Uses: I use this level when one node's compaction queue cannot keep up with the writes. I stop the import first, then tune compaction. That order matters. I learned it the hard way. You cannot merge files faster than new ones arrive.
Dimensions:
WHAT I WOULD ARGUE ABOUT IN A DESIGN REVIEW
I pick LSM when writes dominate and reads can tolerate a little extra work. I pick B-tree when reads dominate and the keys are looked up one at a time. That is the first call. The second call is the WAL, and it is the one that bites people. Fsync on every write is the safest setting and the slowest. Fsync once per second is faster and loses up to a second of acked writes in a crash. No fsync at all is fast until the power goes out, and then it is a resume-generating event. I have never seen a team regret fsync on every write for data that matters. I have seen teams regret the other two.
Memtable size is a real tuning knob, not a magic number. A small memtable flushes often and makes many small files. A big memtable flushes rarely and makes fewer, bigger files, but it eats RAM and replay after a crash takes longer because there is more WAL to read back. I start at 64 MB and I change it only when the flush metrics tell me to. If flushes land every few seconds, the memtable is too small or the burst is too sharp. If replay after a restart takes minutes, the memtable was too big or the WAL was never trimmed.
Compaction has its own trade-offs. Leveled compaction keeps the file count low and reads fast, but it rewrites data many times and that write amplification wears disks and steals I/O. Tiered compaction rewrites less, but it leaves more files lying around and reads pay for it. Size-tiered is the middle ground I reach for first. There is no setting that makes compaction free. There is only a choice about who pays, reads or writes, and when.
At 10x the write rate, the WAL disk becomes the bottleneck before anything else. I put the WAL on its own fast disk, separate from the data files, so a flush storm cannot starve the log. At 100x, one node stops being the answer. I shard by key range or by hash, and each shard gets its own LSM tree, its own WAL, its own compaction queue. Compaction lag on one shard stops being a whole-system outage and becomes a shard problem. That is the point of sharding here. It is not about total size. It is about keeping each compaction queue small enough to keep up.
Two failure modes I watch for that the steps above do not show. First, tombstones. A delete in LSM does not remove data. It writes a marker that says this key is gone, and the old value sits in an old file until compaction reaches it. If compaction never reaches that level, deleted data stays on disk and reads still pay to skip it. Second, bloom filter memory. A bloom filter per file is cheap until there are hundreds of files. Then the filters themselves eat the RAM that the memtable and the cache wanted. When the file count climbs, I check filter memory before I add RAM to the box.
# storage engine settings I would start with, then tune from metrics wal: fsync: every_write # ack only after the WAL entry is durable wal_disk: separate_fast_disk memtable: size_mb: 64 # flush when full; smaller means more files flush_when_full: true compaction: strategy: leveled # low file count, fast reads, more rewrite I/O max_files_per_level: 10 # alert me if a level holds more than this threads: 2 # compaction must never starve reads bloom_filter: bits_per_key: 10 # about 1% false positives; more bits, more RAM
Reads on one database went slow overnight, from about 10 ms to over 200 ms, while writes stayed fast. Nothing had deployed. The app was fine. The disks were fine. The database was doing exactly what it was told to do, and that was the problem.
Three things, and all three are still in place in my head as a checklist. First, a dashboard panel for SSTable count and compaction lag, next to read latency, so the cause and the symptom sit on the same screen. Second, an alert on file count, not just on latency: if uncompacted files pass 50, someone hears about it before the reads go slow. Third, every bulk import now runs throttled by default, and the runbook says stop the import first, then tune compaction. I had been tuning compaction first. That was the wrong order. You cannot merge files faster than new ones arrive.
You are building the storage for a team task app. It stores task records and an activity feed. The load is 5,000 writes per second at peak, mostly new tasks and status updates. Reads are 20,000 per second, and 90% of them ask for tasks updated in the last hour. Data must survive a crash without losing an acked write. You have one database to start with. Which storage engine family do you pick, LSM or B-tree, and what do you watch in the first month?
Clarifying questions I would ask first: How big is a task record, roughly? How long do we keep activity feed entries, days or years? Are reads by task ID only, or do we scan by user and by status? Is a lost acked write a support ticket or a data loss incident? The answers change the sizing, but they do not change the family pick here.
Rough sizing: 5,000 writes per second is 432 million writes a day. At about 500 bytes per record that is roughly 200 GB of new data a day before compaction, and far less after old versions merge away. That write rate rules out a design that pays a random disk write per update. Reads are 20,000 per second, but 90% hit recent data, so the memtable and a small cache cover most of them.
My pick: LSM. The write rate is the deciding factor. Appends are cheap, the WAL gives me the durability promise, and the hot reads land in the memtable. I would size the memtable at 64 MB to start, fsync the WAL on every write, and run leveled compaction so the read path stays short. I would put the WAL on its own disk.
Trade-offs to name out loud: I am buying cheap writes with a compaction bill. If compaction falls behind, the fast reads I promised get slow, exactly like the lesson scenario. A B-tree would give me steadier reads and no compaction queue, but every status update would pay a random write and a possible page split at 5,000 writes per second. I would also name the tombstone problem: status updates leave old versions behind, so disk usage grows until compaction catches up, and deletes need compaction to actually free space.
What I would watch in the first month: SSTable count and compaction lag on the same dashboard as read p99, flush rate per minute, WAL fsync latency, and disk usage growth per day. If the file count climbs past 50, I throttle the writers before I tune compaction. That order matters. I learned it the hard way.
Saved in this browser only. Clearing your browser data clears it too.