Why Log-Structured Storage Won

Two storage engine families dominate: B-trees, which most relational databases use, and LSM-trees, which most of the newer write-heavy stores use. The split comes down to one decision.

B-trees update in place

A write locates the page holding the key and overwrites it. Reads are excellent: a lookup is a handful of page fetches, and the tree stays sorted for free, so range scans are sequential.

The cost is that every write is a random write. On spinning disks that was seek time. On SSDs it's write amplification -- a 4 KB logical write can force a much larger physical erase-and-rewrite cycle, wearing the device and burning IOPS.

LSM-trees never update in place

A write goes to an in-memory table. When that fills, it's flushed to disk as an immutable sorted file. Updates and deletes are appended as new entries; the old value is left alone and shadowed.

write -> memtable (RAM, sorted)
              |  flush when full
              v
         SSTable 1  SSTable 2  SSTable 3 ...   (immutable)
              \_________ compaction _________/

Every disk write is sequential, which is the access pattern storage hardware is fastest at. Throughput on write-heavy workloads is dramatically better.

What it costs

Reads get harder. A key might be in the memtable or any SSTable, so a read may check several files. Bloom filters make the miss case cheap, but it's still more work than a B-tree lookup.

Compaction is a permanent background tax. Merging SSTables competes with live traffic for disk bandwidth. Under sustained heavy writes, compaction can fall behind, files accumulate, and reads degrade -- a failure mode that only shows up at scale, and shows up suddenly.

Latency is less predictable. B-tree performance is boring in the good way. LSM p99 depends on what compaction is doing at that moment.

Choosing

Write-heavy with tolerable read latency -- ingestion, time-series, event logs -- LSM. Read-heavy with predictable latency and transactional semantics -- B-tree.

The useful takeaway is that neither is better. They made opposite bets about which operation deserves the sequential access pattern, and your workload decides which bet was right.