Replication Lag and the Illusion of Consistency

Add a read replica and throughput doubles. Add a read replica and a class of bug appears that only reproduces under load, on someone else's machine, once.

The bug is replication lag, and the reason it's hard to reason about is that "eventually consistent" collapses several distinct guarantees into one phrase.

Read-your-writes

A user posts a comment, the write goes to the leader, the page reloads and reads from a replica that hasn't caught up. The comment is gone. They post it again.

The guarantee needed here is narrow: a user must see their own writes. Not everyone's writes -- their own. That's cheap to implement: route reads to the leader for a short window after that user writes, or track the write timestamp per session and pick a replica that's caught up past it.

Monotonic reads

A user refreshes twice and the second load shows less data than the first, because the two reads hit replicas at different lag.

Time appears to run backwards. This one is easy to miss in testing because it needs two replicas at different offsets, which local dev never has.

The fix is also narrow: pin a user to one replica, usually by hashing the user ID. They may see stale data, but never data that goes backwards.

Consistent prefix reads

Across partitions, causally ordered writes can arrive out of order:

Actual:     A: "How far is it?"   ->   B: "About 20 minutes"
Observed:   B: "About 20 minutes" ->   A: "How far is it?"

The answer arrives before the question. Each partition is internally consistent; the causal relationship crosses partitions and nothing preserved it.

The point

These are three different problems with three different fixes. "We use eventual consistency" tells you which of them you have -- none of them, or all three -- and reaching for a stronger consistency level than you need buys latency you didn't have to spend.

Name the specific guarantee the feature requires. Most features need one of these, not all of them, and almost none need linearizability.