Lesson 5 of 5

Designing a key-value store

10 min read Storage 3-question check Flashcards included

A distributed key-value store answers the single-node limits of capacity, throughput, availability and durability with three layers stacked on each other.

Quorum (N / W / R)
N = replicas per key; W = replicas that must acknowledge a write; R = replicas consulted on a read. Tuning these trades latency against consistency.
CAP trade-off
Under a network partition you can keep Consistency or Availability, not both. Dynamo-style stores choose availability (AP).

Layer 1: partition

Spread keys across nodes with consistent hashing (lessons 0003–0004), so capacity and throughput scale by adding machines and no single node holds everything.

Layer 2: replicate

Copy each key to N nodes, the next N distinct servers clockwise on the ring. Now a node loss costs durability nothing, and any replica can serve a read.

put(k,v) client coordinator replica 1 replica 2 replica 3
client
coordinator
replica 1
replica 2
replica 3
Write acked at W replicas, read quorum R. W + R > N ⇒ strong read.

Layer 3: choose a consistency model

Quorums make consistency a dial. With W + R > N the read and write sets overlap, so reads see the latest write. Lower W or R for faster, more available operations that may return stale data. Most large stores set this for AP: stay up during partitions, reconcile after.

Handling failure

Three mechanisms keep an AP store honest without a central coordinator.

  • Gossip protocol. Nodes swap membership and health state peer-to-peer, so no single node is the one tracking who’s alive.
  • Hinted handoff. If a target replica is down, a neighbor stores the write and forwards it when the node returns.
  • Read repair. When a read finds a stale replica, the coordinator writes the current value back to it in the background.

One write path touches every one of them.

put(k,v) W=2 W=2 timeout neighbor buffers replay on return who is alive stale detected W met client load balancer coordinator ring lookup (consistent hashing) quorum check W + R > N replica 1 replica 2 replica 3 (down) hinted handoff buffer gossip membership read repair ack to client
client
load balancer
coordinator
ring lookup(consistenthashing)
quorum checkW + R > N
replica 1
replica 2
replica 3(down)
hintedhandoffbuffer
gossipmembership
read repair
ack to client
Dashed edges are failure and background paths. Replica 3 is down: the write still meets W=2, a neighbor buffers the hint, and gossip is what told the quorum check to stop waiting.

Together these make the Dynamo-style AP design the default for session caches and shopping-cart stores, where staying available matters more than a moment of staleness.

Source: Alex Xu, System Design Interview Vol 1, Ch. 6

Check your understanding

Answer to reveal the explanation. Nothing is scored.

1During a network partition, what do DynamoDB and Cassandra choose in CAP terms?

They pick availability and partition tolerance: keep serving reads and writes and reconcile later, rather than blocking during a partition.

2With N replicas, when does W + R > N guarantee a read sees the latest write?

If the write set (W) and the read set (R) must overlap, which is what W + R > N forces, then at least one replica in every read saw the most recent write.

3A write's target node is down. What is hinted handoff?

A neighbor holds a 'hint' for the down node and replays the write once it recovers. The write stays available through the outage.

During a network partition, what do most large-scale KV stores (DynamoDB, Cassandra) choose in CAP?
AP — availability + partition tolerance; accept eventual consistency rather than blocking reads/writes.
What do N, W, and R mean in quorum-based replication?
N = total replicas per key; W = min replicas that must ack a write; R = min replicas consulted on a read.
When does W + R > N guarantee a read returns the latest write?
During normal operation — the read and write quorums overlap, so at least one replica in the read set saw the latest write.
What is read repair?
On read, if a replica is stale, the coordinator writes the latest value back to that node in the background.
What is hinted handoff?
When a target node is down, writes meant for it are stored on a neighbor; forwarded when the node returns.
What is the purpose of a gossip protocol in a distributed KV store?
Nodes exchange membership and health state without a central coordinator that could become a single point of failure.
Name the three layers of the distributed KV fix for single-node limits.
Partition data across nodes, replicate each partition for fault tolerance, choose a consistency model matching product needs.