Replication means keeping the same data on more than one machine. It sounds like the easy part of system design. Copy the data, done. It is not easy, because the copies are never exactly the same at the same moment, and most of the interesting bugs come from that gap.
This note walks through why we copy data, how the copies are kept in step, what goes wrong, and what happens on the bad day when a machine dies.
Why copy data at all
There are three reasons. Interviewers like it when you name them separately, because each one leads to a different design choice.
Safety. Disks die. Machines catch fire, or more likely, someone runs the wrong command. If your only copy of the data lives on one machine, one bad afternoon ends the company. With a copy on another machine, you lose nothing.
Read scale. Most apps read far more than they write. A news site might read 100 times for every write. One machine can only answer so many reads a second, so you put copies on several machines and spread the reads over them. Writes are a different story, and we will see why they stay on one machine.
Location. Light is not fast enough. A request from Mumbai to a server in Virginia takes around 200 ms for the round trip, before any real work. A copy of the data in a Mumbai data centre can answer in a few milliseconds. Copies near users also mean one region going down does not take everyone with it.
Leader and follower
The most common design is the simplest one.
One machine is the leader (also called primary or master). All writes go to it. The other machines are followers (replicas or secondaries). They receive every change from the leader and apply it in the same order. Reads can go to any of them.
Think of a school notice board. The headmaster is the only one who may write a new notice. Each class has a copy of the board in its room. When a new notice goes up, a peon walks around and updates each classroom's copy. Students read from the board in their own room.
The peon in this story is the replication log. The leader writes every change to an ordered log, and the followers read that log and replay it. Because everyone replays the same changes in the same order, they end up with the same data.
Why only one leader for writes? Because it keeps life simple. If two machines could both accept a write to the same row at the same time, they could disagree, and you would have to sort out the fight. With one leader there is one order of events, and no fight.
Sync versus async replication
When the leader gets a write, when does it tell the app "done"? There are two answers, and the choice is one of the main tradeoffs in this topic.
Asynchronous. The leader saves the write, replies "done" to the app right away, and sends the change to followers in the background. It is fast. The leader never waits for a slow follower.
The risk is that the leader can die after saying "done" but before the change reached any follower. That write is now gone. The app was told it was safe, and it was not.
Synchronous. The leader saves the write, sends it to a follower, waits for the follower to confirm, and only then says "done". No data is lost if the leader dies, because a second copy already exists.
The cost is speed and availability. Every write now waits for the network trip to the follower. And if that follower is slow or down, writes stop, because the leader is waiting for a reply that is not coming.
| Async | Sync | |
|---|---|---|
| Write speed | Fast | Slower, waits for follower |
| Data loss if leader dies | Possible | None for confirmed writes |
| Follower is down | Leader carries on | Writes can get stuck |
| Followers are up to date | Slightly behind | Up to date |
In practice nobody makes every follower synchronous. If you have five followers and wait for all five, one slow machine drags everything down. A common middle path is semi synchronous: one follower is synchronous and the rest are async. Now there are always at least two machines with every confirmed write, and one slow follower cannot stop the world, because another can take over the synchronous job.
My view: for most apps, async is the default and the honest answer is "we accept losing the last second or so of writes in a rare crash". For payments or anything where lost writes mean lost money, use sync or semi sync and say so.
Replication lag
With async replication, a follower is always a little behind the leader. The gap is called replication lag. Normally it is milliseconds. During a traffic spike, a heavy query on the follower, or a network hiccup, it can grow to seconds or even minutes.
If you read only from the leader, you never notice. The trouble starts when you read from followers to scale reads. The data you get back can be old. Two bugs show up again and again.
Read your own writes
Say a user on a photo app changes their display name from "Asha" to "Asha K" and presses save. The write goes to the leader. The page reloads, and the read goes to a follower that has not yet received the change. The user sees "Asha" again.
To them it looks like the save failed. They press save again. They complain. Nothing is actually wrong with the data, and that is what makes this so annoying to debug.
Fixes:
- Read from the leader for data the user just changed. For example, for 30 seconds after a write by this user, send their reads for their own profile to the leader.
- Always read a user's own profile from the leader, and other people's profiles from followers. Only the owner would notice the lag on their own data.
- Remember the position of the user's last write in the log, and only use a follower that has caught up to that position.
Going back in time
This one is stranger. A user reads a page and sees a new comment. They refresh. This time the read lands on a different follower, one that is further behind, and the comment is gone. Then they refresh again, and it is back.
Time seemed to run backwards for them. Each follower is individually fine, but together they gave an inconsistent story.
The fix is called monotonic reads: make sure a user never reads from a follower that is older than one they already read from. The easy way is to pin each user to one follower, for example by hashing the user id to pick it. Their view may be a bit stale, but it never goes backwards.
If you hit one of these in a design, name it and give the one line fix. That alone sets you apart from most candidates.
When the leader dies
Everything so far was about normal days. Now for the bad day. The leader machine stops. Writes are failing. Someone has to take over, and this takeover is called failover. It can be done by a person, but at any real scale it is automatic.
The steps are:
- Detect the failure. Machines send each other regular "I am alive" messages (heartbeats). If the leader is silent for some time, say 10 to 30 seconds, it is declared dead.
- Choose a new leader. Usually the follower that is most up to date, meaning it has applied the most of the log. This is often decided by a vote among the remaining machines.
- Point everyone at the new leader. The other followers start copying from it, and the apps and proxies send their writes to it.
- Deal with the old leader. When it comes back, it must realise it is no longer the boss and become a follower.
Each step has a trap.
Lost writes
With async replication, the new leader may not have the last few writes of the old leader. Those writes are lost. If the old leader comes back with them, what do you do? Usually you throw them away, which means the data is gone and anything built on top of it, like an id handed out to another system, may now be wrong. This is the price of async that is easy to forget.
The timeout dilemma
A short timeout means fast recovery, but you may declare a healthy leader dead during a slow moment, like a long pause or a busy network. Now you trigger a failover that was not needed, which itself causes a short outage and can make the load worse. A long timeout means fewer false alarms but longer real outages. There is no perfect number. Many teams pick a middle value and tune it with experience.
Split brain
This is the scary one. Suppose the old leader was not dead, just cut off from the network. The followers elected a new leader. Now there are two machines who both think they are the leader, and both accept writes. Each has data the other does not. This is split brain.
It is like a school where the headmaster is stuck in traffic, so the staff pick a new acting headmaster. Then the first one arrives, and both start issuing orders. Students get two different notices.
Ways to prevent it:
- Use a majority vote. A new leader needs the votes of more than half of the machines. In a network split, only one side can hold a majority, so only one side can pick a leader. This is why clusters use an odd number of machines, like 3 or 5.
- Use a number for each term of leadership (often called an epoch or term). The new leader gets a higher number. Followers and storage reject anything from a lower number, so the old leader's writes are turned away. This is called fencing.
- In some setups, an old leader that loses contact with the majority shuts itself down.
When you describe failover in an interview, mention the heartbeat, the election, lost writes, and split brain with a majority vote as the fix. That covers what most interviewers are looking for.
Multi leader and leaderless, briefly
One leader is not the only design. Two others exist, and you should know what they are for and why they are less common.
Multi leader. Several machines accept writes, often one per data centre. Users in Europe write to the Europe leader, and users in Asia write to the Asia leader. The leaders send changes to each other. It is great for low write latency across regions, and it keeps working if a whole region is cut off.
The ugly part is conflicts. Two people edit the same document in two regions at the same moment, and the leaders disagree on the result. Someone has to decide, with rules like "last write wins" (which can silently throw away a real edit), or by merging, or by asking a human. Collaborative editors and calendar apps live here. For most other things, I would avoid multi leader unless there is a clear need.
Leaderless. There is no leader. The client sends each write to several machines at once and reads from several at once. Dynamo, Cassandra and Riak work this way. It survives machine failures very smoothly, because no single machine is special. The price is that you have to think about consistency yourself, which brings us to quorums.
Quorum, with numbers
A quorum is the minimum number of machines that must agree before an operation counts. It is the same idea as needing a majority of a committee present before a vote is valid.
Three letters describe it:
- N is the number of copies of each piece of data.
- W is how many copies must confirm a write before it counts as done.
- R is how many copies you ask when reading.
The rule to remember is this: if W + R > N, a read is guaranteed to touch at least one copy that has the latest write.
Why? Say N = 3, W = 2, R = 2. A write succeeds once 2 of the 3 machines have it. A read asks 2 of the 3. Any group of 2 and any other group of 2 out of 3 must share at least one machine, since 2 + 2 = 4 is more than 3. So the read always meets at least one machine with the new value, and it picks the newest by version number.
Here is the same setup played out:
| Machine | Has the new value? |
|---|---|
| A | Yes |
| B | Yes |
| C | No, it was slow |
A write got confirmed by A and B, which is W = 2. A reader asks B and C. B returns the new value, C returns the old one. The reader sees two different answers, checks the versions, takes the newer one, and can fix C in the background.
Other settings, for N = 3:
| W | R | Result |
|---|---|---|
| 2 | 2 | Balanced. Survives one machine being down for both reads and writes |
| 3 | 1 | Fast reads, but one machine down blocks all writes |
| 1 | 3 | Fast writes, but one machine down blocks all reads |
| 1 | 1 | Fastest, but a read can easily miss the latest write |
So you tune W and R to your needs. A read heavy app picks a small R and a large W. A write heavy app does the opposite.
One honest warning. Even with W + R > N, odd cases exist, like a write that only half succeeded or clocks that disagree. A quorum makes stale reads much less likely. It is not a magic guarantee of perfect consistency. If someone pushes on this in an interview, saying so shows maturity.
Adding a new follower
One small practical thing that is easy to skip. How does a new follower get started? You cannot just tell it to read the log, because the log only goes back so far.
The usual way is to take a snapshot of the leader at one moment, copy it to the new machine, and then ask the leader for every change since that moment. The follower replays those changes until it has caught up, and from then on it follows the live log. All of this happens while the leader keeps serving traffic.
It matters in an interview for one reason: a new copy takes time, and during that time you are running with fewer healthy copies than you planned. If a 2 TB database needs two hours to copy, then losing one follower leaves you exposed for those two hours. That is why people watch replication lag as a number on a dashboard, and alert on it, long before users complain.
Interview talking points
Keep it as a short story with a reason at each step.
- Start with why. "I would replicate for safety and to scale reads. One leader takes the writes, and two or more followers serve reads."
- Say sync or async, and why. "Async for speed, accepting that a crash may lose the last moments of writes. For the payments table I would make one follower synchronous."
- Show you know about lag. "Reading from followers can return stale data. For the user's own recent changes, I would read from the leader, so they always see what they just saved."
- Cover the bad day. "If the leader dies, a heartbeat timeout triggers an election. The most up to date follower wins. I would use an odd number of nodes with a majority vote to avoid split brain."
- Mention alternatives only if asked. "Multi leader helps across regions but brings write conflicts, so I would not start there."
Numbers help a lot. Mention things like "3 copies across 3 availability zones" or "failover within about 30 seconds". They make the answer feel lived in rather than memorised.
And one last habit. Replication copies mistakes too. If someone runs a bad delete on the leader, that delete reaches every follower within milliseconds. Replicas are not backups. Say that you also take regular backups, and you will sound like someone who has been burned before, or at least read about those who were.
Interview questions to check yourself
"Name three reasons to replicate data."
Safety if a machine dies, read scale by spreading reads over copies, and location, by putting copies near users.
"What is the tradeoff between sync and async replication?"
Sync loses no confirmed writes but is slower and can block when a follower is down. Async is fast but can lose the latest writes if the leader dies.
"A user saves their profile and then sees the old version after the reload. What happened and how do you fix it?"
They read from a follower that had not caught up. Read the user's own recent data from the leader, or only from a follower that has reached their last write.
"What is split brain, and how do you prevent it?"
Two machines both think they are the leader and accept writes. Prevent it with majority votes, an odd number of nodes, and fencing with term numbers.
"With N = 3, W = 2 and R = 2, why does a read see the latest write?"
Any 2 machines overlap with any other 2 out of 3, because W + R is greater than N. So at least one machine in the read has the newest value.