Idempotency and Retries

You tap "Pay 500 rupees" on a food app. The screen spins. Nothing happens for ten seconds, then it shows "Something went wrong". Did the payment go through or not? You do not know. The app does not know either.

This one small moment is behind a huge part of system design. The network can lose messages, and when it does, nobody can tell which message got lost.

Why duplicates happen

A request has two halves: the request going to the server, and the reply coming back. Either half can fail.

Look at what the client sees. Only silence. That silence looks exactly the same whether the request never arrived, the server crashed halfway, or the server finished and the reply got lost. A timeout does not mean "it failed". It means "I do not know".

So the client has two choices. Give up (and maybe the customer paid but got no food), or try again (and maybe the customer pays twice). Retrying is the better default, since most failures are short and temporary. But a blind retry of a payment is how people get charged twice.

Duplicates also come from other places: a user double clicking a button, a mobile app that resends when the connection comes back, a message queue that delivers a message twice (many queues promise "at least once", which means "never lost, but maybe repeated"), or a load balancer that retries on its own.

You cannot stop duplicates from arriving. What you can do is make them harmless.

What idempotent means

An operation is idempotent if doing it once and doing it ten times leave the world in the same state.

The picture I like is the lift button. You press "up" and the lift is on its way. Pressing it again, three more times because you are in a hurry, does not call four lifts. The state is "lift is coming", and pressing again changes nothing. Now compare with a "tip the waiter 50 rupees" button. Each press adds 50 more. That is not idempotent.

Some examples in code terms:

• Set the user's email to a@b.com. Doing it twice gives the same result. Idempotent.
• Delete order 42. After the first call it is gone. The second finds nothing, but the state is still "order 42 does not exist". Idempotent. (The second call may return 404 instead of 200, and that is fine. The word is about state, not about the response code.)
• Add 500 to the balance. Twice means 1000. Not idempotent.
• Create a new order. Twice means two orders. Not idempotent.

A common mix up: idempotent is not the same as safe. A safe operation does not change anything at all, like reading a page. Delete is idempotent but definitely not safe.

Which HTTP methods are idempotent

HTTP defines this for you, and interviewers like to ask it.

The table describes what the rules say. Your own server can still break them. A GET that quietly increments a counter or sends an email is not idempotent in practice, and it will bite you when a browser or a proxy repeats it. Keep GET honest.

The practical point: for PUT and DELETE you often get safe retries for free, as long as you built them the way HTTP intended. The trouble is POST, and POST is exactly where payments, orders and signups live. That is where idempotency keys come in.

Idempotency keys

The idea is simple. The client makes up a unique id for the action it wants to do, and sends it with the request. If the same id arrives again, the server knows it is a repeat and returns the earlier answer instead of doing the work again.

Think of the token you get at a bank counter. If you walk up with token 47 twice, the clerk looks at the log, sees token 47 was already served, and hands you the same receipt.

The key is usually a UUID that the client creates once per user action, and sends in a header, for example Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7. The word "once per user action" matters. If the user taps Pay, you create a key. If the network fails and you retry, you send the same key. If the user later starts a new payment, new key.

A full payment example

The flow, including a retry after a lost reply:

What the server stores, per key:

Now the three cases the server has to handle.

First request with a new key. The insert works. The server does the charge, saves the response on that row, and replies.

Second request, same key, same body. The insert fails because the key already exists. The server reads the row. If status is done, it returns the stored response with no new charge. If status is in_progress, the first request is still running (maybe the client got impatient and retried quickly). Return 409 Conflict and let the client try again in a moment, or wait and then return the stored result.

Same key, different body. The client sent key K1 with 500 rupees first, and now K1 with 900 rupees. That is a bug in the client, or someone trying something odd. The server compares the stored request hash with the new one, sees they differ, and refuses with an error such as 422. It must not quietly do either payment. Returning the old response for the wrong body would be confusing, and charging again would defeat the whole point.

The detail that makes this work is the unique constraint in the database. Do not do "check if the key exists, then insert" as two separate steps. Two requests arriving together would both see "not found" and both charge. Let the database decide who wins with a single insert that fails on a duplicate.

A small Java sketch

This is a tiny in memory version, only to show the shape of the check. Run on Java 17 or newer.

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class IdempotencyDemo {

    record Saved(int bodyHash, String response) {}

    private final Map<String, Saved> store = new ConcurrentHashMap<>();
    private int chargesMade = 0;

    String pay(String key, String body) {
        Saved old = store.get(key);
        if (old != null) {
            if (old.bodyHash() != body.hashCode()) {
                return "422 key reused with a different body";
            }
            return old.response() + " (replayed)";
        }
        chargesMade++;
        String response = "200 charged " + body;
        store.put(key, new Saved(body.hashCode(), response));
        return response;
    }

    public static void main(String[] args) {
        IdempotencyDemo api = new IdempotencyDemo();
        System.out.println(api.pay("k-1", "500 INR"));
        System.out.println(api.pay("k-1", "500 INR"));
        System.out.println(api.pay("k-1", "900 INR"));
        System.out.println("charges made: " + api.chargesMade);
    }
}

Output when I ran it:

200 charged 500 INR
200 charged 500 INR (replayed)
422 key reused with a different body
charges made: 1

Three calls came in, and only one charge happened. Be honest about what this sketch skips. The get then put is the two step check I just warned you about, so two threads sending the same key at the same instant could both charge. In a real service, that part is a database insert with a unique key. It also skips the in_progress state, expiry, and the case where the server crashes after the card was charged but before the row was updated.

That last case is worth a sentence. If your payment provider also supports idempotency keys (the big ones do), pass your key along to them. Then even a crash in the middle is safe, because the retry reaches the provider with the same key and they will not charge twice.

Who makes the key

The client does, not the server. If the server made the id, the client would have to ask for one first, and that request could be lost and repeated too. Same trap, one level up. Often the key is made when the user opens the checkout screen or taps the button, and kept in memory or storage until that payment is finished.

Another good option is a natural key. "One review per user per product" can be enforced with a unique constraint on (user_id, product_id), with no extra header at all. When a business rule gives you a natural key, use it.

Retries

Now the client side. A failed call can be retried, but only if it is safe to retry, and only in a polite way.

Which failures are worth retrying:

• Timeouts and connection errors. Worth a retry, if the call is idempotent or carries a key.
• 503 Service Unavailable and 429 Too Many Requests. The server is saying "later". Retry, and obey a Retry-After header if present.
• 500 and 502. Sometimes temporary. Retry a few times.
• 400, 401, 403, 404, 422. These mean you did something wrong. Sending the exact same request again will fail the exact same way. Do not retry.

Exponential backoff

If you retry instantly in a loop, you hammer a server that is probably already struggling. Backoff means waiting between tries, and waiting longer each time. Exponential means the wait doubles.

A small schedule with a base of 1 second:

Put a cap on the wait (say 30 seconds) and a cap on the number of tries (say 5). A retry loop with no end is a bug that is waiting for a bad day.

Jitter

Backoff has a hidden flaw. Say 10,000 clients all hit an error at the same moment because a server restarted. They all wait exactly 1 second, then all retry together: 10,000 requests in one instant. Then all wait exactly 2 seconds and do it again. The server gets hit in sharp waves, and every wave can knock it over again.

Jitter means adding randomness to the wait. The common form is "full jitter": pick a random wait between 0 and the backoff value.

With 10,000 clients, the retries now spread across the whole first second, about 10 per millisecond, instead of one wall of 10,000. A server can recover from that. Think of a school gate when the bell rings: if all the children are pushed out in one second you get a crush, but let them leave over a minute and it flows.

In pseudocode, the wait is roughly random(0, min(cap, base * 2^attempt)). Most HTTP client libraries and cloud SDKs have this built in. Use theirs before writing your own.

Retry storms

A retry storm is when retries make an outage worse, or turn a short blip into a long one.

Here is how it grows. A database slows down. Requests start timing out. Every client retries. Now the database gets the original load plus all the retries, so it gets slower, so more requests time out, so there are more retries. It cannot catch up.

There is a nastier version in layered systems. Say the app calls service A, which calls service B, which calls service C. Each layer retries 3 times. When C fails, B tries 3 times, and each of A's 3 attempts causes that, so C sees 3 x 3 x 3 = 27 calls for a single user click. The deeper the chain, the worse the multiplication.

What helps:

• Retry in one place only, usually the edge closest to the user, not at every layer.
• Use backoff with jitter, always.
• Cap the number of tries.
• Use a retry budget: allow retries to be at most 10 percent of normal traffic. If more than that is failing, stop retrying and fail fast.
• Use a circuit breaker. After many failures in a row, it "opens" and rejects calls for a short time without even trying, giving the broken service some quiet. Then it lets a single test call through to see if things recovered.
• Shed load on the server side: return 429 or 503 quickly rather than slowly dying, and send Retry-After.

Timeouts

Every call over a network needs a timeout. Without one, a hung server makes your thread wait forever, threads pile up, and soon your own service is down because of someone else's problem.

Some rules of thumb:

• Set both a connect timeout (how long to wait to open the connection, usually short like 1 second) and a read timeout (how long to wait for the answer).
• Pick numbers from real data. If a call normally takes 50 ms and the slowest 1 percent takes 400 ms, a timeout of 1 second is sensible. Not 60 seconds because "that feels safe".
• The timeout of the caller should be longer than the total time of what it calls. If A waits 2 seconds for B but B waits 5 seconds for C, A has given up long before B finishes, and the work happens anyway.
• Remember what a timeout means: you do not know if it worked. That is why timeouts and idempotency go together. A timeout without an idempotency key is a coin toss.

The outbox idea, briefly

Here is a different duplicate problem. When an order is placed, you want to save it in your database and also send an "order placed" message to a queue, so other services (email, delivery) react.

If you do these as two separate steps, one can succeed and the other fail. Saved the order but crashed before sending the message: the email never goes out. Sent the message but the database save failed: the email says an order that does not exist.

The outbox pattern fixes this by writing the message into a table in the same database, in the same transaction as the order. Either both rows are saved or neither is. A separate small worker then reads the outbox table and publishes the messages to the queue, marking each as sent.

Be honest about the cost. You need to run and watch that worker. Messages arrive a little late, since they wait for the worker's next poll. And duplicates are still possible: if the worker publishes a message and crashes before marking it sent, it will publish it again. So the consumers, like the email service, must be idempotent too. Outbox moves the problem to a smaller, safer place. It does not delete it.

I would not add this on day one for a small app. Reach for it when losing or repeating an event would really hurt, like money or orders.

Interview talking points

When you design anything with payments, orders, or messages, say this out loud before the interviewer asks:

"Networks can duplicate or lose requests, so the client sends an idempotency key with every create request. The server stores the key with a hash of the request body in a table with a unique constraint, and replays the saved response if it sees the same key again. If the same key comes with a different body, we reject it. Clients retry with exponential backoff and jitter, with a cap on attempts, and only in one layer so we avoid retry storms."

If you have time, add: "Message consumers are also idempotent, because the queue gives at least once delivery."

The mistake almost everyone makes is saying "exactly once delivery" as if it is something you can buy. In practice, you get at least once delivery plus idempotent handling, and together that behaves like exactly once.

Interview questions to check yourself

"A client times out waiting for a reply. What does that tell you about whether the server did the work?"

Nothing. The request may have never arrived, failed halfway, or finished with the reply lost. That is why retries need idempotency.

"Is DELETE idempotent? What about POST?"

DELETE is, since the end state (gone) is the same however many times you call it. POST usually is not, since each call tends to create something new.

"A client sends key K1 with a 500 rupee body, then K1 again with a 900 rupee body. What should the server do?"

Reject it with an error like 422. The stored body hash does not match, so it is a client bug, and neither replaying the old response nor charging 900 is right.

"Why does jitter matter on top of exponential backoff?"

Without it, all clients that failed together retry together in sharp waves. Random waits spread the retries out so the server can recover.

"Why can the outbox pattern still deliver a message twice?"

The worker may publish a message and crash before marking it sent, so it publishes again after restart. Consumers must handle repeats safely.