Most system design problems forgive small mistakes. A chat message arrives a second late and nobody cares. A wallet does not forgive anything. If it loses 500 rupees, someone is angry. If it shows 500 rupees twice, the company is out of pocket. This note is about building something that is boring, careful and correct, and about how to explain why each careful choice is there.
1. Problem Statement
Design a digital wallet like Paytm, PhonePe, Cash App or the balance inside a ride app. A user holds money in the app. They can add money from a bank, send it to another user, pay a merchant, and pull it back out to the bank. They can see their balance and a list of everything that happened.
The product sounds small. The design is not, because of one rule: money must never be lost, never be created from nothing, and never be moved twice. Everything below exists to keep that rule true when servers crash, networks drop and users tap the Pay button three times.
2. Questions to Ask First
| Question | Assumption I will make |
|---|---|
| How many users? | 50 million registered, 10 million active each day |
| What can money do? | Add from bank, send to another user, pay a merchant, withdraw to bank |
| One currency or many? | One currency per wallet. No currency conversion |
| Can a balance go negative? | Never, for normal users |
| How right must it be? | Exactly right. Money is the one place where eventual consistency is not acceptable |
| How long do we keep records? | 7 years, because auditors and regulators ask |
| What speed do users expect? | A transfer feels done in under 2 seconds |
| Is there a real bank behind the wallet? | Yes, the company holds pooled money in a bank account and a payment provider connects us to it |
| Fraud? | Basic rule checks inside the design. A full fraud system is a separate topic |
| One region or many? | One region, with a copy in a second zone |
Say the bank row out loud: "the wallet is only a record of who owns how much of the pooled money at the bank." It tells the interviewer you understand where the real money lives.
3. Requirements
3.1 Functional Requirements
| Requirement | Detail |
|---|---|
| Balance | Show the current balance, right now, correctly |
| Add money | Pull money from the user's bank into the wallet |
| Transfer | Send money from one wallet to another |
| Pay merchant | Same as a transfer, to a merchant account |
| Withdraw | Send money from the wallet back to the user's bank |
| History | A list of every movement, newest first |
| Refund | Reverse a payment later, fully or partly |
| Audit | Anyone with permission can see exactly why a balance is what it is |
3.2 Non Functional Requirements
- Correctness before everything. No lost money, no double money, no negative balances.
- A retry never causes a second payment. Phones and networks are unreliable, so retries are normal.
- Every change is recorded and records are never edited.
- High availability for reads, and fail safe, not fail open, for writes. If we cannot be sure, we refuse.
- Handles payday and festival peaks, about 5 times a normal day.
- Every day, our books agree with the bank's books, and any difference is found and explained.
3.3 Out of Scope
Cards issued by the wallet, loans, currency exchange, tax reports, KYC document checks.
4. Back of the Envelope Numbers
Money movements per day:
10M active users x 3 money movements = 30 million per day
30M / 86,400 seconds = about 350 per second
Peak is about 5 times = about 1,750, so plan for 2,000 per secondThat is not a big number for a database. It sounds small because the work per movement is large, so count the rows.
Rows written per transfer (this is the cross shard design from section 9.2): one transfers row, four ledger entries, and a few updates to balances and a work queue. Say about 10 row writes.
2,000 transfers per second x 10 writes = 20,000 row writes per second at peakStorage. A ledger entry is about 120 bytes: ids, amount, account, time.
30M transfers x 4 entries = 120 million entries per day
120M x 120 bytes = about 14 GB per day
14 GB x 365 = about 5.3 TB per year
Indexes roughly double it = about 10 TB per year
7 years = about 70 TBReads. A user checks the balance about 10 times a day.
10M x 10 = 100 million balance reads per day
100M / 86,400 = about 1,200 per second
Peak x 5 = about 6,000 per secondHow many shards? I want honesty here. One strong database server can do 20,000 simple row writes per second, and 6,000 reads on top. So on throughput alone, I might start with one primary and a copy. What pushes me to shard is the 70 TB over 7 years, and the blast radius: if one machine holds every user's money, one bad day affects everyone.
8 shards:
writes 20,000 / 8 = 2,500 per second per shard
reads 6,000 / 8 = 750 per second per shard
storage 70 TB / 8 = about 9 TB per shard after 7 yearsEach shard is comfortable. I choose 8 and say plainly that if the interviewer says "start small", one shard with the same schema is a fine first step, because the design does not change when you add more.
Bank facing traffic. Say 10 percent of money movements touch the real bank (add money and withdraw).
30M x 10% = 3 million bank movements per dayThat is the size of the daily bank file we must check, 3 million lines. A computer matches that in minutes.
5. High Level Design
Walk through it box by box.
The API gateway checks who the user is, applies rate limits, and rejects anything without an idempotency key on money movements. It does no money logic.
The wallet service is the brain. It checks the request, asks the fraud rules whether to go ahead, finds the right shard, and runs the money step. It is stateless. If a copy dies, another takes over, because all the truth sits in the database.
The fraud rules are a small fast box that answers allow, ask for an OTP, or block. See the side box right after this walk through.
The shard router answers "which database holds account 4711?". The shard key is the account id (a hash of it), so one account and all its ledger entries always sit in the same shard. That is the only way a single database transaction can protect one account.
Each shard is a normal relational database with strong transactions (Postgres or MySQL). I pick relational on purpose. Money needs real transactions, row locks and constraints, and I do not want to rebuild them by hand on top of something looser. Each shard has a synchronous copy in another zone, so a committed transfer survives the loss of the primary.
The outbox relay watches a small table in each shard and puts its rows onto Kafka. The outbox is how a shard says "step 2 of this transfer is waiting for another shard" without needing any special distributed machinery. More in section 9.2.
The transfer worker reads those jobs and performs the second half of a transfer on the other shard. It can run any number of times for the same job and the result is the same, which is what makes crashes safe.
The payment provider is the company that actually talks to banks and card networks. We never see a card number. The webhook handler receives its "this payment succeeded" calls.
The reconciliation job compares our ledger with the bank's daily statement and puts differences into a breaks queue for a human. It also checks our own books against each other.
The audit and history replica is a read only copy used for statements and compliance queries, so that users scrolling history never slow down money movement.
Side box: fraud checks
Fraud checks sit in front of the debit, and they must be quick, about 50 ms, because the user is waiting. Start with simple rules, they catch most of it:
- Velocity: more than 5 transfers in 10 minutes from one account.
- Size: a transfer much larger than this user's usual, or above a daily limit.
- New things together: a new device, a new recipient and a large amount, all in the same hour.
The answer is allow, step up (ask for an OTP or fingerprint), or block. A model that scores risk can run alongside and train from outcomes, but it feeds the same three answers. If the fraud box is down, small amounts go through under tighter limits and big ones wait. The design for that system is its own topic, so here I only show where it plugs in.
6. API Design
Every call that moves money carries an Idempotency-Key header. That is a random id (a UUID) the app makes once per user intention. If the app has to retry, it sends the same key. The server uses it to recognise "I have seen this one".
6.1 Endpoints
| Method | Path | Purpose |
|---|---|---|
| POST | /v1/transfers | Send money to another wallet or merchant |
| GET | /v1/transfers/{id} | Status of one transfer |
| POST | /v1/topups | Start adding money from a bank or card |
| POST | /v1/withdrawals | Send money back to the user's bank |
| GET | /v1/accounts/me/balance | Current balance |
| GET | /v1/accounts/me/entries?cursor=...&limit=50 | History, newest first |
| POST | /v1/transfers/{id}/refund | Reverse a payment |
| POST | /v1/webhooks/payments | Provider tells us a top up or withdrawal finished |
Create a transfer:
POST /v1/transfers
Idempotency-Key: 5b0c7d62-8c4a-4e7f-a2a1-6f1c0b7e9a11{
"toAccountId": "acc_9021",
"amount": 50000,
"currency": "INR",
"note": "Rent share"
}The amount is a whole number in the smallest unit: 50000 paise is 500 rupees. Never use decimals or floats for money. In most languages 0.1 + 0.2 gives 0.30000000000000004, and that tiny error multiplied by 30 million transfers is a real problem. Integers do not drift.
Response when the debit has been recorded:
{
"transferId": "tr_77120",
"status": "PROCESSING",
"amount": 50000,
"currency": "INR",
"createdAt": "2026-10-10T09:41:07Z"
}The status is PROCESSING until the other side gets the money, then COMPLETED. The app can show "Sent" at once, because the money already left the sender's balance, and it can poll GET /v1/transfers/tr_77120 or get a push when it completes.
6.2 Status codes
| Code | When |
|---|---|
| 201 | New transfer accepted |
| 200 | Same key seen before, returning the original result |
| 409 | Same key is still being processed right now, try again in a moment |
| 422 | Same key but a different body (amount or recipient changed). This is a bug in the app and we refuse |
| 422 | Insufficient balance, with an error code INSUFFICIENT_FUNDS |
| 429 | Too many requests |
| 403 | Fraud rules blocked it, or the account is frozen |
The 422 for "same key, different body" is easy to forget. Without it, a buggy app that reuses a key could quietly send 500 when the user meant 5,000.
7. Data Model
Everything money related lives in the sharded relational database, keyed by account.
7.1 Accounts
accounts
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key. Shard key. All data about one account lives on one shard |
owner_id | uuid | The user or merchant. Indexed |
type | text | USER, MERCHANT, SYSTEM (bank pool, transit, fees) |
currency | char(3) | One currency per account |
balance | bigint | Current balance in the smallest unit, kept up to date in the same transaction as every ledger entry |
status | text | ACTIVE, FROZEN, CLOSED |
version | bigint | Counts changes, helps debugging and optimistic checks |
created_at | timestamptz |
There is a database rule on user accounts: CHECK (balance >= 0). Even if the code has a bug, the database itself refuses to let a user go negative. That is a second lock on the same door.
System accounts are ordinary accounts that belong to the company: the pooled bank money, the transit accounts I explain in 9.2, a fees account. They are how everything stays balanced.
7.2 Transfers
transfers
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key |
idempotency_key | uuid | The key the app sent |
from_account_id | bigint | Shard key. A transfer lives on the sender's shard |
to_account_id | bigint | May be on another shard |
amount | bigint | Smallest unit, always positive |
currency | char(3) | |
kind | text | P2P, MERCHANT, TOPUP, WITHDRAWAL, REFUND |
status | text | PROCESSING, COMPLETED, FAILED, REVERSED |
request_hash | text | Hash of the body, to catch "same key, different request" |
failure_reason | text | Why it failed, for support |
created_at, updated_at | timestamptz |
A unique index on (from_account_id, idempotency_key) is the entire idempotency system. I explain why in 9.1. Because the transfer sits on the sender's shard, and so does the key, the database can enforce it without asking any other machine.
7.3 Ledger entries
ledger_entries
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key, increasing |
transfer_id | bigint | Which transfer caused this line |
step | smallint | 1 debit side, 2 credit side, 3 refund side |
account_id | bigint | Shard key. Lines live on the shard of their account |
amount | bigint | Signed. Negative takes money out of the account, positive adds money |
balance_after | bigint | The balance right after this line, which makes statements trivial and lets us check for drift |
created_at | timestamptz |
Two indexes. A unique one on (transfer_id, step, account_id), which stops the same step from ever being applied twice. And one on (account_id, id), which serves the history screen as one short range read.
Rows are never updated or deleted. The database user that the app uses has no UPDATE or DELETE permission on this table. If something was wrong, you add a new line that cancels it. I like this rule because the history stays true even when someone made a mistake.
7.4 The outbox
outbox
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key |
transfer_id | bigint | |
step | smallint | Which step needs doing next |
payload | jsonb | What the worker needs |
status | text | PENDING, DONE |
attempts | int | How many tries so far |
next_attempt_at | timestamptz | For retries with a delay |
Written in the same transaction as the money step. That is the point of it. We cover why in 9.2.
7.5 Bank side tables and audit
bank_transactions
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key |
bank_ref | text | The bank's own reference. Unique, so loading the same file twice does nothing |
direction | text | IN or OUT |
amount | bigint | |
value_date | date | Day the bank booked it |
matched_transfer_id | bigint | Null until matched |
status | text | MATCHED, UNMATCHED, DISPUTED |
reconciliation_breaks
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key |
run_date | date | Which day's check found it |
type | text | MISSING_IN_LEDGER, MISSING_AT_BANK, AMOUNT_MISMATCH |
details | jsonb | Both sides of the story |
status | text | OPEN, EXPLAINED, CORRECTED |
assigned_to | text | The person looking at it |
audit_log
| Column | Type | Why |
|---|---|---|
id | bigint | Primary key |
actor | text | User, admin or system job |
action | text | FREEZE_ACCOUNT, MANUAL_ADJUSTMENT, LIMIT_CHANGED |
subject | text | What it was done to |
before, after | jsonb | State on each side of the change |
ip, created_at | text, timestamptz |
The ledger is the audit trail for money. The audit log is the audit trail for everything else, like an admin freezing an account. Two different questions, two different tables.
8. Key Flows
8.0 Double entry in plain words
Before the flows, one idea that makes the rest simple. Every time money moves, we write at least two lines, and those lines add up to zero. One line takes money out of somewhere. Another puts the same money somewhere else. Money never appears or disappears, it only moves.
A shopkeeper's cashbook works like this. If you pay 100 for sugar, one column shows 100 out of the cash box, another shows 100 into the stock. A page where the two columns do not match means someone made a mistake, and you can see it straight away.
Example. Asha has 1,000. She sends 300 to Ben.
| Account | Line | Balance after |
|---|---|---|
| Asha | -300 | 700 |
| Ben | +300 | (his old balance + 300) |
Add up the lines: -300 + 300 = 0. If a bug ever writes only one of them, the sum is no longer zero, and a nightly check will shout about it.
Adding money from a bank is the same idea. The company really does hold the user's 500 in its pooled bank account, so one line says "the bank pool gave away a claim of 500" (negative) and the other says "the user now owns 500" (positive). Sum is zero again. Accountants use the words debit and credit and have their own sign rules, so I will not pretend to match them. In an interview, say "every movement is two or more lines that sum to zero" and you are right.
8.1 A transfer between two shards
Asha is on shard 3. Ben is on shard 6. Each shard has a transit account, which is like a tray on the counter where money sits for a moment between two desks.
The step 1 transaction is the one that matters. In SQL:
BEGIN;
INSERT INTO transfers (id, idempotency_key, from_account_id, to_account_id,
amount, currency, kind, status, request_hash)
VALUES ($1, $2, $3, $4, $5, 'INR', 'P2P', 'PROCESSING', $6);
UPDATE accounts
SET balance = balance - $5, version = version + 1
WHERE id = $3 AND status = 'ACTIVE' AND balance >= $5;
-- rows affected must be 1, otherwise ROLLBACK with INSUFFICIENT_FUNDS
UPDATE accounts SET balance = balance + $5 WHERE id = $transit3;
INSERT INTO ledger_entries (transfer_id, step, account_id, amount, balance_after)
VALUES ($1, 1, $3, -$5, $new_balance_asha),
($1, 1, $transit3, $5, $new_balance_transit);
INSERT INTO outbox (transfer_id, step, payload, status)
VALUES ($1, 2, '{"to": 6021}', 'PENDING');
COMMIT;Look at the balance update. It checks the balance and subtracts in one statement. There is no "read the balance, think about it, then write". Between a read and a write, another request could slip in and spend the same money. The conditional UPDATE locks the row, checks and changes it in one step. Two requests for the same account line up one behind the other.
Either all six statements happen or none do. That is what a database transaction promises, and it is the reason this step is safe.
If both accounts happen to be on the same shard, you could do both sides in one transaction and skip the outbox. I would not write that second path at first. With 8 shards, about 7 out of 8 transfers are cross shard anyway (7/8 is 87 percent), so the two step path is the main path. One code path to test beats two.
8.2 The app times out and retries
The user taps Pay. The request reaches our server, the debit commits, but the response is lost on a bad mobile network. The app sees a timeout. What now?
Nobody is charged twice, because the second try hit the unique index and the database said no. We then look up the existing transfer and give back what we gave the first time. The app cannot tell the difference between the first reply and the second, which is exactly what we want.
If the second request has a different hash (amount 5,000 instead of 500), we return 422 and do nothing.
If the first request is still halfway through (not committed yet) when the retry comes in, the retry waits on the row lock for a moment, or we return 409 "in progress, try again". Either way, only one wins.
8.3 Daily reconciliation with the bank
Reconciliation means comparing our records to the bank's records and explaining every difference. Think of checking your own passbook against the bank statement at month end. We do it every day, in the morning.
Three kinds of difference turn up:
- In the bank file but not in our ledger. Perhaps a top up where our webhook never arrived. Fix: credit the user now, with a ledger entry that points to the bank reference.
- In our ledger but not at the bank. Perhaps we credited a top up that the bank later reversed. Fix: reverse our entry, and if the user already spent the money, flag it for collection.
- Both exist but the amounts differ. Usually a fee or a provider bug. Raise it with the provider.
If 0.01 percent of 3 million lines break, that is about 300 items a day, which a small team can clear. Most are timing issues that disappear by themselves the next morning.
There are two more checks that need no bank, and they run every hour:
- Add up every ledger line in the whole system. The answer must be exactly zero.
- For each account, add up its lines and compare to
balance. A mismatch means drift, which should never happen, and it pages someone.
And one more, a bigger picture check: the sum of all user balances should equal the money the bank pool account says we hold. If not, someone's money is not backed.
9. Deep Dives
9.1 Idempotency: one tap, one payment
The problem. Networks fail in the awkward middle: the request arrived and worked, but the answer got lost. The app cannot tell the difference between "never arrived" and "done, but I did not hear". Apps and gateways retry because they must. A retry that charges twice is the worst bug a wallet can have.
Option A: check, then insert. Before making a transfer, look in a table for the key. If it is not there, go ahead and make the transfer, then save the key.
It works on a calm day. Under load, two copies of the same request (a user double tapping, or a retry that overlaps the first try) both look, both see nothing, and both go ahead. Between the check and the save is a gap, and the money moves twice.
Option B: let the database say no. Make the key part of a unique index and insert it as the first step of the same transaction that moves the money. The database guarantees only one insert of that key can ever succeed, even if a hundred arrive at the same instant. The loser gets a unique violation, and we answer from the stored transfer.
I choose B, and the reason is simple: I would rather rely on one rule the database already enforces than on a gap I am hoping nobody falls into. And it costs nothing extra, the key rides in the same transaction.
Some practical details.
The key is made once per user intention, in the app, before the first attempt. Tapping Pay twice quickly sends the same key twice. Starting a new payment makes a new key.
The key is scoped to the sender, (from_account_id, idempotency_key), so two users who happen to pick the same random string do not clash. UUIDs make that very rare anyway.
We save a hash of the request body next to the key. Same key plus same hash means "retry, give back the old answer". Same key plus a different hash means a bug in the client, so refuse with 422.
How long do we keep keys? A client retries within minutes or hours, not months. But the transfer row stays for 7 years anyway because it is part of the record, so the key lives as long as the transfer does. At 30 million rows a day with a 16 byte key, the index adds under 1 GB a day. I would not bother expiring them.
The same idea works one level down. Every step has a unique line (transfer_id, step, account_id) in the ledger. So even if the transfer worker runs step 2 three times, only the first one writes lines. The others hit the unique index and stop. Each layer is idempotent, so it does not matter which layer gets retried.
The same applies to bank webhooks, which also arrive twice sometimes. Use the provider's event id or bank_ref as the key.
9.2 Moving money across shards without a distributed transaction
The problem. Asha's account is on shard 3, Ben's on shard 6. A transfer must take money from one and give it to the other, all or nothing. A single database transaction only covers one database. Two databases need something more.
Option A: a distributed transaction (two phase commit). A coordinator asks both shards "can you commit?". If both say yes, it says "commit" to both. It gives all or nothing across shards on paper.
Why I avoid it:
- Between "yes, I can" and "commit", both shards hold their row locks and wait. If the coordinator crashes in that gap, those rows stay locked until someone recovers it. Asha's account is frozen for every other payment too.
- Every transfer now waits on the slowest shard and the coordinator, and a slow or dead shard slows everyone who touches it.
- Availability multiplies down. If each shard is up 99.9 percent of the time, a transfer needing two of them works about 99.8 percent of the time, and a transfer needing three, 99.7 percent. Add a coordinator and it falls more.
- Many managed databases do not support it, or support it badly.
Option B: one transaction per shard, glued by a durable to do list. Break the transfer into steps and make each step a normal local transaction.
- On Asha's shard, in one transaction: take 500 from Asha, put 500 in the transit account of that shard, and write an outbox row saying "step 2 is needed".
- A worker reads that outbox row and, on Ben's shard, in one transaction: take 500 from that shard's transit account and give it to Ben.
- Mark the transfer done.
The outbox row being in the same transaction as the debit is the clever part. Either both exist or neither does. So there is never a debit with no job to finish it, and never a job without a debit. If the worker or Kafka dies, the row is still there on disk. When things come back, the relay finds it and tries again. Because step 2 is idempotent (the unique ledger line from 9.1), trying twice is harmless.
The transit account solves a small puzzle: where does the 500 live between the two steps? It sits in the transit account of Asha's shard. Each shard has one. When every transfer has completed, the transit accounts across all shards add up to zero. At any moment, the sum across all transit accounts is exactly the money in flight. If that sum stays above zero for more than a few minutes, some step 2 is stuck, and that is an alert.
What if step 2 can never succeed? Ben's account was closed or frozen. Then we run the opposite: on Asha's shard, a refund step moves 500 back from the transit account to Asha, with new ledger lines (step 3) and the transfer marked REVERSED. This pattern, steps with matching "undo" steps, is called a saga. A saga is just "a series of local transactions where each has a planned way to undo it". I do not need to go deeper than that in an interview, and there are fancier forms with a central orchestrator.
What does the user see? After step 1 their balance is already down and the app says "Sent", usually in under 300 ms. Ben sees the money a moment later, typically within a second. For about a second the money is in neither account, only in transit, and that is fine because the books still balance and nobody can spend it twice.
I choose B. It trades "instant, everywhere" for "safe, and a second later". For money, I would take that trade every time.
9.3 Hot accounts
The problem. Most accounts get a few transactions a day. A big merchant can get 1,000 payments a second during a sale. Every payment updates that merchant's balance row, and a row can only be updated by one transaction at a time. If each holds the lock for 2 ms:
1 second / 2 ms = 500 updates per second on one rowSo a single balance row tops out near 500 per second. Payments queue up and users see slow payments.
Option A: keep one balance row and make the transaction shorter. Do less work while holding the lock, use fast disks. This can give you maybe 2 times, not 20 times.
Option B: split the hot account into slices. A merchant gets, say, 16 sub accounts. Each incoming payment is credited to one of them, picked at random. The merchant's balance is the sum of the 16 slices, read when needed.
16 slices x 500 per second = 8,000 payments per second for one merchantCredits are easy to spread out because adding money never needs a balance check ("is there enough?"). Withdrawals need the total, so a payout reads the sum of the slices, then takes money from slices one by one.
I choose B, but only for accounts that prove they are hot. A normal user keeps a single row, because the sum of 16 rows on every balance read would add cost for no benefit. A background job promotes a merchant when its updates per second pass, say, 200. Everyone else gets the simple path. I would not build this until the numbers ask for it.
The ledger lines do not suffer from this problem, because inserts of new rows do not block each other. Only the single running balance does.
One more rule that matters here: whenever a transaction touches two account rows on the same shard (for example the debit and the transit row), always lock them in the same order, lowest account id first. If one transaction locks A then B while another locks B then A, they wait on each other forever. A fixed order makes this impossible.
10. Failure Handling
| What breaks | What the user sees | What happens underneath |
|---|---|---|
| App loses the response | Spinner, then the right result after retry | Same idempotency key returns the stored transfer. No second payment |
| Wallet service dies mid request | A moment of delay | The database transaction either committed or rolled back. A retry resolves it either way |
| Shard primary dies | Short pause on accounts of that shard, retry works | The synchronous copy is promoted. Committed transfers are on it. Other shards are unaffected |
| Kafka or relay is down | Sender sees Sent, receiver sees the money a bit later | Outbox rows wait on disk. They are published when the relay returns |
| Worker dies after step 2 but before marking done | Nothing | Job retried. The unique ledger line stops a second credit. The mark is then made |
| Step 2 keeps failing for a closed account | Transfer shows Failed, and the money is back in the sender's balance | Refund step moves it back out of transit. Transfer marked REVERSED |
| Bank webhook arrives twice | One credit | Webhook event id is the key. The second one is ignored |
| Bank webhook never arrives | Top up shows Pending | A job asks the provider about every top up still pending after 5 minutes |
| Provider is down | Top up and withdrawal unavailable. Wallet to wallet still works | Circuit breaker on the provider calls. Show a clear message |
| Fraud service is down | Small payments go through under tighter limits, large ones wait | Fail safe for large amounts, usable for small ones |
| Ledger and balance disagree | Nothing at first | Hourly check pages someone. Account may be frozen until it is explained |
| A person tries to edit a ledger row | Not possible | The app database user has no UPDATE or DELETE on that table. Corrections are new lines |
| Reconciliation finds a break | Nothing, usually | Break queue, a person explains it, and a correcting entry is posted |
A general rule that ties it together: in doubt, refuse and let the user try again. A refused payment costs a retry. A wrong payment costs money and trust.
11. Tradeoffs
| Decision | Chosen | Alternative | Reason |
|---|---|---|---|
| Database type | Relational, with strong transactions | A looser store | Money needs row locks, constraints and real commits |
| Amounts | Integers in the smallest unit | Decimals or floats | Floats drift, integers do not |
| Balance | Stored column, updated with every ledger line | Sum all lines on each read | Reading 1 row is fast, summing years of lines is not. The hourly check keeps it honest |
| Records | Append only ledger | Update rows in place | History stays true, and audit is free |
| Idempotency | Unique index on key | Check, then insert | The database enforces it, with no gap to fall into |
| Across shards | Local transactions plus outbox and saga | Two phase commit | No locks held across machines, and one slow shard does not stall others |
| Money in flight | Transit accounts per shard | A flag on the transfer | Books keep balancing, and stuck money is easy to spot |
| Hot accounts | Split into slices, only when hot | One row for all | A row handles about 500 updates per second |
| Shards | 8, by hash of account id | One large database | Storage of 70 TB and limiting the damage from one failure, not raw speed |
| Fraud | Fast rules inline, model on the side | Model only | Rules are quick and explainable, and a model can be added later |
| Failure stance | Refuse when unsure | Let it through | A refused payment is cheap, a wrong one is not |
12. Interview Talking Points
- Start with the rule: money is never lost, never created, never moved twice. Then ask your questions (one currency, no negative balances, 7 years of records) and say that correctness beats speed here.
- Do the numbers in small steps: 30M transfers a day is about 350 per second, 2,000 at peak, about 10 row writes each, and about 70 TB over 7 years. Then say honestly that throughput alone does not need 8 shards, but storage and blast radius do.
- Explain double entry in one picture: every movement is at least two lines that add up to zero, like a cashbook. Say that amounts are integers in the smallest unit and ledger rows are never edited, only cancelled by new rows.
- Show the single shard transaction with the conditional UPDATE (
balance >= amount). Say why there is no read then write: someone else can slip in between. - Cover idempotency early. The key is made by the app per intention, enforced by a unique index inside the same transaction. Mention the 422 when the same key arrives with a different body.
- Say that you avoid two phase commit, with the reasons (locks held while waiting, one slow shard slows everyone, availability multiplies down). Then describe the replacement: local transaction, outbox row in the same transaction, an idempotent worker, a transit account, and a refund step as the undo. Name it a saga and move on.
- Describe the daily bank reconciliation: statement file, match by bank reference, a short queue of breaks for humans, plus the hourly checks (all lines sum to zero, each balance equals its lines).
- Close with the things that show experience: hot merchant accounts and slicing, webhooks arriving twice or never, the fraud check as a small fast box in front of the debit, and the rule that when the system is unsure, it refuses.