Message Queues and Streams

Think about a busy bakery. The person at the till takes your order, writes it on a paper slip, and clips the slip to a rail above the kitchen. Then they turn to the next customer. In the kitchen, a baker looks at the rail, takes the oldest slip, and starts working on it.

The cashier never walks into the kitchen and waits. The baker never talks to customers. If the kitchen gets slammed at lunch, slips pile up on the rail, but the till keeps taking orders. If a baker goes on a break, the slips wait for them. Nothing is lost.

That rail is a message queue. Almost every large system has one somewhere, and it comes up in nearly every system design interview. This note covers what it is, the promises it can make (and the ones it cannot), and when you should leave it out.

Why put a queue between two services

Say you build a photo app. When a user uploads a picture, you need to do several things: save the original, make three smaller sizes, run a check for bad content, and send a "your photo is live" notification.

You could do all of that inside the upload request. The user then waits 8 seconds, looking at a spinner. And if the resize step crashes, the whole upload fails.

The other way: save the original, drop a message on a queue saying "photo 981 needs processing", and answer the user right away. Other workers pick up the message and do the slow parts in the background. A queue gives you four things.

It absorbs spikes. If 10 times the normal traffic arrives for five minutes, the queue grows and the workers catch up afterwards. Without a queue, the spike goes straight to the workers and they fall over.

It hides slow work. The user does not wait for the resize or the email. They only wait for the cheap part.

It helps with retries. If a worker crashes halfway through, the message is still on the queue and another worker can try again. With a direct call, a crash means the work is gone unless the caller remembers to try again.

It separates the two sides. The uploader does not need to know who processes the photo, how many workers there are, or whether they are running right now. You can restart or deploy the workers without the upload service noticing.

Notice what you give up. The user gets an answer before the work is done, so the result shows up a little later. We will come back to that in the last section.

The three parts

A message is a small piece of data, often a few lines of JSON: {"photoId": 981, "action": "resize"}. It describes something that happened or something to do. Keep messages small, and put big things like the image itself in storage, with just the id in the message.

The producer is the code that creates messages and sends them. The consumer is the code that receives and handles them. The broker is the server in the middle that stores messages until a consumer takes them. RabbitMQ, Amazon SQS and Kafka are all brokers.

One broker is a single point of failure, so real brokers run as a cluster of several machines and keep copies of each message. That is the broker's problem, not yours, but it is good to say out loud that messages are replicated.

Queue or topic

There are two ways to deliver a message, and mixing them up is a common source of confusion.

In a queue, each message goes to exactly one consumer. If you have five workers reading from a queue, they share the work. Message 1 goes to worker A, message 2 to worker B. This is for jobs: resize this photo, send this email. Each job should be done once.

In a topic (also called publish and subscribe), each message goes to every subscriber. This is for news. Say an order was placed. The email service needs to know, the stock service needs to know, and the analytics service needs to know. The order service just publishes "order 77 placed" to a topic, and all three get a copy. It does not even know who they are.

New subscribers can be added later without touching the producer. That is the main selling point.

Consumer groups

Now a real problem. Say the email service runs three copies for capacity. When "order 77 placed" is published, we want the email service to get it once, not three times and send three emails.

A consumer group solves this. All copies of one service join the same named group, like email-service. A topic delivers each message to every group, but inside a group only one member gets it. So across groups it behaves like a topic, and inside a group it behaves like a queue.

Kafka made this idea famous, but SQS plus SNS fan out, and RabbitMQ exchanges with several queues, give you the same shape. For the interview, the important thing is the sentence "a topic for fan out, a group to share the work inside each service".

Acknowledgements

How does the broker know a message was handled? The consumer tells it. That is called an acknowledgement, or ack.

The flow goes like this. The broker hands the message to a consumer, and it keeps the message but hides it from other consumers for a while (SQS calls this the visibility timeout, say 30 seconds). The consumer does the work. When it is done, it sends an ack and the broker deletes the message. If no ack arrives in time, because the consumer crashed or got stuck, the broker makes the message visible again and gives it to someone else.

This is what makes retries work. It also means a message can be delivered more than once, which leads to the next topic.

At most once, at least once, exactly once

These three are about how many times your code will see a message, and they come directly from when you ack.

At most once. The consumer acks first, then does the work. If it crashes after the ack and before finishing, the message is gone and the work never happens. No duplicates, but you can lose messages. Fine for things like a metrics counter, where a missing data point does not hurt.

At least once. The consumer does the work first, then acks. If it crashes after the work and before the ack, the broker does not see the ack and delivers the message again. The work happens twice. No loss, but you can get duplicates. This is the default for almost every broker and what you will use for most things.

Exactly once. Every message is handled one time, no more, no less. It sounds like what we all want, and I will be honest: as a promise between two separate systems, it is mostly a story.

Why? Look at the crash point. The consumer sends an email and then crashes before the ack. The broker cannot know whether the email went out. If it redelivers, you may send two emails. If it does not, you may send none. There is no third choice, because the email system and the broker are two machines that cannot agree on one shared moment. The network can lose the message, or the ack, at any time.

Some products do advertise exactly once. Kafka has transactions that can make a read from Kafka, a process step, and a write back to Kafka happen as one unit. That is real and useful, but only inside Kafka. As soon as the work touches the outside world, such as an email, a payment gateway or your database, the guarantee ends at the edge.

So what do experienced people do? They use at least once delivery and make the consumer idempotent. Idempotent means that doing the same thing twice has the same result as doing it once. The result is "effectively once".

A tiny example. A message says "charge order 77, payment id p-123". The consumer first checks a table of payment ids it has already handled. If p-123 is there, it skips the work and acks. If not, it does the charge and records p-123, ideally in the same database transaction. Now a duplicate is harmless. Other cheap tricks: use "set balance to 50" rather than "add 10", and use unique keys in the database so a second insert fails.

If an interviewer asks about exactly once, a good answer is: "I would not rely on the broker to give it. I would use at least once, and give every message an id so the consumer can ignore duplicates."

Ordering and partitions

Does the queue keep messages in order? With one queue and one consumer, yes. The consumer sees message 1, then 2, then 3.

Add a second consumer to go faster and the order is gone. Consumer A takes message 1, consumer B takes message 2, and B may finish first. For a job queue that is fine. For things like "user changed name to X" followed by "user changed name to Y", it is not. If they are processed in the wrong order, the user ends up with X.

You also cannot have a single line forever. One queue on one machine has a limit on how much it can handle. So the big systems split a topic into partitions. A partition is one independent, ordered line of messages. A topic with 6 partitions is six lines.

The rule is that order is guaranteed only inside a partition, not across them. The producer picks the partition by hashing a key, such as the user id or order id. Same key, same partition, always. (This is the same idea as hashing keys onto servers, from the last note.)

All events for user 7 go to partition 0, so they are read in order by one consumer. Events for different users may be handled in any order, and that is fine because they do not depend on each other.

Inside a consumer group, each partition is read by exactly one member. That gives you a simple rule about speed: the number of partitions is the maximum number of consumers that can work in parallel. With 6 partitions, a 7th consumer in the group sits idle. So people choose a partition count a bit higher than they need today.

Two traps. If one key is much busier than the rest (one huge customer), its partition becomes a hot spot and the others sit bored. And changing the number of partitions later changes the hash result, so old and new messages for one key can end up in different partitions.

SQS standard queues give no ordering promise at all. SQS FIFO queues keep order within a "message group id", which is the same idea as the key.

Dead letter queues

Some messages just cannot be processed. The JSON is broken, a bug in the code crashes on one particular input, or the user it refers to was deleted. We call this a poison message.

Left alone, it does this: the consumer takes it, crashes, no ack, the broker redelivers, the consumer takes it again, crashes again. Forever. It burns CPU and, in some systems, blocks everything behind it.

The usual fix is a dead letter queue (DLQ). It is an ordinary second queue that you set up beside the main one. You tell the broker, "after 5 failed attempts, move the message to the DLQ". The main queue goes on with healthy messages.

Two things make a DLQ useful rather than a graveyard. First, set an alarm on it, since a message in the DLQ means something is wrong and nobody will notice otherwise. Second, make it possible to look at the message and send it back to the main queue once the bug is fixed (this is often called a redrive).

Also wait a bit between retries, and wait longer each time (1 second, 2, 4, 8). This is called exponential backoff. If the failure came from a downstream service that is down, hammering it every 10 ms makes things worse.

Backpressure

A queue sits happily between a fast producer and a slow consumer, but only for a while. Say the producers add 1,000 messages a second and the consumers can handle 800. The queue grows by 200 every second. After an hour, that is 720,000 messages waiting. Each new message now waits ages before it is handled, so the delay the user sees keeps growing, and eventually the broker runs out of disk or memory.

Backpressure means pushing the problem back towards the source instead of letting it pile up. You have a few options, and usually you combine them.

• Add more consumers (the simplest, and the first thing to try).
• Set a limit on the queue size, and when it is full, slow down or reject the producers so they get an error instead of an endless wait.
• Drop work that is no longer useful. A "location update" from 10 minutes ago does not matter any more, so skip stale messages.
• Limit how fast producers can send.

The number to watch is not just queue length but the age of the oldest message. A queue of 10,000 that drains in two seconds is healthy. A queue of 100 that has not moved in an hour is broken. Put an alarm on the age.

The mistake almost everyone makes is treating the queue as a way to make a slow thing fast. It does not. It only moves the waiting from the user to the queue. If the consumers are permanently slower than the producers, no queue is big enough.

Queue or log

So far I have mixed two designs together. They look alike from the outside and work very differently inside.

A classic queue (SQS, RabbitMQ, ActiveMQ) is like the bakery rail. The broker keeps track of each message. Once a consumer acks, the message is deleted. It is a to do list.

A log (Kafka, Kinesis, Redpanda) is more like a diary or a ledger. Messages are appended to the end of a file, in order, and they are never removed when read. They stay for a retention period, such as 7 days. Each consumer keeps its own position in the log, called an offset, like a bookmark. To read more, it moves the bookmark forward.

Here is when I would pick each. For "do this job later, once, and retry if it fails", such as sending emails or resizing images, a queue is the right tool, and SQS is boring and cheap to run. For "something happened, and several systems care, and I might want to reread it", a log is the right tool. Think of activity feeds, click streams, change data from a database, and anything where a new service should be able to start from last week's events.

Replay is the big one. If a bug in your analytics consumer corrupted yesterday's numbers, with a log you fix the bug, move the offset back to yesterday, and process again. With a queue, the messages are gone.

Be careful of the trap on the other side too. Kafka is not free. It needs more care to operate, and the single bad message holding up a partition surprises many teams. Do not choose it just because it is famous.

A small note: both worlds are drifting towards each other. RabbitMQ now has a stream type, and Kafka has added queue like features. For interviews the two shapes are still the useful way to think.

When not to use a queue

A queue is not always the answer, and saying so in an interview shows judgement.

When the caller needs the answer now. A user searching for a product wants results in 100 ms. Putting the search request on a queue and waiting for a reply only adds delay. Use a direct call.

When the volume is small. If you send 50 emails a day, a plain function call, or a simple jobs table in your database that a small loop reads, is enough. I would skip the broker until the first real pain, like spikes or retries you keep writing by hand. A queue is one more system to run, monitor and pay for.

When you need all or nothing across steps. "Take money from account A and add it to B" must succeed or fail together. A queue makes the two steps happen at different moments, and you must handle the case in the middle. A database transaction is simpler where it applies.

When it would hide a real problem. If a query takes 12 seconds because it is missing an index, put the index in. Do not move it behind a queue.

When users must see the result immediately. After a queue, the state is eventually correct, not instantly correct. A user who clicks "post" and does not see their post for 3 seconds feels it. You can handle it (show the post locally while it saves), but it costs work.

Also remember the new things a queue brings: duplicates to handle, ordering to think about, messages to monitor, a DLQ to look after, and a harder time debugging, since a request is now spread over several services and several moments.

Interview talking points

A short flow that works well: "The upload service writes the file and puts a message on a queue, then returns right away. Workers read the queue, with at least once delivery, and every message carries an id so processing is idempotent. Failed messages retry with backoff and land in a dead letter queue after five tries, with an alarm on it. I would watch queue depth and oldest message age, and add workers when it grows."

Then, if the problem is about events with several readers: "I would use a log like Kafka, with the topic partitioned by user id so each user's events stay in order, and a consumer group per downstream service."

Interview questions to check yourself

"Name three reasons to put a queue between two services. It absorbs traffic spikes, it takes slow work out of the user's request, and it lets failed work be retried. It also separates the two services so either can be deployed alone."

"What is the difference between a queue and a topic with consumer groups?"

In a queue each message goes to one consumer, so they share the work. In a topic each message goes to every group, and inside one group only one member gets it.

"Why is exactly once mostly a story?"

The broker and the outside system (email, payments) cannot agree on whether a crashed step finished. In practice you use at least once delivery and make the consumer idempotent.

"How do partitions keep order, and what is the limit?"

Messages with the same key go to the same partition, which is read in order by one consumer. Order is only guaranteed inside a partition, and the number of partitions caps the parallel consumers in a group.

"When would you pick a log like Kafka over a queue like SQS?"

When many services need the same events, when you want to replay old events, or when throughput is very high. For plain background jobs with retries a queue is simpler.