Not every useful action belongs between an HTTP request and its response. Sending a welcome email, resizing an uploaded image, producing a monthly report, and calling a slow third-party service all take time and can fail for reasons unrelated to the user’s immediate request.
A task queue lets the API record “this work needs doing” and reply quickly. Separate worker processes do the work later. The important word is later: once work becomes asynchronous, delivery, retries, visibility, duplicates, and user feedback all need an honest design.
Why not do it in the request?
Consider signup that sends a verification email synchronously:
- create the user;
- call the email provider;
- wait for its response;
- then tell the user signup worked.
If the provider is slow, the user stares at a spinner. If it times out after the user row is committed, did signup fail or did only the email fail? Retrying the whole HTTP request may create a duplicate account. A queue separates the reliable core action from the best-effort follow-up.
The API can instead create the account, register a send-verification-email job, and return 201 Created. The worker gets its own retry policy and monitoring. The user can be told, truthfully, “Account created; email is being sent.”
The moving pieces
- A producer creates a job with a type and payload.
- A broker stores and delivers jobs (for example SQS, RabbitMQ, Redis-backed queues, or a streaming platform for a different kind of workload).
- A worker consumes jobs and performs the work.
- An acknowledgement tells the broker the job completed successfully.
- A dead-letter queue or failed-job store holds jobs that have exhausted sensible retries.
{
"id": "job_01J...",
"type": "send-verification-email",
"payload": { "userId": "usr_123" },
"attempt": 1,
"createdAt": "2026-10-11T08:30:00Z"
}Keep the payload small and stable. Prefer an id such as userId over copying a full user object that can be out of date by the time a worker runs.
Acknowledgements mean duplicates are normal
Most practical queues give at-least-once delivery. A worker receives a job, performs the side effect, and then acknowledges it. If the worker crashes after sending the email but before the acknowledgement reaches the broker, the broker will deliver the job again.
That is not a broken queue; it is the price of not losing work during failures. Workers must be idempotent: doing the same job more than once should not create an incorrect result.
For an email, record a delivery key or use a provider idempotency feature. For a billing action, an idempotency key is mandatory. For an image resize, writing to a deterministic destination can make repeats harmless.
Retries should be selective and slow down
Not all failures deserve the same response:
| Failure | Usually retry? | Example response |
|---|---|---|
| Provider timeout / temporary 503 | Yes | exponential backoff with jitter |
| Invalid email address | No | mark job failed; surface a fixable product state |
| Missing referenced user | Usually no | investigate data consistency |
| Rate limit | Yes | respect Retry-After and reduce concurrency |
Exponential backoff avoids turning one provider outage into thousands of requests per second. Add jitter so many workers do not retry in lockstep. After a bounded number of attempts, place the job somewhere humans or an automated repair flow can see it. A silent infinite retry is not reliability.
The database-to-queue gap
There is a subtle failure mode in this code:
await users.insert(user); // succeeds
await queue.enqueue(emailJob); // process crashes before this callThe account exists but its verification email job was never created. Reversing the calls has the opposite problem: a job may run for a user that was never committed.
For important work, use an outbox. In the same database transaction that changes the user, insert an outbox row. A separate publisher reliably reads unsent rows and submits jobs, marking them sent only after success. This makes the committed business fact and the intention to notify part of one durable write.
Job design guidelines
Make jobs small and explicit
generate-monthly-report is a clear job type. do-everything-for-customer is a future debugging nightmare. A job should have one observable outcome and one owner.
Set a timeout and a visibility period
Workers can hang. The broker’s visibility timeout must be longer than normal job runtime but not so long that failed work stays hidden for hours. Measure actual duration and set a sensible limit.
Do not assume global ordering
Multiple workers improve throughput but can process two jobs for the same customer out of order. If order matters, use a partitioning key, a per-entity sequence check, or a design that tolerates reordering. “The queue is FIFO” rarely answers the whole system question.
Give users the right kind of answer
For a short task, return 202 Accepted and expose a status endpoint or notification. For a long export, create an export resource that moves through queued, running, ready, and failed. A request that only enqueues work has not completed the work.
Watch the queue like a product feature
Monitor queue depth, age of the oldest job, completion rate, retry rate, dead-letter count, and worker failure rate. Log job id, attempt, type, and the relevant entity id. An empty queue is not necessarily healthy if producers are failing to enqueue; compare it with the rate of incoming jobs.
Background work buys responsiveness and resilience, but it introduces an asynchronous boundary. Make delivery semantics, retries, duplicate handling, and user-visible status explicit. Then a queue becomes more than a performance trick—it becomes a dependable way to let the system keep working when the world is slow.
Decide whether a job belongs on the queue
Queue work when the user does not need its result before continuing, when it may take seconds or minutes, or when an external dependency deserves isolated retries. Image processing, notification fan-out, exports, and third-party synchronisation are good candidates.
Keep work synchronous when the user needs a definite answer now. Checking whether a project exists before creating a task should not become a background job. Nor should a job queue be used as a vague replacement for a transaction: if an operation must atomically reserve a seat and charge a card, the core consistency boundary needs a deliberate synchronous design first.
A useful final question is: if this job runs twice, runs late, or never runs, what will the user see? If the answer is clear—and your monitoring would reveal the last case—you have designed an asynchronous workflow rather than merely moved code to another process.