Offered by Dead Brands
How to move work out of the request path and into reliable asynchronous processing: the producer/queue/worker model, why delivery is at-least-once and exactly-once is a property you build, idempotent
Use it over MCP: connect to https://mcp.deadbrands.co/hub/mcp and call skills_get with slug queues-and-background-jobs.
# Queues and Background Jobs
Credit: SteTy — published free in the Skills Hub
How to move work out of the request path and into reliable asynchronous processing: the producer/queue/worker model, why delivery is at-least-once and exactly-once is a property you build, idempotent consumers, visibility timeouts, retries with backoff, dead-letter queues, and what ordering guarantees really cost.
## Overview
A queue decouples *asking* for work from *doing* it. A **producer** enqueues a message (a job: "resize this image", an event: "order 4472 was placed"); a **broker** stores it durably; **workers** (consumers) pull messages, do the work, and **acknowledge (ack)**. This buys three things: latency (the web request returns immediately), resilience (a traffic spike becomes a backlog, not an outage), and independent scaling (add workers, not web servers). It also buys a new failure physics — duplicates, reordering, poison messages — that this skill teaches you to design for instead of discover.
## Core Principles
- **Delivery is at-least-once by default.** Brokers delete a message only after an ack. Anything that can interrupt "do the work, then ack" — a crash, a network blip, a timeout — produces a redelivery. Duplicates are not an edge case; they are the normal cost of never losing work.
- **Exactly-once delivery is impossible across two systems; exactly-once *effect* is achievable.** The work and the ack live in different systems, and a crash can always land between them — no protocol closes that gap. What you can build is an exactly-once effect: at-least-once delivery plus a consumer that makes reprocessing harmless.
- **Therefore every consumer must be idempotent.** Processing the same message twice must produce the same result as processing it once. This is the single most important sentence in queue work.
- **Acks come last.** Ack only after the side effect is durably done. Ack first and you've chosen at-most-once: fine for typing indicators and presence beacons, never for anything financial.
- **Failure is a classification problem.** Transient (downstream 503), throttling (429), not-yet (referenced row hasn't arrived), permanent (malformed payload, failed validation). Each class gets a different response; retrying them all the same way is how queues die.
## Techniques
**Visibility timeout.** When a worker takes a message, it becomes invisible to other workers for a window (Amazon SQS default: 30s). Finish and delete it inside the window, or it reappears and gets redelivered. Set it **longer than your slowest legitimate run**, with headroom — a job that takes 40s under a 30s timeout runs twice concurrently, every time. Long jobs should extend ("heartbeat") the timeout while working. If the timeout is huge, a crashed worker's message waits that long to retry — balance both pains.
**Idempotent consumers.** The standard toolkit:
- **Idempotency key**: a stable ID per logical operation, derived from the business event (e.g., `order-4472:rider-assignment`), never minted at receive time — a redelivery must reproduce the same key.
- **Dedup record**: before or with the work, claim the key with an atomic conditional insert (a database `UNIQUE` constraint is the classic). If the claim fails, this is a duplicate: skip the work, ack, move on. Give dedup records a TTL (e.g., 24h+) so the store doesn't grow forever.
- **Naturally idempotent operations**: prefer upserts (`SET balance = 500`) over increments (`balance += 500`), absolute state writes over deltas, and pass the same idempotency key onward to downstream APIs that accept one (payment providers do) so a duplicated call collapses to one charge.
- **Ordering the write and the side effect**: record the dedup key and the state change in one transaction (the **transactional outbox** pattern: write the outgoing event in the same transaction as the state change, publish from the outbox after) so a crash can't leave one without the other.
**Retries with backoff.** Retry transient failures with **exponential backoff plus full jitter**: delay ≈ random(0, min(cap, base × 2^attempt)) — e.g., ~30s, 2m, 8m, 32m. Jitter is not decoration: when a dependency recovers, ten thousand jobs whose timers expire in the same second will knock it straight back over. Rules:
- Honor `Retry-After` on 429s instead of your own schedule.
- Cap **attempts and total elapsed time**; retry at **one level only** (a client retrying 3× inside a job retrying 5× is 15 calls nobody planned).
- Never retry a permanent failure — a malformed payload will be malformed forever. Classify, don't hope.
- A circuit breaker per dependency stops the flood while it's down.
**Dead-letter queues (DLQ).** After a configured number of failed deliveries (e.g., `maxReceiveCount: 5`), the broker routes the message to a separate DLQ instead of retrying forever. This is the escape hatch for **poison messages** — messages that deterministically crash or fail the consumer and would otherwise loop forever, blocking or starving everything behind them. Operating a DLQ is a workflow, not a destination:
- **Alert on depth > 0.** A DLQ nobody watches is a delete queue with extra steps — the most common DLQ failure.
- Preserve context with the message: error, attempt count, timestamps, correlation ID. Triage shouldn't mean reproducing from scratch.
- Match types (a FIFO source needs a FIFO DLQ), set retention to the maximum, and **redrive** (replay) only after the fix ships — replaying into the same bug just refills it.
**Ordering guarantees.** Standard queues are typically **best-effort ordering**: usually in order, never promise it. FIFO queues guarantee order *per group* (per ordering key — messages for one order stay sequenced; unrelated orders still parallelize), usually at lower throughput and with broker-side dedup windows. True global ordering forces single-threaded consumption and is almost never worth it. Design consumers to tolerate reordering anyway (check versions/timestamps: "apply this update only if it's newer than the state I hold") — networks will eventually reorder something you depend on.
**Scaling and backpressure.** Workers scale horizontally; the queue absorbs the difference between arrival rate and processing rate. Watch **queue depth and age of the oldest message** — age is the user-facing number ("emails are 9 minutes behind"). If producers outrun consumers forever, the answer is more workers, faster work, or shedding load — the queue only buys time.
## Worked Examples
1. **The double rider.** A consumer assigns rider Alex to order 4472, saves it, then crashes *before* the ack. Thirty seconds later the message is redelivered; a second worker assigns Sam. Two riders, one order — pure at-least-once physics. Fix: idempotency key `order-4472:rider` claimed in the same transaction as the assignment; the second delivery sees the claim and acks without acting.
2. **The poison email job.** One job references a template variable the renderer can't parse; it throws instantly, redelivers, throws — 40,000 attempts in a day, workers burning on it while real mail waits. With `maxReceiveCount: 5` it lands in the DLQ, an alert fires on DLQ depth, the producer bug is fixed, and the single message is redriven cleanly.
3. **Retry storm.** A payment provider has a 2-minute outage; 5,000 queued jobs all fail and all retry at a fixed 60s — the synchronized second wave trips the provider's rate limiter for an hour. With capped exponential backoff and full jitter, retries spread across minutes, the provider recovers once, and the backlog drains.
## Common Pitfalls
- Designing for exactly-once and skipping idempotency "because our queue guarantees it." It doesn't, past its own borders.
- Minting the idempotency key when the message is *received* — every redelivery gets a fresh key and dedup never fires.
- Visibility timeout shorter than job runtime: guaranteed concurrent duplicate execution.
- Retrying permanent errors with the same policy as transient ones; retrying at multiple layers multiplicatively.
- Enqueueing work the request should have validated first — a queue is not a schema.
- A DLQ with no alert, short retention, or no redrive procedure.
- Assuming queue order in a standard queue, or paying for global FIFO when per-entity order was the real requirement.
- Giant payloads in messages: pass an ID and fetch the data; messages should be small, durable, and self-describing (type, version, key, correlation ID).
## Practice Drills
1. Trace one message through crash-before-ack and list exactly which side effects happen twice; then add the dedup claim and re-trace.
2. Classify ten failure modes for a webhook-delivery worker into transient / throttling / not-yet / permanent, and assign each a policy.
3. Compute a backoff schedule: base 10s, factor 2, cap 10 min, full jitter — write the delay ranges for attempts 1–6.
4. Set a visibility timeout for a job with p50 runtime 20s, p99 3 min; justify the number and the heartbeat plan.
## When to Use and Limits
- **Use it when:** a task calls for queues and background jobs, as laid out in the Techniques and Worked Examples above.
- **Watch first for:** (see Common Pitfalls).
- **Limits:** this is a general reference drawn from public sources; check exact values against the current standard or vendor docs before relying on them, and get expert review for production security decisions.
## Quick Reference
- Assume at-least-once. Ack after the work. Make the work idempotent.
- Idempotency key: stable, business-derived, claimed atomically, TTL'd.
- Visibility timeout > slowest run; heartbeat long jobs.
- Retries: exponential backoff + full jitter, capped, classified, one layer.
- DLQ: maxReceiveCount, alert on depth > 0, preserve context, redrive after the fix.
- Order: per-entity FIFO if you must; design for reordering regardless.
---
*Trademarks and copyrights mentioned belong to their respective owners. Educational reference only; not affiliated with or endorsed by them.*