Offered by Dead Brands
Webhooks are how systems talk without asking permission first: something happens *there*, and a POST arrives *here*.
Use it over MCP: connect to https://mcp.deadbrands.co/hub/mcp and call skills_get with slug webhooks-and-event-driven-design.
# Webhooks and Event-Driven Design
Credit: SteTy — published free in the Skills Hub
Webhooks are how systems talk without asking permission first: something happens *there*, and a POST arrives *here*. This skill covers the full discipline — anatomy, HMAC signature verification, retries, duplicates, out-of-order delivery, idempotency, schema design, and when plain polling beats pushing.
## Overview
A webhook is a user-defined HTTP callback. Instead of your code polling "did anything change?", the provider sends an HTTPS POST to your endpoint the moment an event occurs — `payment.succeeded`, `build.finished`, `user.created`. The payload is usually JSON describing the event. The delivery guarantee across the industry is **at-least-once**: you will receive some events twice, some out of order, and some long after they happened (providers replay history). Exactly-once delivery over HTTP does not exist; exactly-once *processing* is something you build yourself with idempotency. Everything in this skill follows from those two facts.
## Core Principles
- **Assume duplicates, disorder, and delay.** Design the handler for the delivery you fear, not the one you hope for.
- **Verify before you trust.** An endpoint anyone can POST to will be POSTed to by everyone. Signature first, business logic second.
- **Acknowledge fast, process asynchronously.** Verify, persist, return 2xx in milliseconds; do the real work in a queue. Slow handlers time out, and timeouts trigger retries — which you then have to dedupe.
- **The webhook is a hint, not the truth.** For anything valuable (money, access), fetch the object's current state from the provider's API before acting. Payloads can be stale, replayed, or — if you skipped verification — forged.
- **Events are facts in the past tense.** Name them `invoice.paid`, never `pay_invoice`. Commands can be refused; facts just happened.
## Techniques
**Anatomy of a delivery.** A POST with headers carrying metadata and a JSON body carrying the event. The **Standard Webhooks** specification (from Svix, now widely adopted) fixes the header names: `webhook-id` (unique per message), `webhook-timestamp` (Unix seconds), `webhook-signature`. A typical event envelope:
```json
{ "id": "evt_9f2…", "type": "invoice.paid", "created": 1790000000,
"api_version": "2026-01-01", "data": { "object": { "id": "inv_123", "amount": 4200 } } }
```
**HMAC signature verification.** The sender computes `HMAC-SHA256(secret, signed_content)` and sends it in the signature header; you recompute and compare. Non-negotiable details:
- Verify against the **raw request body bytes**, exactly as received. Parsing the JSON and re-serializing changes key order, whitespace, and Unicode escaping — and every signature check fails. Capture the body before your framework's JSON parser touches it.
- The signed content conventionally includes the ID and timestamp: `"{webhook-id}.{webhook-timestamp}.{body}"` — so neither can be swapped between messages.
- Compare in **constant time** (`crypto.timingSafeEqual`, `hmac.compare_digest`). `==` leaks how many leading characters matched.
- Signatures are often versioned/prefixed, e.g. `v1,<base64>`; during **key rotation** the sender signs with both old and new secrets and sends both signatures — accept a match on any, then retire the old.
- Verify **before** writing any dedup record: otherwise an attacker can "reserve" a real event's ID with a forged request and block the genuine delivery.
**Replay protection.** A captured valid request stays valid forever unless you check freshness. Reject timestamps more than **5 minutes (300 s)** from your clock — the industry-standard tolerance, wide enough for clock skew and retries, narrow enough to matter. Timestamp check (stateless) + event-ID dedup (stateful) together defeat replays.
**Retry schedules (sender side).** No response, a timeout, or a 5xx means retry with exponential backoff; a 4xx means the receiver rejected it — most senders stop or treat it as permanent. Real schedules: **Stripe** retries live-mode events for up to **3 days** with exponential backoff (and shows every attempt in its dashboard); **GitHub does not automatically redeliver** failed deliveries — a human or your own code must replay them. Endpoints that fail persistently get auto-disabled by many providers, with a warning email first. As a sender: retry for hours-to-days, alert early, and give receivers a manual replay button.
**Idempotency and dedup (receiver side).** Keep a table of processed event IDs with a **unique constraint**. Flow: verify signature → insert event ID (constraint violation = duplicate → return 200, do no work) → store raw payload → enqueue processing → return 200. Critical subtleties: dedupe on the **event/message ID, never the signature** — each retry attempt is re-signed with a fresh timestamp and signature, but the ID stays constant. Answer duplicates with 2xx; answering an error makes the sender retry a thing you've already handled. Retain IDs at least as long as the sender's retry window (Stripe's 3 days → keep a week+).
**Ordering.** Arrival order is not generation order. Defenses, in increasing strength: (1) compare the event's timestamp/version against the stored record and ignore older updates; (2) upsert by the object's ID so `updated` before `created` still converges; (3) buffer briefly (5–10 s) and process in event-time order when sequence truly matters; (4) on any state-changing event, re-fetch the object from the API and reconcile. Run a periodic reconciliation job as the safety net — events get lost entirely too.
**Event schema design.** Stable envelope, versioned `type` names (`invoice.paid` vs breaking-change `invoice.payment_completed`), additive-only changes (new fields fine, renamed/removed fields are a new version), and include the object's ID even if you also embed a snapshot — snapshots go stale, IDs don't.
**Polling vs pushing.** Poll when: events are rare and latency-tolerant, the source sits behind NAT/firewalls that can't receive inbound calls, you need a full-state sync anyway, or call volume is tiny. Push when: latency matters, change rate is high relative to poll rate (polling would be 99% empty responses), or the provider charges per API call. Production systems usually do both: webhooks for speed, a scheduled reconciliation poll for truth.
## Worked Examples
*The handler, end to end.* POST arrives → read raw body → verify HMAC + 5-minute freshness (fail: 401, log, stop) → insert `evt_9f2` into dedup table (already there: return 200, stop) → save raw payload → push job → return 200 in 40 ms. Worker later processes: charge ledger updated once, even though the provider delivered the event three times.
*Out of order.* `subscription.updated` (plan=pro, event time 12:00:05) arrives before `subscription.created` (11:59:58). Naive code errors on the missing record. Version-guarded upsert creates the row from the update, then *ignores* the older create when it lands because its timestamp is earlier than the stored one. State converges.
*Duplicate with a new signature.* A network blip eats your 200; Stripe retries. Attempt 2 has a different `webhook-signature` and timestamp but the same `webhook-id`. Dedup by ID catches it; dedup by signature would have processed the payment twice.
## Common Pitfalls
- **Verifying the re-serialized JSON** instead of the raw body — the #1 cause of "signature always invalid."
- **Processing inline and timing out**, causing retry storms against your own slow endpoint.
- **Returning 500 for duplicates or unknown event types.** Unknown types: return 200 and ignore (the sender enabled events you don't consume). Duplicates: 200.
- **Trusting the payload for money movement** without re-fetching state — a replayed `payment.succeeded` from last month is still correctly signed.
- **Assuming the sender retries.** Some don't (GitHub). Know your provider's contract, and build replay/reconciliation for the gaps.
- **Secrets in code or shared across endpoints.** Per-endpoint signing secrets, stored in a secrets manager, rotatable without downtime.
## Practice Drills
1. Build a receiver implementing the full flow: raw-body capture, HMAC-SHA256 verification with constant-time compare, 300 s freshness window, dedup table. Send it a valid event, the same event again, a tampered-body copy, and a stale-timestamp copy; assert 200/200-no-op/401/401.
2. Write a sender with exponential-backoff retries (1 m, 5 m, 30 m, 2 h, then dead-letter). Point it at an endpoint that fails the first three times and log the attempt schedule.
3. Take an out-of-order event pair (`updated` before `created`) and implement the timestamp-guarded upsert so both arrival orders converge to identical state.
4. Break your own drill-1 receiver by parsing JSON before verifying. Observe the failure mode and write down why it happens.
## When to Use and Limits
- **Use it when:** a task calls for webhooks and event-driven design, as laid out in the Techniques and Worked Examples above.
- **Watch first for:** Verifying the re-serialized JSON; Processing inline and timing out; Returning 500 for duplicates or unknown event types. (see Common Pitfalls).
- **Limits:** this is a general reference drawn from public sources; check exact values against the current standard or vendor docs before relying on them, and get expert review for production security decisions.
## Quick Reference
| Concern | Rule |
|---|---|
| Delivery guarantee | At-least-once; build exactly-once processing yourself |
| Verify against | Raw body bytes, before JSON parsing |
| Signed content | `id.timestamp.body`, HMAC-SHA256, constant-time compare |
| Freshness window | ±5 minutes (300 s) |
| Dedup key | Event/message ID (stable across retries) — never the signature |
| Duplicate arrives | Return 2xx, do nothing |
| Ordering | Timestamp/version guard + upsert + re-fetch; reconcile on a schedule |
| Response time | 2xx in well under the sender's timeout; process async |
| Sender retries | Backoff for hours–days on 5xx/timeout; stop on 4xx |
---
*Trademarks and copyrights mentioned belong to their respective owners. Educational reference only; not affiliated with or endorsed by them.*