Offered by Dead Brands
How to make a running system answer questions at 3 AM: structured logs you can query, the metric frameworks that guarantee coverage (the four golden signals, RED, USE), distributed traces that show wh
Use it over MCP: connect to https://mcp.deadbrands.co/hub/mcp and call skills_get with slug observability-logs-metrics-traces.
# Observability: Logs, Metrics, Traces
Credit: SteTy — published free in the Skills Hub
How to make a running system answer questions at 3 AM: structured logs you can query, the metric frameworks that guarantee coverage (the four golden signals, RED, USE), distributed traces that show where time went, and alerting discipline — page on symptoms users feel, never on causes machines notice.
## Overview
**Monitoring** answers questions you knew in advance ("is error rate above 1%?") with prebuilt metrics, dashboards, and alerts. **Observability** is the property of being able to ask *new* questions of a running system without shipping new code ("why is p99 slow only for checkout, only in one region, only since the 2 PM deploy?"). You build it from three telemetry types that share one identity: **metrics** (numbers over time — cheap, aggregable, alertable), **logs** (discrete events with detail), and **traces** (one request's path across services, timed hop by hop). Metrics tell you *that* something is wrong; traces tell you *where*; logs tell you *why*. If they don't share an ID, you own three silos instead of a system.
## Core Principles
- **One request, one ID, everywhere.** A correlation/trace ID is generated at the edge and propagated on every call (HTTP headers, gRPC metadata, message attributes — the W3C `traceparent` header is the standard carrier). Every log line, span, and metric exemplar carries it. This single discipline is worth more than any tool purchase.
- **Alert on symptoms, not causes.** Page on what users experience — request success rate, latency against the SLO — and treat CPU, heap, and disk as *diagnostics* you consult after the page. A green infrastructure dashboard and a failing user journey coexist happily; only one of them should wake a human.
- **Distributions, not averages.** Average latency hides the tail your angriest users live in. Track p50/p95/p99, and track latency of *failed* requests separately — a fast error storm and a slow success path are different incidents.
- **Every dashboard answers a question.** If a panel doesn't map to a decision ("if this moves, I do X"), it's decoration consuming attention. Curate ruthlessly.
- **Telemetry has a budget.** High-cardinality labels and log-everything pipelines cost real money and can drown the signal. Sample deliberately; keep the errors and the slow ones.
## Techniques
**Structured logging.** Log JSON events, not prose: `{"ts":"...","level":"ERROR","service":"checkout","trace_id":"4bf9...","route":"/pay","status":504,"duration_ms":842,"error_code":"UPSTREAM_TIMEOUT","msg":"payment call timed out"}`. Stable field names and a stable `error_code` vocabulary make logs queryable and countable. Levels with intent: **ERROR** = something failed and needs attention in aggregate; **WARN** = handled abnormality; **INFO** = business-meaningful state changes (order placed, job finished), not loop iterations; **DEBUG** = off in production by default, enableable per-service. Log the *boundaries* of a unit of work — initiated / succeeded / failed, each with the correlation ID and duration — and those three event shapes alone can compute success rate and latency percentiles per operation. Two disciplines: **never log secrets or personal data** (tokens, passwords, full card numbers — redact at the logger), and beware context leakage — in thread-pooled servers, an uncleared request context will eventually mislabel someone else's request with your ID.
**The metric frameworks.** Three complementary checklists guarantee you didn't forget a dimension:
- **Four Golden Signals** (Google SRE) for any user-facing system: **Latency** (distribution, successes and failures apart), **Traffic** (demand: requests/s, messages consumed), **Errors** (rate of failed requests — explicit 5xx, implicit wrong content, and policy failures like "slower than the SLO"), **Saturation** (how full: queue depth, pool usage, memory pressure). Saturation is the leading indicator; the other three mostly trail.
- **RED** (Tom Wilkie, 2015) for every request-driven service: **Rate, Errors, Duration**. It's the golden signals minus saturation, from the caller's point of view — three panels per service, uniform across the fleet.
- **USE** (Brendan Gregg) for every finite resource — host, disk, connection pool, queue: **Utilization** (fraction of time busy), **Saturation** (work queued beyond capacity), **Errors** (fault count). If RED says the service is slow, USE tells you which resource underneath is the constraint.
Rule of thumb: RED per service, USE per resource, golden signals as the umbrella review.
**Distributed tracing.** A **trace** is the tree of **spans** for one request: each span is a named, timed unit of work with a start, duration, status, and attributes (route, status code, attempt, error code). The root span is the inbound request; child spans cover each downstream call, query, and queue hop, all sharing the trace ID, each with its own span ID and a parent pointer. Propagation is the whole game — an unpropagated hop breaks the tree into confetti. Read a trace waterfall to find where time concentrates: the long span is the suspect; its attributes usually name the cause. Practice details:
- **Name spans for humans** ("inventory.check", "payment.authorize") around units of work a person would name — a few meaningful custom spans beat forty auto-generated ones nobody reads mid-incident.
- **Sample with intent**: head-based sampling (keep 1%) controls cost but throws away most errors; **tail-based sampling** — decide after the trace completes, keeping all errors and slow traces — is what you actually want during incidents.
- **Link, don't duplicate**: logs carry `trace_id`/`span_id` so one click moves from a slow span to its log lines.
**SLIs, SLOs, and alerting.** An **SLI** is a measured ratio (successful requests / total requests; requests under 300ms / total). An **SLO** is its target (99.9% over 30 days), which implies an **error budget** (0.1% may fail) — budget remaining licenses shipping speed; budget burning fast means stop and fix. Alert on **burn rate**: how fast the budget is being consumed relative to the window. A 2% budget burned in an hour pages; the same total spread over a month doesn't. Measure SLOs per meaningful operation (per API, per journey) — one global average can pass while your worst endpoint breaches alone.
**Dashboards that answer questions.** One service dashboard, top to bottom in incident order: (1) the golden signals / RED row — is it us, and which signal moved; (2) saturation of its key resources (USE); (3) change markers — deploys and config changes annotated on every graph, because "what changed?" is the first question of every incident; (4) top error codes and slowest routes. During an incident the workflow is metrics → exemplar trace → its logs, minutes instead of an hour of grep-and-guess.
## Worked Examples
1. **Checkout slow at flash-sale time.** Golden signals: latency p99 up 4×, errors climbing, traffic 10× normal, saturation normal. A trace shows 800ms+ inside a fraud-check span for high-value orders — exactly the orders a flash sale floods. Logs filtered by that trace ID confirm timeouts. Root cause found in ~12 minutes; without traces, the same incident is two hours of restarting healthy services.
2. **The alert that cried wolf.** A team pages on "CPU > 80%". It fires most afternoons; users are fine (batch work, by design); the team starts ignoring pages — including the one that mattered. Replaced with an SLO burn-rate alert on checkout success, CPU demoted to a diagnostic panel. Pages drop to a handful a month, all real.
3. **One request, found.** A user reports "my upload failed at 14:03." With a correlation ID in the error response, support searches one ID and gets the exact log lines across three services and the failing span — instead of timestamp-guessing across hundreds of machines.
## Common Pitfalls
- Logging prose strings, then trying to count or filter them in an incident. Structure first, or it isn't data.
- Averages everywhere; p99 unmeasured until a customer measures it for you.
- Alerting on causes (CPU, disk, GC) — noisy, usually unactionable, trains people to ignore the pager.
- IDs that stop at service boundaries: logs correlated within a service, blind between them.
- Cardinality explosions: user IDs or raw URLs as metric labels — a metrics bill and a useless dashboard in one move. Labels are for bounded sets (route template, status class, region).
- Sampling away the evidence: 1% head sampling keeps 1% of your rare errors too.
- Dashboards as metric museums — forty panels, no question, no owner, no threshold.
## Practice Drills
1. Take one endpoint and write its three boundary log events (initiated/succeeded/failed) with the fields needed to compute its success rate and p95 from logs alone.
2. Build the RED row for a service and the USE row for its database pool; state which panel you'd open first for "slow but no errors" and why.
3. Given a trace waterfall of five spans, identify the critical path and the span whose optimization actually moves p99.
4. Rewrite three cause-based alerts (CPU, disk, thread count) as symptom-based SLO alerts with burn-rate windows.
## When to Use and Limits
- **Use it when:** a task calls for observability: logs, metrics, traces, as laid out in the Techniques and Worked Examples above.
- **Watch first for:** (see Common Pitfalls).
- **Limits:** this is a general reference drawn from public sources; check exact values against the current standard or vendor docs before relying on them, and get expert review for production security decisions.
## Quick Reference
- Metrics = that, traces = where, logs = why — joined by one propagated trace ID.
- RED per service (Rate, Errors, Duration). USE per resource (Utilization, Saturation, Errors). Golden signals overall (Latency, Traffic, Errors, Saturation).
- Structured JSON logs; levels with intent; no secrets; correlation ID on every line.
- Page on SLO burn rate and user symptoms. Causes are for dashboards, not pagers.
- Latency in percentiles; failed-request latency tracked separately.
---
*Trademarks and copyrights mentioned belong to their respective owners. Educational reference only; not affiliated with or endorsed by them.*