Observability

Observability for system design interviews, from metrics, logs and traces to percentiles, SLOs, burn-rate alerts and telemetry pipelines at scale.

Mid-level33 min core · 38 min full depth

Read first: Estimation

Observability answers two questions about a running system, from the outside. Is it working for its users? And if not, which part is broken? It is built from three things:

Three kinds of telemetry: metrics, logs and traces.
A promise about what “working” means: an SLO.
Alerts that wake someone only when that promise is in danger.

The metrics and monitoring problem designs the metrics backend itself, at 100 million active series.

The problem it solves

A checkout platform runs about 300 services on 2,000 nodes and takes 12,000 requests a second at the edge. At 14:02 a deploy of the payments client starts failing 10% of checkouts. What happens next depends entirely on the telemetry:

Without	With
A customer posts about it at 14:22. Support escalates at 14:40.	A burn-rate alert pages the checkout on-call at about 14:11.
Engineers log into hosts and grep files that rotate every hour.	The SLO dashboard shows errors on one route, one version, every zone.
Nine teams each say their service looks fine.	A trace of a failed request shows the 503 coming from the payments call.
The fix ships at 15:30. Nobody knows how many orders were lost.	Rolled back at 14:21. The error budget says exactly how much it cost.

So observability has two jobs:

Prove the system meets its promises. Measure what users experience (availability, latency, freshness) against a target, continuously, and know how much slack is left.
Find the broken component fast. Go from “something is wrong” to “this dependency, this version, this zone” in minutes. Time to detect (MTTD) plus time to repair (MTTR) is how long users suffer.

Monitoring answers questions you knew to ask in advance: dashboards, and alerts on known failure modes. Observability lets you ask new questions during an incident, without shipping code. That needs telemetry rich enough to slice by route, version, tenant or single request.

Each signal answers a different question and costs money in a different way:

Signal	Answers	Unit	Cost grows with	High-cardinality data
Metrics	Is it broken, how much, since when, where	A number per series per scrape	Number of distinct series	No: it multiplies series
Logs	What exactly happened in this event	One record per event	Event volume × bytes × retention	Yes
Traces	Where the time went across services	A tree of spans per request	Requests × spans × sample rate	Yes

How it works

Step 1: Count things: metrics

The cheapest useful telemetry is a counter in memory. Each request increments http_requests_total, with a few labels. The process exposes the current values on /metrics. A collector reads them every 15 to 30 seconds. One sample is a timestamp and a float: 16 bytes raw. After compression it’s about 1.4 bytes, using the delta-of-delta and XOR scheme Facebook described for Gorilla.

Three metric types cover nearly everything:

Type	What it holds	Example	Read it with
Counter	A total that only goes up (resets on restart)	Requests, errors, bytes sent	`rate()` over a window: per second
Gauge	A value that goes up and down	Queue depth, memory in use, open connections	The value itself, or `max` over a window
Histogram	Counts of observations per bucket, plus sum	Request latency, payload size	`histogram_quantile()` over summed rates

# TYPE http_requests_total counter
http_requests_total{route="/checkout",method="POST",code="200"} 1843302
http_requests_total{route="/checkout",method="POST",code="503"} 911
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{route="/checkout",le="0.1"} 1520011
http_request_duration_seconds_bucket{route="/checkout",le="0.25"} 1790320
http_request_duration_seconds_bucket{route="/checkout",le="+Inf"} 1844213

A counter is a running total for a reason. If a scrape is missed, the next one still has the full count, so rate() loses resolution, not data. A gauge of “requests in the last 10 seconds” would lose every request in a missed window. Counter resets on restart are detected by the value going down and handled by rate().

Step 2: Labels multiply

Each distinct combination of metric name and label values is a series, stored and indexed separately. The count is up to the product of the label value counts:

Label	Distinct values	Running series count
bucket (`le`), plus sum, count	14	14
route	40	560
method	4	2,240
status code	8	17,920
pod	200	3.6 million (bound)

Real data is sparser. But one latency histogram on one service is easily a million series. A time-series database such as Prometheus keeps every active series in memory. Each one takes roughly a few kilobytes, including its index. So one server typically tops out at a few million.

Now add user_id as a label. Every active user becomes 14 new series, on every pod that served them. That’s tens of millions on one deploy, and the backend runs out of memory. The metrics and monitoring problem covers the limits that contain this.

The rule: a label’s values must be few, bounded and known in advance. The route template /users/:id is a label. The raw path /users/48151623 is not. User ids, request ids, emails and error messages belong in logs and traces, which are built for them. To link the two, a histogram bucket can carry an exemplar. That’s the trace id of one request that landed in it. Then a latency spike on a dashboard is one click from a real slow request.

Step 3: Percentiles, and why you cannot average them

Averages hide the tail. Say 99 requests take 20 ms, and one takes 3 s. The mean is 50 ms, and looks healthy. Meanwhile, one user in a hundred waits three seconds. Percentiles describe the distribution: p50 is the typical request, and p99 is the slowest 1%.

The tail matters more than it seems, because one page load fans out to many backend calls. Take 30 calls, each with an independent 1% chance of hitting p99. Then 1 − 0.99^30 ≈ 26% of page loads include at least one p99 call.

There are two ways to get a percentile from a fleet, and only one of them adds up:

A summary computes quantiles inside each process. It is accurate for that process, and useless for the fleet: there is no correct way to combine two pods’ p99s.
A histogram counts requests per fixed bucket. Buckets with the same bounds add up across pods, zones and time, and the percentile is estimated once, from the sum.

Two pods behind one load balancer, over the same five minutes:

Pod	Requests	Its p99	What is going on
A	9,000	119 ms	Healthy
B	1,000	2,450 ms	On a bad host: 300 requests at 1 to 2.5 s
Average	-	1,284 ms	Not the p99 of anything
Merged	10,000	2,000 ms	The real p99: 3% of requests take 1 to 2.5 s

The average weighs a pod with 1,000 requests the same as one with 9,000. It can be off in either direction, and no dashboard should ever show “average of p99”. Merge first, then estimate:

histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))

Histograms.javaexample.pyHistograms.csmain.goexample.ts

import java.util.Arrays;

final class Histograms {
    // Bucket upper bounds in ms, shared by every pod. Fixed, shared bounds are what make
    // histograms mergeable: the same bucket on two pods means the same range.
    static final int[] BOUNDS_MS = {10, 25, 50, 100, 250, 500, 1000, 2500, 5000};

    // Estimate a quantile from bucket counts, the way Prometheus's histogram_quantile does.
    static double quantile(double q, long[] counts) {
        double rank = q * Arrays.stream(counts).sum();
        long seen = 0;
        for (int i = 0; i < counts.length; i++) {
            if (seen + counts[i] >= rank) {
                double lower = i > 0 ? BOUNDS_MS[i - 1] : 0;
                // Assume requests are spread evenly inside the bucket. The error is bounded by the
                // bucket's width, which is why bounds should be dense near the SLO threshold.
                return lower + (BOUNDS_MS[i] - lower) * (rank - seen) / counts[i];
            }
            seen += counts[i];
        }
        return BOUNDS_MS[BOUNDS_MS.length - 1];
    }

    // A latency SLI as a ratio of good requests: exact when the limit is a bucket bound.
    static double shareUnder(int limitMs, long[] counts) {
        int i = Arrays.binarySearch(BOUNDS_MS, limitMs);
        return (double) Arrays.stream(counts, 0, i + 1).sum() / Arrays.stream(counts).sum();
    }
}

class Main {
    public static void main(String[] args) {
        // Requests per bucket over the same 5 minutes. Pod B is the one on a bad host.
        long[] podA = {2000, 4000, 2200, 700, 80, 15, 5, 0, 0}; // 9,000 requests
        long[] podB = {100, 200, 200, 150, 40, 10, 0, 300, 0}; // 1,000 requests, 300 of them at 1 to 2.5 s

        double p99A = Histograms.quantile(0.99, podA), p99B = Histograms.quantile(0.99, podB);
        System.out.println(Math.round(p99A) + " " + Math.round(p99B)); // 119 2450
        // Wrong: a mean of two p99s ignores that pod A served 9 times the traffic. It is not
        // the p99 of anything.
        System.out.println(Math.round((p99A + p99B) / 2)); // 1284
        // Right: add the buckets first (sum by le), then estimate once.
        long[] merged = new long[podA.length];
        for (int i = 0; i < merged.length; i++) merged[i] = podA[i] + podB[i];
        System.out.println(Math.round(Histograms.quantile(0.99, merged))); // 2000: 3% of all requests take 1 to 2.5 s
        System.out.println(Histograms.shareUnder(500, merged)); // 0.9695: the share under 500 ms, the SLI that adds up exactly
    }
}

The estimate assumes requests are spread evenly inside a bucket. So its error is bounded by the bucket’s width. Put bounds close together near the latency you care about. Better still, define the latency SLI as a ratio: “the share of requests faster than 500 ms”. Make 500 ms a bucket bound. Then it is an exact count, and ratios of counts add up perfectly.

Go deeperSketches, native histograms and coordinated omission2 min

Fixed buckets trade accuracy for mergeability. Two alternatives keep both:

Exponential histograms choose buckets that grow by a constant factor. So the relative error is bounded everywhere, without picking bounds by hand. Prometheus native histograms (added experimentally in 2.40) and OpenTelemetry’s exponential histogram work this way.
Quantile sketches such as t-digest and DDSketch are small, mergeable summaries with a stated error. DDSketch guarantees a relative error, for example 1%, on every quantile. They are probabilistic structures, in the same family as HyperLogLog.

Coordinated omission is a measurement bug, not a storage one. Some load generators send the next request only after the previous one returns. They stop sending while the server is stalled. So a 2 s stall shows up as one slow request, instead of the hundreds that would have queued behind it. Load tests should send at a fixed rate, and measure from the intended send time. HdrHistogram’s tooling corrects for it.

Step 4: Record events: structured logs

A metric says the error rate is 3%. A log says what happened in one specific failure. Logs should be structured: one JSON object per event with consistent field names, so the log store can filter and aggregate without regular expressions.

{
  "ts": "2026-10-28T14:02:17.412Z",
  "level": "error",
  "service": "checkout",
  "version": "4.18.2",
  "route": "/checkout",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "user_id": "u_48151623",
  "msg": "payment authorization failed",
  "upstream": "payments",
  "status": 503,
  "duration_ms": 2004
}

The trace_id field is the most valuable one on the line. It joins the log to the trace, and to every other log of the same request.

Logs are the most expensive signal, because volume scales with traffic. Take 12,000 requests a second, five lines per request, and about 1 KB per line. That’s 60 MB a second, or about 5 TB a day. If a managed service charges around $0.50 per GB ingested, that’s roughly $2,500 a day. (List prices vary several-fold by vendor and tier.) The controls:

Log less on the happy path. Keep every warning and error, sample successful requests at 1%, and keep debug logs off in production.
Index less. A full-text index can be as large as the logs. Indexing a few labels and scanning compressed chunks is far cheaper.
Tier retention. One to two weeks searchable, then object storage if audits need it.
Never block on logging. Write through a bounded buffer that drops (and counts drops) when full. Redact secrets and personal data before the line leaves the host.

Step 5: Follow one request: traces

A checkout that takes 2.4 s passes through nine services. Metrics show that each service’s latency went up a little. None shows where this request spent its time. A trace does. It is a tree of spans, one per operation. Each span has a trace id shared by the whole request, its own span id, its parent’s span id, a start time, a duration, attributes and a status.

The trace survives across services through context propagation. Every outgoing call carries the current trace id and span id, and the receiver creates a child span. The W3C Trace Context standard defines the header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                └ flags: 01 = sampled
             │  │                                └ parent span id, 8 bytes
             │  └ trace id, 16 bytes
             └ version

A companion tracestate header carries vendor data, and W3C Baggage carries application key-values (tenant, experiment) to every downstream service. Through a queue, the same header travels in the message’s headers, so a consumer’s span joins the producer’s trace.

One trace across three services and a queue

Scroll or pinch to zoom, drag to pan

Color key
Service
Queue or stream

SpanContext.javaspan_context.pySpanContext.csspan_context.gospan-context.ts

import java.security.SecureRandom;
import java.util.HexFormat;
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

// traceId: 16 bytes, the whole request across every service. spanId: 8 bytes, this one
// operation; it becomes the parent id downstream. sampled: the caller's sampling decision, so
// one trace is kept or dropped whole.
record SpanContext(String traceId, String spanId, boolean sampled) {
    // version-traceid-parentid-flags, all lowercase hex (W3C Trace Context).
    private static final Pattern PATTERN =
            Pattern.compile("^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$");
    private static final SecureRandom RANDOM = new SecureRandom();

    static String randomHex(int bytes) {
        byte[] b = new byte[bytes];
        RANDOM.nextBytes(b);
        return HexFormat.of().formatHex(b);
    }

    static Optional<SpanContext> parse(String header) {
        Matcher m = PATTERN.matcher(header.strip());
        // Version ff is forbidden, and all-zero ids are invalid. A bad header starts a new
        // trace instead of failing the request: telemetry must never break traffic.
        if (!m.matches() || m.group(1).equals("ff")
                || m.group(2).chars().allMatch(c -> c == '0') || m.group(3).chars().allMatch(c -> c == '0')) {
            return Optional.empty();
        }
        return Optional.of(new SpanContext(m.group(2), m.group(3), (Integer.parseInt(m.group(4), 16) & 1) == 1));
    }

    static SpanContext newRoot(boolean sampled) {
        // The first service to see a request (usually the edge) starts the trace and, with head
        // sampling, makes the keep-or-drop decision for every service after it.
        return new SpanContext(randomHex(16), randomHex(8), sampled);
    }

    SpanContext child() {
        // Same trace, new span id. The child's parent is this span, which is how the trace
        // store rebuilds the tree.
        return new SpanContext(traceId, randomHex(8), sampled);
    }

    String header() {
        return "00-" + traceId + "-" + spanId + "-" + (sampled ? "01" : "00");
    }
}

class Main {
    public static void main(String[] args) {
        SpanContext incoming = SpanContext.parse("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01").orElseThrow();
        System.out.println(incoming.traceId() + " " + incoming.sampled()); // 4bf92f3577b34da6a3ce929d0e0e4736 true
        // Our span for this request, and the header we send on every downstream call.
        System.out.println(incoming.child().header()); // 00-4bf92f3577b34da6a3ce929d0e0e4736-<new span id>-01
        System.out.println(SpanContext.parse("00-00000000000000000000000000000000-00f067aa0ba902b7-01")); // Optional.empty
        System.out.println(SpanContext.parse("garbage").orElseGet(() -> SpanContext.newRoot(false)).header()); // 00-<new trace id>-<new span id>-00
    }
}

Tracing every request is expensive. 12,000 requests a second × 20 spans × about 500 bytes is 120 MB a second, or roughly 10 TB a day. So traces are sampled:

Head sampling decides at the first service, and passes the decision downstream in the flags. It’s cheap. But at 1%, it throws away 99% of the errors, because it decides before anyone knows how the request ends.
Tail sampling buffers every span at a collector for a fixed wait (about 30 s). Then it keeps every error and slow trace, plus 1% of the rest.

The options below compare them.

Step 6: Decide what to watch: RED, USE and the golden signals

Three checklists decide what to measure:

Method	For	Measure	Origin
RED	Every request-driven service	Rate, Errors, Duration (as a histogram)	Tom Wilkie
USE	Every resource: CPU, disk, pool, queue	Utilization, Saturation, Errors	Brendan Gregg
Golden signals	User-facing systems	Latency, Traffic, Errors, Saturation	Google’s SRE book

They overlap on purpose. RED tells you a service is unhappy. USE tells you which resource behind it is the cause.

Saturation is the one teams forget: how full is the thing that will run out first? Examples are a connection pool at 95% of its size, a thread pool with a queue, consumer lag on a message queue, or a disk at 85%. Saturation rises before latency does. That makes it the best early warning on a dashboard.

Measure latency for successful and failed requests separately. A fast 500 pulls the latency line down, and makes an outage look like an improvement.

Step 7: Promise a number: SLIs, SLOs and error budgets

An SLI (service level indicator) is a ratio of good events to valid events, measured as close to the user as you can:

Availability: requests that did not fail with a 5xx or time out, over all requests, measured at the load balancer.
Latency: requests faster than 500 ms, over all requests.
Freshness, for pipelines: the share of minutes in which the newest data is less than 2 minutes old.

An SLO (objective) is a target for that ratio over a window: 99.9% of checkout requests succeed over 30 days. An SLA is a contract with penalties, set looser than the SLO so the team reacts before money is owed. The error budget is what the SLO allows to fail:

SLO	Budget, share of requests	Full outage allowed in 30 days
99%	1%	7.2 hours
99.9%	0.1%	43.2 minutes
99.95%	0.05%	21.6 minutes
99.99%	0.01%	4.3 minutes

The budget turns reliability into a decision. With budget left, ship features and take risks. When it is spent, an agreed policy kicks in. Freeze risky launches, and spend engineering time on reliability until the window recovers. That is also why 100% is the wrong target. It leaves no room to deploy. And through their own flaky networks, users cannot tell 100% from 99.999% anyway.

Measure the SLI where a dead process still counts. If the checkout service crashes, its own /metrics endpoint dies with it and reports nothing. The load balancer still sees every request fail.

Step 8: Page on burn rate, not on thresholds

The obvious alert, “page if the 5 minute error ratio is above 0.1%”, fails both ways. It pages for a 3 minute blip that spent a thousandth of the budget, and it is noisy enough that people start ignoring it. An alert should page only when the budget is in danger.

Burn rate is how fast the budget is being spent. It’s relative to the rate that would spend it exactly over the SLO window. Burn rate 1 spends a 30 day budget in 30 days. An error ratio of 1.44% against a 99.9% SLO is a burn rate of 14.4. That spends 2% of the month’s budget every hour, and all of it in about two days.

The Google SRE workbook recommends several rules. Each one uses two windows:

A long window that proves the burn is significant.
A short window, one twelfth as long, that proves it is still happening.

Burn rate	Long window	Short window	Budget spent when it fires	Response
14.4	1 hour	5 minutes	2%	Page
6	6 hours	30 minutes	5%	Page
1	3 days	6 hours	10%	Ticket

The fast rule catches outages within minutes. The slow page catches a burn that would empty the budget in five days. The ticket catches a slow leak that can wait for working hours.

The short window fixes the reset time. Without it, a 1 hour window keeps the page firing for most of an hour after the fix. How fast a rule detects follows from the numbers. At a 100% error ratio, the 1 hour rule reaches 14.4 × 0.1% after 60 min × 14.4 × 0.001 ≈ 52 seconds.

Inject incidents, and watch which rules fire, when, and what they cost. The SLO is 99.9% over 30 days, and the baseline error ratio is 0.02%. A short blip of 1.5% errors arrives every 9 to 13 hours, the way a garbage collection pause or a bad pod does:

Error ratioforRequire the short windowMetrics stop arriving

Error ratio (log scale)

100%1%SLO 0.1% = 1×0.6% = 6×1.44% = 14.4×

Budget left (30 days)

100%

Static: 5 m ratio above 0.1%

1 h and 5 m above 14.4×

6 h and 30 m above 6×

3 d and 6 h above 1×

No samples for 10 m

This incident burned: 0%
Budget left after 4 days: 0%
Static threshold pages: 0

Rule	Alerts in 4 days	First alert after the start	Budget gone by then	Kept firing after the fix

Things to try:

Error spike. 20% for 10 minutes burns 4.6% of the budget. The fast page fires 5 minutes in and clears 4 minutes after the fix. The static threshold paged too, and nine more times in four days, once per harmless blip.
Untick “Require the short window”. The same spike keeps paging for 55 minutes after the fix.
Steady burn. 1% for 8 hours never reaches 14.4×. The 6× rule pages after 3.6 hours, with 5% of the budget gone.
Slow leak. 0.4% for 3 days never trips a burn-rate page. A ticket opens after about 14 hours, and the leak spends 41% of the month. The static threshold pages at once, as loudly as for an outage.
Metrics vanish. Users see 20% errors but the exporter is down. No rule fires except the absence alert.

The rule is two counters and two subtractions per window:

BurnRateAlerts.javaburn_rate_alerts.pyBurnRateAlerts.csburn_rate_alerts.goburn-rate-alerts.ts

import java.util.ArrayList;
import java.util.List;
import java.util.OptionalDouble;

// Fire when BOTH windows burn the error budget faster than `factor` times the sustainable rate.
record BurnRule(String name, int longMin, int shortMin, double factor) {}

final class BurnRateAlerts {
    private final double budget;
    // Running totals per minute. Counters, not ratios, so any window is two subtractions and
    // windows over many pods add up correctly.
    private final List<Long> total = new ArrayList<>(List.of(0L));
    private final List<Long> errors = new ArrayList<>(List.of(0L));

    BurnRateAlerts(double slo) {
        // The budget rate: the error ratio that spends exactly the whole budget in the SLO window.
        this.budget = 1 - slo;
    }

    void recordMinute(long requests, long failed) {
        total.add(total.get(total.size() - 1) + requests);
        errors.add(errors.get(errors.size() - 1) + failed);
    }

    OptionalDouble burn(int windowMin) {
        // Burn rate = error ratio in the window / budget rate. 1.0 spends the budget exactly
        // on time; 14.4 spends 30 days of budget in about 2 days.
        int now = total.size() - 1;
        int start = Math.max(0, now - windowMin);
        long requests = total.get(now) - total.get(start);
        if (requests == 0) {
            return OptionalDouble.empty(); // no data is NOT "healthy": a separate absence alert must cover it
        }
        return OptionalDouble.of((double) (errors.get(now) - errors.get(start)) / requests / budget);
    }

    boolean firing(BurnRule rule, boolean useShort) {
        // The long window proves the burn is significant (2% of the budget for 14.4x over 1 h).
        // The short window proves it is still happening, so the page clears minutes after the fix.
        OptionalDouble longBurn = burn(rule.longMin());
        if (longBurn.isEmpty() || longBurn.getAsDouble() <= rule.factor()) return false;
        if (!useShort) return true;
        OptionalDouble shortBurn = burn(rule.shortMin());
        return shortBurn.isPresent() && shortBurn.getAsDouble() > rule.factor();
    }
}

class Main {
    static int[] firstAndClear(boolean useShort) {
        var fast = new BurnRule("fast page", 60, 5, 14.4); // the SRE workbook's first page
        var alerts = new BurnRateAlerts(0.999); // 99.9%: 0.1% of requests may fail
        int fired = -1, cleared = -1;
        for (int minute = 0; minute < 240; minute++) {
            boolean spike = minute >= 60 && minute < 70; // 20% of requests fail for 10 minutes
            alerts.recordMinute(10_000, spike ? 2_000 : 2);
            boolean on = alerts.firing(fast, useShort);
            if (on && fired < 0) fired = minute - 60 + 1;
            if (!on && fired >= 0 && cleared < 0) cleared = minute - 70;
        }
        return new int[] {fired, cleared};
    }

    public static void main(String[] args) {
        System.out.println(java.util.Arrays.toString(firstAndClear(true))); // [5, 4]: pages 5 min in, clears 4 min after the fix
        System.out.println(java.util.Arrays.toString(firstAndClear(false))); // [5, 55]: the 1 h window alone keeps paging for 55 minutes
    }
}

In Prometheus, recording rules precompute the error ratio per window, and the alert is the conjunction:

- alert: CheckoutErrorBudgetFastBurn
  expr: |
    job:slo_errors_per_request:ratio_rate1h{job="checkout"} > (14.4 * 0.001)
    and
    job:slo_errors_per_request:ratio_rate5m{job="checkout"} > (14.4 * 0.001)
  labels:
    severity: page

Page on symptoms, not causes. Users feel errors, latency and stale data, not CPU at 90%. Causes belong on dashboards, or in tickets when they predict trouble days ahead: a disk full within 48 hours, a certificate expiring in a week.

Step 9: Read it: one dashboard per service, and a path to the cause

Each service gets one dashboard, top to bottom in the order of an investigation:

The SLOs and remaining budget, with deploys drawn as markers on every graph.
RED per route: request rate, error ratio, latency percentiles from merged histograms.
Saturation of its own resources: pools, queues, CPU and memory against limits.
Each dependency’s error ratio and latency, as seen from this service.

Labels make the dashboard slice the problem: one zone, one version, one route, one tenant. From there, the path to a cause runs across all three signals:

From a page to the root cause

Scroll or pinch to zoom, drag to pan

Color key
Service
Database
Background worker

Metrics find that something is wrong, and narrow down where. Traces show which hop. Logs explain why. A design that has only one of the three gets stuck at one of these steps.

Step 10: Ship it: the telemetry pipeline

One Prometheus stops being enough when one machine cannot hold every series, logs die with their pods, and nobody can follow a request across services. The replacement has the same shape whatever the vendor:

An agent on every node (OpenTelemetry Collector, Fluent Bit, Vector) scrapes metrics, tails logs, and receives spans over OTLP. It batches and buffers, so a slow backend never pushes back into the application.
A gateway tier of collectors tail-samples, redacts, drops bad labels and routes each signal. Agents route spans by trace id so a whole trace meets on one gateway.
A buffer (usually Kafka) in front of the log and trace stores, because incidents are exactly when log volume jumps tenfold.
Three stores, because the access patterns differ: a time-series database, a log store and a trace store. Rules and alert routing read the metrics store.

Grow it stage by stage, trace the error spike to a phone call, or follow one slow checkout:

Telemetry from services to the on-call phone · architecture

The full design, stage by stage

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Stage

Stage 3 of 3

A buffer, and watchers outside

Why this stage

During an incident the log store slowed under a 10× burst of error logs, the gateways backed up and dropped spans exactly when they were needed.

Separately, a regional network failure took the alerting down with the services.

What we add

Kafka between the gateways and the log and trace stores absorbs bursts and lets the stores replay.

Synthetic probes run from another provider every 30 s.

A small meta-monitor in another region watches the pipeline and receives a dead-man heartbeat from the alerting path.

What it costs

A Kafka cluster sized for the worst burst, a second monitoring stack kept independent in machines, credentials and deploys, and probes to keep in step with the product.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Telemetry from services to the on-call phone · architecture

Trace a request: One slow checkout, kept and found

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

One slow checkout, kept and found: step 1 of 6

Shoppers to Services

A checkout takes 2.4 s. The edge starts a trace and sends traceparent: 00-<trace id>-<span id>-01 on every downstream call, so all 9 services add spans to the same trace.

2400 ms of a 60000 ms budget

Shoppers → Services2400 ms
Services → Collector agentasync
Collector agent → Collector gatewayasync
Collector gateway → Kafka bufferasync
Kafka buffer → Trace storeasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Step 11: Probe from outside

Everything so far is white-box: the system reporting on itself. It misses DNS, an expired certificate, a CDN misconfiguration, or a whole region gone along with the monitoring in it.

Synthetic checks (black-box probes) act like users. Every 30 to 60 seconds, from at least three locations outside production, they resolve the name, and complete TLS. They log in with a test account, and run the critical path. They also cover quiet hours, when 20 requests a minute give no meaningful error ratio. Page when two or more locations fail. And tag synthetic traffic, so it stays out of business metrics.

Options and tradeoffs

Head or tail sampling

Option	Decides	Keeps errors?	Cost	Wins when
No sampling	Never	All	Every span stored: about 10 TB a day here	Low traffic, or a few critical flows
Head, by trace id hash	At the first service	Only the sampled share	Cheapest: dropped spans are never even exported	Cost matters most; latency profiles over many traces
Tail, at collectors	After a fixed wait for spans (~30 s)	All, plus slow ones	Every span buffered; trace-id routing to collectors	Debugging rare errors and slow outliers
Head plus always-on errors	At the first service, overridden on error	Error spans, not their full trace	Cheap, partial traces for errors	A middle step before tail sampling infrastructure

Delivery and storage

How telemetry travels and which stores hold it matter less than the pipeline’s shape.

Go deeperPull or push, log stores and metrics backends3 min

Pull or push.

Model	How	Wins when	Watch for
Pull (scrape)	The collector reads `/metrics` from targets it discovers	Long-lived services; a failed scrape is itself a signal (`up == 0`)	Short-lived jobs end before a scrape; targets behind NAT or firewalls
Push (OTLP, StatsD)	The application or agent sends to an endpoint	Batch jobs, serverless functions, clients outside the network, all traces and logs	A silent client looks like a healthy one; needs per-sender limits
Agent hybrid	A node agent scrapes locally and pushes upstream	Large fleets: one connection per node instead of per target	One more component to run on every node

Logs.

Store	Indexes	Wins when	Costs
Elasticsearch or OpenSearch	Every field, full text	Ad hoc search over any field, fast, mature tooling	Index can be as large as the data; heavy to run
Loki and similar	A few labels only	Most queries start from a service and time range; cheapest storage	Searches scan chunks: slow on wide ranges
ClickHouse (columnar)	Sort key, skip indexes	Analytics over structured logs: counts, group by, top N	Schema design matters; less suited to free text
A managed vendor	Varies	Small teams, no platform staff	Priced per GB ingested and indexed; grows with traffic

Metrics.

Option	Scale	Wins when	Costs
One Prometheus per cluster	A few million series, 15 days locally	Most companies’ first years; simple, reliable	No global view, no long retention, one machine’s memory
Prometheus plus Thanos	Many Prometheus servers, years in object storage	Keep the local servers, add global queries and history	Queries fan out to every server; more moving parts
Horizontally scalable (Mimir, Cortex, VictoriaMetrics, M3)	Hundreds of millions of series	One multi-tenant store with per-team limits	A distributed database to operate
Managed (Datadog, Grafana Cloud, cloud provider)	Whatever you pay for	No platform team; integrated with logs and traces	Priced per series or host: custom metrics get expensive

Alert rules

Rule	Detection	False pages	Reset after the fix	Wins when
Static threshold (5 m ratio > SLO)	Fast	Every blip	Minutes	Never as a page; fine on a dashboard
Threshold with `for: 10m`	Slower by 10 minutes, always	Fewer; long blips still page	Minutes	Simple systems without SLOs yet
Single window burn rate (1 h > 14.4×)	Minutes for big burns	Rare	Up to an hour	A first step to SLO alerts
Multi-window, multi-burn-rate	Minutes for outages, hours for slow burns	Rare	Minutes (short window)	Any service with an SLO

How it fails

Monitoring that shares fate with the system. Say alerting runs in the same region, cluster, account or network as the services it watches. Then it goes down with them, so the worst outages page no one. On 28 February 2017, the S3 outage in AWS’s us-east-1 region also stopped AWS from updating its own service health dashboard. The dashboard depended on S3.

The fix is independence:

A small meta-monitor in another region or provider.
Synthetic probes from outside.
A dead-man switch: an alert that always fires, and pings an external heartbeat service every minute. Then silence itself pages.

Alert fatigue. Teams that get 50 pages a week stop reading them, and the real one is ignored at 3 a.m. Every page must be urgent, actionable and real.

Missing data treated as healthy. A crashed exporter, a renamed metric or a dropped label makes the error series disappear. Then every alert on it resolves, at the moment things got worse. A window with no samples returns nothing, and nothing never crosses a threshold. Every important alert needs an absence partner (up == 0, absent()). And a failed query in the rule engine must count as an evaluation error, never as “all clear”.

A cardinality explosion takes down the metrics backend. One deploy adds a high-cardinality label. Ingesters run out of memory, and every team loses alerting at once. Limits per team at ingest keep the damage inside one team: new series are refused, and existing ones kept.

Failure	Blast radius	Detect	Fix
Shared fate	No pages during the worst outages	Dead-man heartbeat from outside stops	Meta-monitor and probes in another region or provider
Alert fatigue	Real pages ignored	Pages per shift, share needing no action	Burn-rate alerts, delete cause pages, review every page
Missing data looks healthy	Alerts resolve as things get worse	`up == 0`, `absent()`, rule evaluation failures	Absence alerts, SLI at the load balancer
Cardinality explosion	Every team’s metrics and alerts	Active series per team and metric	Ingest limits, relabeling, growth alerts
Log storm	Log pipeline, then the app if it blocks	Ingest rate, drop counters, buffer fill	Async bounded buffers, Kafka, per-message rate limits
Clock skew	Misleading traces	Children outside parents, NTP offset	NTP or PTP, per-span durations, viewer adjustment
Sampling drops rare errors	The one trace you need is gone	Errors in metrics with no matching traces	Tail sampling on errors and latency
Windows hide short outages	A 30 s outage is a dip in a 5 m `rate()`	Counters and the budget still record it	Know the graph’s resolution, read the budget

Go deeperLog storms, clock skew and lost traces1 min

Log storms and backpressure. An outage makes every request log a stack trace. Volume goes up tenfold, and the log pipeline slows. If logging is synchronous, the application blocks on it, and the outage spreads. Three things keep the blast radius inside the telemetry: bounded async buffers that drop (and count drops), rate limits on repeated messages, and a Kafka buffer before the log store.

Clock skew in traces. Spans carry timestamps from each host’s clock. With 50 ms of skew, a child span can appear to start before its parent. Or a 3 ms call can look like 53 ms. Use spans’ own durations, which are measured on one clock, rather than differences across hosts. Keep NTP or PTP healthy, and alert on offset. And let the trace viewer adjust children that fall outside their parent.

Sampling hides the evidence. With 1% head sampling, a bug that hits one checkout in 10,000 leaves a trace once in a million requests. Tail-sample on errors, and keep metrics unsampled: they are the record of what happened, traces are examples.

A failure of the pipeline itself, and what still works:

Telemetry from services to the on-call phone · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseQueue or streamBackground workerNew in this stageAsync or occasional

Break

Failure: click any box to break it

The metrics store goes down

What breaks

Queries fail or time out, so rules cannot evaluate.

How we notice

Rule evaluation failures, and the dead-man heartbeat stops reaching the meta-monitor within 5 minutes.

What users see

Nothing directly.

On-call is blind: no burn-rate alert can fire while it is down.

What we do

Three replicas per series and zone spreading.

The meta-monitor runs the few critical alerts on its own copy of the key counters, so a page still goes out.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

At scale

Numbers to remember.

What	Typical figure
One metric sample, raw and compressed	16 bytes raw, about 1.4 bytes with Gorilla-style compression
Memory per active series in a Prometheus-style TSDB	Roughly a few KB with index; a few million per server
Histogram series per label set	Number of buckets + 2 (sum and count), so 14 for 12 buckets
Scrape interval and rule evaluation	15 to 60 s scrapes, rules every 30 s to 1 min
A structured log line	300 bytes to 2 KB; 1 KB is a fair estimate
A span	200 bytes to 1 KB with attributes
Tail sampling decision wait	About 30 s
99.9% over 30 days	43.2 minutes of full outage, or 0.1% of requests
Fast burn page (14.4× over 1 h and 5 m)	Detects a full outage in about a minute, at 2% of the budget
Synthetic probes	Every 30 to 60 s, from 3 or more locations, page on 2 failing

At 10x (3,000 services, 40 million series, 50 TB of logs a day), three things change. A sharded, multi-tenant metrics store with per-team limits replaces the per-cluster Prometheus servers. The collector fleet gets its own on-call. And recording rules precompute every SLO ratio.

At 100x, telemetry is its own distributed system, per failure domain. Google’s Monarch works this way: its zones ingest, store and evaluate their own data, so one zone’s failure does not blind the others. Collectors aggregate away per-pod labels for most metrics. And series, log bytes and spans are charged to the team that emits them. Observability spend can outgrow the infrastructure it watches.

What to monitor about the monitoring:

The dead-man heartbeat, from an external service, so silence pages.
Ingest: samples, log bytes and spans accepted versus dropped, per team.
Active series per team and per metric, and their growth after deploys.
Rule evaluation duration and failures. A rule that cannot evaluate is an alert that cannot fire.
Delay from a sample’s timestamp to queryable, and from an alert firing to the page.
Collector buffer fill and Kafka consumer lag for the log and trace stores.

In the interview

Interviewer asksHow would you monitor this system?

I start from what users need. For the read path, I’d use an availability SLI measured at the load balancer. And a latency SLI, as the share of requests under 300 ms. Both get a 99.9% SLO over 30 days. I page on multi-window burn rates against those SLOs: 14.4 times over an hour and five minutes, and 6 times over six hours and thirty minutes. A 1 times burn over three days makes a ticket. Each service has RED metrics as histograms, and USE metrics for its pools, queues and disks. Those go on its dashboard, not into pages. Requests carry a W3C traceparent. Tail sampling keeps every error and slow trace, and structured logs carry the trace id. Synthetic probes from outside the region cover the path in front of the load balancer, and the quiet hours. The alerting runs somewhere that does not fail with the system, with a dead-man heartbeat.

Interviewer asksYour latency dashboard shows the average of each pod's p99. What is wrong with it?

A mean of percentiles is not a percentile. It weighs a pod serving 1,000 requests the same as one serving 9,000. It can be wrong in either direction. In a two-pod case, it says 1.3 s while the fleet p99 is 2 s. Pods export histograms with shared bucket bounds. The dashboard sums the bucket rates across pods, and estimates once. For the SLO, count the share of requests under the threshold, with the threshold as a bucket bound. Ratios of counts aggregate exactly.

Interviewer asksWhy not just page when the error rate goes above 0.1% for five minutes?

Because it ignores how much budget the event costs. A three-minute blip at 1.5% spends about a thousandth of the month’s budget. Yet it pages someone at night, exactly as loudly as a full outage. Every page looks the same, so on-call learns to ignore them. Burn-rate alerts tie the page to the damage. A 14.4 times burn over an hour has spent 2% of the budget, which is worth waking someone for. The short window makes the alert clear minutes after the fix, instead of an hour later. I would keep the static threshold as a dashboard line, not a page.

Interviewer asksA team added user_id as a metric label and the metrics cluster is falling over. What do you do now, and next time?

Now: drop the label with a relabel rule at the collectors, or at the ingest tier if that is faster to push. Let the extra series age out of the head block. Next time: per-team and per-metric limits on active series at ingest. They refuse new series past the limit, with an error that names it, so the damage stays inside one team. Add a warning at 80%. Per-user questions go to logs or traces, linked from the metric by exemplars.

Seen in

Metrics and monitoring: the metrics backend itself. It has cardinality limits per team, 1.4 byte samples, the alerting pipeline, and a meta-monitor with a dead-man switch.
Notification system: receipt rates per provider as the signal for silent failures, and a synthetic notification per cell every 10 seconds.
Chat: end-to-end delivery latency measured by synthetic clients messaging each other, and connected devices compared with last week to catch what health checks miss.
Payment system: approval rate per payment provider is the SLI, because broken integrations show up as declines, not errors. Ledger invariants page on any violation.
File sync: sync latency between synthetic device pairs per region, paged on p99 above 10 seconds.
Stock exchange: wire-to-wire latency at p99 and p99.99 per partition, where a latency spike is the early warning of an outage.
Distributed cache: hit ratio against its baseline as a page, because it moves before the database falls over.
Ad click aggregator: freshness as an SLI, wall clock minus the stream job’s watermark, paged above 2 minutes.
Message queue: consumer lag in seconds as the saturation signal of a log, and under-min-ISR partitions paged because they refuse writes.
News feed: error budgets as the argument for regions, since one hour-long regional outage spends more than a year of a 99.99% budget.
Pastebin: a synthetic probe that creates an expiring paste and checks it is gone, from several regions.

Recap

The jobs. Prove the system meets its SLOs, and find the broken component fast.

Metrics. Counters, gauges and histograms. Every label combination is a series held in memory, so labels must be few and bounded. User ids go in logs and traces, linked by exemplars.

Percentiles. Averages hide the tail. Never average p99s: sum histogram buckets across pods, then estimate once. Latency SLIs as “share under X ms” aggregate exactly.

Logs. Structured, with a trace id. The costliest signal: sample the happy path, index few fields, tier retention, never block on a write.

Traces. Spans joined by W3C traceparent. Head sampling is cheap, and drops most errors. Tail sampling keeps every error and slow trace, at the cost of buffering.

What to watch. RED for services, USE for resources, golden signals for user-facing systems. Saturation rises first.

SLOs. SLI = good over valid, measured at the user boundary. Page on multi-window burn rates: 14.4× over 1 h and 5 m, 6× over 6 h and 30 m, and a ticket at 1× over 3 d and 6 h. Page on symptoms, not causes.

Pipeline and failure. Agents, collector gateways, a Kafka buffer, a store per signal. Monitoring lives outside the system’s failure domain, alerts on absence and caps cardinality per team.

Observability

The problem it solves

How it works

Step 1: Count things: metrics

Step 2: Labels multiply

Step 3: Percentiles, and why you cannot average them

Step 4: Record events: structured logs

Step 5: Follow one request: traces

Step 6: Decide what to watch: RED, USE and the golden signals

Step 7: Promise a number: SLIs, SLOs and error budgets

Step 8: Page on burn rate, not on thresholds

Step 9: Read it: one dashboard per service, and a path to the cause

Step 10: Ship it: the telemetry pipeline

The full design, stage by stage

Why this stage

What we add

What it costs

Trace a request: One slow checkout, kept and found

Step 11: Probe from outside

Options and tradeoffs

Head or tail sampling

Delivery and storage

Alert rules

How it fails

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

At scale

In the interview

Seen in

Recap

Further reading

The designbeyondcode digest, every Saturday

Observability

The problem it solves

How it works

Step 1: Count things: metrics

Step 2: Labels multiply

Step 3: Percentiles, and why you cannot average them

Step 4: Record events: structured logs

Step 5: Follow one request: traces

Step 6: Decide what to watch: RED, USE and the golden signals

Step 7: Promise a number: SLIs, SLOs and error budgets

Step 8: Page on burn rate, not on thresholds

Step 9: Read it: one dashboard per service, and a path to the cause

Step 10: Ship it: the telemetry pipeline

The full design, stage by stage

Why this stage

What we add

What it costs

Trace a request: One slow checkout, kept and found

Step 11: Probe from outside

Options and tradeoffs

Head or tail sampling

Delivery and storage

Alert rules

How it fails

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

At scale

In the interview

Seen in

Recap

Further reading

Keep reading

The designbeyondcode digest, every Saturday