Design a metrics and monitoring system

Design a metrics and monitoring system like Prometheus or Datadog for 100 million active series. Start with one monitoring server, grow it with collectors, sharded ingesters, blocks on object storage, a separate alert path and a query frontend, then stress it with cardinality, compression and paging, and watch the watchers, with code.

Senior28 min core · 45 min full depth

Read first: Estimation Partitioning
Scaling patterns: Backpressure and load shedding
Building blocks: Partitioning Consistent hashing Replication Blob storage Caching Observability
Related: Ad click aggregator Top-k heavy hitters

“Design a metrics and monitoring system, something like Prometheus or Datadog. Every service in the company sends metrics, engineers look at dashboards, and on-call gets paged when something breaks.”

A metric sample is 16 bytes: a timestamp and a number. Storing them looks like the easy part. The real limits are elsewhere. The number of distinct series held in memory can be multiplied by a thousand by one careless label. A year of history must be cheap, yet a 30-day dashboard must load in under a second. And alerting must still work when the system it depends on, including itself, is broken.

So this page starts with one monitoring server, and lets the numbers grow it. Then it tests the result against those three problems.

The answer in one minute

If you only have a minute, this is the design. The rest of the page explains each part.

Size by series, not samples. 100 million active series decide the memory, and the memory decides the cluster. A limit per team stops one bad label from hurting anyone else.
Pull inside each cluster, push to the center. A collector scrapes nearby services. That gives a free “is it up?” signal. Then it pushes batches to the central system.
Recent data in memory, history on object storage. Ingesters hold two hours, compressed to about 1.4 bytes a sample. Every two hours they write a block to object storage.
Alerts on their own path. Rule engines read the ingesters directly, never the dashboards’ query path. An alert router turns 400 firing alerts into one page.
A monitor for the monitor. A small, separate stack and a heartbeat that pages when it stops, because silence must never look like health.

Scope it

Four questions decide the design: how many series, how often they are sampled, how long we keep them, and how fast an alert must arrive.

Interviewerminute 0

Design a metrics and monitoring system for our company. Every service sends metrics, and on-call gets paged when something breaks.

What they are testing

Whether you pin down series count, retention and who the users are before drawing anything. Series count sizes the cluster, and retention sizes the storage. The red flag is asking “how many requests per second”, as if this were a web service, and never asking about series or labels.

These four questions matter most:

How many series? A series is a metric name plus a set of labels, such as http_requests_total{service="checkout", status="500"}. We have 200,000 pods and hosts, about 500 series each.
How often? Every series is sampled every 10 seconds.
How long do we keep it? Raw data for 15 days, 5-minute rollups for 90 days, 1-hour rollups for 13 months.
How fast must an alert arrive? Within a minute of its condition holding long enough.

From these answers we make five promises:

Alerts always work. 99.99% available. A missed alert is the worst failure this system has.
Alerts are quick. A page within 1 minute of the condition holding, at p99.
Fresh. A sample can be queried within 15 seconds of being scraped.
Fast dashboards. Under 1 second for 24 hours of data, under 5 seconds for 30 days.
Isolated. One team’s label explosion or runaway query can’t hurt anyone else’s ingest, dashboards or alerts.

We leave out logs and traces (they are bytes, not numbers, and need a different store), anomaly detection, and the dashboard UI.

Go deeperMore questions, and what each changes1 min

Question	We assume	If the answer were different
Internal, or a product?	Internal: one large company, about 2,000 teams	A product needs hard isolation per customer and billing per series
Sampled every 1 s?	No, every 10 s	Ten times the samples and storage, but not the memory
Raw data for 13 months?	No	Storage grows from about 46 TB to about 500 TB
Push or pull?	Services expose `/metrics`; short batch jobs push	All-push (serverless) loses the free “target is up” signal
Can we lose samples?	A few seconds in a rare double failure is acceptable	Billing from metrics needs a replayable log in front of storage
Who writes rules?	Teams, as reviewed code: about 200,000 rules	A few hundred central rules fit on one pair of servers

The numbers that decide it

Now we work out the numbers. Requests per second is the wrong unit here: count series.

Assumptions

Scrape targets: 200,000
Active series per target: 500
Samples per series a minute: 6
Extra load at peak: 50%
Bytes per compressed sample: 1.4 bytes
Days of raw samples kept: 15
Ingester replication factor: 3
Memory per in-memory series: 8 KB
Alert and recording rules: 200,000

Series100Mactive series

200,000 targets × 500 series.

So we size every tier by series count, and treat any change that multiplies it as an outage risk.

Samples10M/s15M/s at peak

100 million series every 10 seconds. Deploys and collectors catching up after a network blip push that to 15 million. One server takes about 1 million a second.

So we split ingestion by series.

Memory2.4 TBabout 80 ingesters

Each series in memory costs about 8 KB: its labels, its index entries and its open chunk. 100 million × 3 copies × 8 KB, on servers of 50 GB kept 60% full.

So we size the ingest tier by memory, which is set by series.

Storage1.2 TBa day; 46 TB kept

Raw, a sample is 16 bytes: 14 TB a day. Compressed to about 1.4 bytes, it is 1.2 TB a day. Fifteen days raw is 18 TB, and rollups add 28 TB.

So we keep only recent data on ingesters, and the rest on object storage, at about $1,000 a month.

Queries2,000/s6,700 rule checks/s

3,000 open dashboards with 20 panels refreshing every 30 seconds, against 200,000 rules every 30 seconds.

So we give alerting, the largest query workload, its own path.

Cross-zone400 MB/sabout $20,000 a month

Writing every sample to 3 zones sends 2 copies across zones, at about 20 bytes a sample on the wire.

So we treat traffic between zones as a cost driver: it is 20 times the storage bill.

Rule checks outnumber dashboard queries three to one, which surprises most people. Try the calculator. Raise series per target from 500 to 5,000, which is one label explosion across the fleet. Ingesters go from 80 to 800. Double the sample rate instead, and the ingester count doesn’t move.

Change an assumption

Scrape targets200,000Active series per target500Samples per series a minute6Extra load at peak50%Bytes per compressed sample1.4 bytesDays of raw samples kept15Ingester replication factor3Memory per in-memory series8 KBAlert and recording rules200,000

Active series: 100 million; Size every tier by this number, not by samples.
Samples per second, average: 10,000,000/s
Samples per second, peak: 15,000,000/s; More than one server ingests (about 1 million/s): shard ingestion by series hash.
Ingester memory, all replicas: 2.4 TB; The biggest line on the bill: RAM for the in-memory head.
Ingesters (30 GB of series each, 60% of 50 GB): 80; Memory per series, not samples per second, sets the ingester count.
Raw samples per day, compressed: 1.2 TB; Too much to keep for months on replicated SSDs: ship blocks to object storage.
Raw tier on object storage: 18 TB
Rollups: 5 min for 90 days, 1 h for 13 months: 28 TB; About 8 bytes per rollup point: min, max, sum and count, compressed.
Total on object storage: 46 TB; About $1,000 a month at $0.02 per GB: storage is not the cost driver.
Ingest bandwidth, peak: 300 MB/s; About 20 bytes a sample, snappy-compressed with labels. Two of the three replica copies cross zones: about 1 PB a month at the average rate, roughly $20,000 at $0.01 per GB on each side. A cost driver.
Rule evaluations: 6,700/s; More than the dashboard queries: alerting is the largest query workload.

Build the design

We start with the simplest design that works. Then we let the numbers break it, one at a time. Each step fixes only what broke, adds only the boxes that fix needs, and says what the fix costs. Throughout, follow one question: how does a number on a service end up on a graph, and in a page at 3 am?

Step 1. One monitoring server

The simplest version is one Prometheus server: it scrapes every service, stores samples locally, answers queries, checks rules and sends pages. It’s right for a few thousand services.

Metrics and monitoring · architecture

Stage 1: One monitoring server

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Samples ingested per second: 15,000,000

Stage 1 of 6

One monitoring server

Why this stage

Nothing yet.

This is the smallest design that collects, stores, graphs and alerts.

What we add

One server that scrapes every service, writes samples to a local time-series database, evaluates alert rules and pages on-call.

What it costs

One server ingests about 1 million samples/s and we have 10 million.

It is also a single point of failure for every alert.

At 15M samples ingested per second, Monitor server is 1500% busy. Stage 2 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

Here it takes about 1 million samples a second against our 10 million, and it is a single point of failure for every alert.

Step 2. Collectors and sharded ingesters

Collectors: pull nearby, push to the center

Pull and push both work at scale. Pull gives one thing free: if a scrape fails, the target is down, and a series called up says so. Push works through any firewall and for jobs that live a few seconds. A single central scraper, though, would have to reach every pod in every network.

So each cluster runs a collector (Prometheus in agent mode, or an OpenTelemetry collector). It finds and scrapes local targets, takes pushes from short jobs, and pushes batches to the center. Two collectors per cluster split the targets, and each keeps a few hours on disk, so a network outage leaves no gap.

Ingesters, sharded by series

Stateless distributors check each batch against the team’s limits, hash every series, and forward it. Ingesters hold recent data in memory and in a write-ahead log on disk. Each series goes to 3 ingesters, one per zone, chosen by its hash. A write succeeds when 2 of 3 confirm.

Why a hash of the series? It spreads load evenly. Sharding by metric name would put the busiest metrics on one node. Sharding by team would make the biggest team a hot shard. The cost is that every query asks every ingester, which step 5 addresses.

Metrics and monitoring · architecture

Stage 2: Collectors and sharded ingesters

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 2 of 6

Collectors and sharded ingesters

Why this stage

At 10 million samples/s and 100 million active series, one server is 10 times over its limit on samples, and 800 GB of series does not fit in one machine’s memory.

What we add

A collector per cluster that scrapes locally and pushes batches, stateless distributors that enforce limits and hash each series to 3 ingesters, and queriers that fan out to them.

What it costs

Every query now fans out to every ingester, and ingesters only hold recent data.

At 15M samples ingested per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

Follow one scrape to a confirmed write:

Metrics and monitoring · architecture

Trace a request: One scrape, ingested

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

One scrape, ingested: step 1 of 4

Collectors to Services

Every 10 s the cluster’s collector sends GET /metrics to a pod: 500 series as gzipped text. It stamps every sample with the scrape time and records up = 1. A failed scrape records up = 0, which is how we know a target is down.

5 ms of a 50 ms budget

Collectors → Services5 ms
Collectors → Distributors5 ms
Distributors → Ingesters5 ms
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

A retry needs no special key: a sample is identified by its series and timestamp, so the same sample sent twice is stored once.

Go deeperWhat happens when an ingester restarts1 min

An ingester holds about two hours of samples in memory. Every sample is also appended to its write-ahead log (WAL) on local disk before the write is confirmed. A WAL is a plain file of recent writes in order, kept only so they can be replayed after a crash.

When an ingester restarts, it reads its WAL back into memory before it takes new writes. With tens of gigabytes of recent data, that replay can take several minutes. During that time, its series still have two other copies, one in each other zone. Writes need only 2 of 3 to confirm, and reads ask all three and merge the answers, so nobody sees a gap.

This is also why deploys go one zone at a time. Restart every ingester in one zone, wait until they have replayed and caught up, then move to the next zone. At no moment does any series lose two of its three copies.

Step 3. History on object storage

Thirteen months of raw data, three times over on ingester SSDs, would be about 1.4 PB of the most expensive storage in the system. So every 2 hours, each ingester cuts its recent data into an unchanging block and uploads it to object storage.

Three replicas upload three overlapping copies. The compactor merges them, removing duplicate samples, then merges small blocks into day-sized ones, writes the 5-minute and 1-hour rollups, and deletes blocks past retention. Store gateways keep each block’s index in memory, so queriers can read history. Queriers read ingesters for the last few hours, and gateways for everything older.

Metrics and monitoring · architecture

Stage 3: Long-term storage

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 3 of 6

Long-term storage

Why this stage

Not load: the 13 month retention requirement.

Raw data is 1.2 TB a day, and keeping 13 months of it on replicated ingester SSDs would take about 1.4 PB.

What we add

Ingesters ship each 2 hour block to object storage.

A compactor merges blocks and writes 5 minute and 1 hour rollups.

Store gateways serve old blocks to queriers.

What it costs

Long-range queries now read object storage, and at the 15 million peak they push the querier pool past capacity while it also runs every alert rule.

At 15M samples ingested per second, Queriers is 110% busy. Stage 4 is the fix.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

Step 4. Alerts on their own path

Rule checks and 30-day dashboards together would overload the queriers, and a team opening a big dashboard could delay someone else’s alert. So rules run on their own rule engines. Rules look at the last 5 to 10 minutes, so they read the ingesters and never touch object storage. Firing alerts go to an alert router, 3 replicas that group them into pages and send each page to the right team. The stress tests walk through it.

Metrics and monitoring · architecture

Stage 4: A separate alerting pipeline

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 4 of 6

A separate alerting pipeline

Why this stage

At the peak, rule evaluation and dashboards overload the shared queriers, so a heavy dashboard can delay an alert.

Alerts also need dedup, grouping, routing and silences.

What we add

Rule engines, sharded by rule group, that read recent data straight from the ingesters, and an alert router cluster that dedups, groups, routes and applies silences.

What it costs

Two more systems to run, and rule state that must survive restarts.

At 15M samples ingested per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

Metrics and monitoring · architecture

Trace a request: One rule evaluation

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

One rule evaluation: step 1 of 4

Rule engines to Ingesters

Every 30 s the rule engine that owns this group runs the expression, for example the 5 minute error ratio per service, as an instant query. Recent data is all in the ingesters, so object storage is never on this path.

25 ms of a 100 ms budget

Rule engines → Ingesters25 ms
Rule engines → Alert router3 ms
Alert router → Pagerasync
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

This is enough for a mid-level answer. Collectors, ingesters sharded by series in three zones, blocks on object storage, and alerts on their own path. Step 5 and the stress tests are what make it a senior answer.

Go deeperThe API2 min

Each service serves its current values as text. Counters only go up. Queries compute rates.

GET /metrics

# TYPE http_requests_total counter
http_requests_total{route="/cart",method="GET",status="200"} 1027341
http_requests_total{route="/cart",method="GET",status="500"} 112

Collectors push batches, protobuf-encoded and compressed:

POST /api/v1/push
Content-Type: application/x-protobuf
Content-Encoding: snappy
X-Tenant: checkout-team

200 OK                    stored on 2 of 3 replicas
400 Bad Request           out of order, too old, bad labels, or the team's series limit: don't retry
429 Too Many Requests     ingest rate limit: retry with backoff

A series limit returns 400, not 429, on purpose: retrying won’t make room, and a collector retrying forever blocks the healthy series queued behind it.

GET /api/v1/query_range?query=sum by (route) (rate(http_requests_total{service="checkout",status=~"5.."}[5m]))&start=1730000000&end=1730086400&step=60

200 OK
{ "status": "success", "partial": false, "data": { "resultType": "matrix", "result": [ ... ] } }

Each query is limited to 50,000 series, 50 million samples and 2 minutes. partial: true says some store didn’t answer, so a dashboard warns instead of drawing a confidently empty graph.

Go deeperBlocks, the index, and the stores we turned down2 min

A series’ ID is a hash of its sorted labels. The metric name is just the label __name__. Samples are packed into compressed chunks of about 120. A block is 2 hours of one ingester’s series:

01JB7KQ2.../               one block, never modified
  meta.json                time range, series and sample counts, resolution
  index                    series (labels + chunk references),
                           postings: label=value → sorted list of series IDs
  chunks/000001            512 MB files of compressed chunks
  tombstones               deletions, applied when the compactor rewrites the block

A query intersects postings lists: service="checkout" gives one sorted list of series IDs, status="500" another, and their intersection is the series to read. Each block’s meta.json holds its time range, so a query for last week never touches the ingesters.

Option	Costs
Wide-column store, a row per series per day	No label index, little compression
Columnar analytics store	Label matching every 30 seconds is costly
In-memory only	RAM for every sample; history needs a second store
Memory head + blocks on object storage	We run the parts ourselves

Cortex, Thanos and Grafana Mimir all use the last layout. Its cost: writes only append in time order per series, so late samples are rejected, and deleting a series means rewriting blocks.

Quorum writes with no log in front take about 15 ms, and losing one ingester or zone loses nothing. A rare double failure can lose a few seconds. Kafka in front would make ingest replayable, at the cost of a large cluster and more cross-zone traffic: worth it only if metrics drove billing.

Step 5. A query front end

The last thing to break is the read side. Traffic grows ten times.

At 100 million samples a second, distributors and ingesters simply grow. The queriers are what break. Most dashboard work is repeated. A 7-day panel refreshing every 30 seconds asks for the same 7 days again, and only the last 30 seconds changed. And one 30-day query can fill a querier’s memory.

The fix is a query frontend. It splits a long query into one per day, run in parallel. It caches each finished day, so only today is recomputed. And it queues fairly per team, so one team’s dashboards can’t starve another’s. Querier load drops about fivefold.

Metrics and monitoring · architecture

Stage 5: A query frontend

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 5 of 6

A query frontend

Why this stage

At 10 times the load, dashboard queries need about 50,000 querier units a second against a pool of 16,000, and one 30 day query can exhaust a querier’s memory.

What we add

A query frontend that splits long queries by day, caches the results of finished days, and queues work fairly per team.

What it costs

Cached results for the last few minutes can be slightly stale, and the cache needs invalidating when old data is backfilled.

At 15M samples ingested per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

Metrics and monitoring · architecture

Trace a request: A 7 day dashboard panel

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasionalHop in motionHops doneResponse

A 7 day dashboard panel: step 1 of 6

Engineers to Query frontend

The browser asks for 7 days at a 5 minute step. The frontend aligns start and end to the step, splits the query into 7 one-day queries and checks the results cache: the 6 finished days hit.

20 ms of a 1000 ms budget

Engineers → Query frontend20 ms
Query frontend → Queriers1 ms
Queriers → Ingesters40 ms
Queriers → Store gateways60 ms
Store gateways → Object storage80 ms
Response

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Stress it

The design is complete. Now we test it where monitoring is hard. One label that multiplies everything. Storage that must be tiny. The path from a rule to a phone ringing. Long-range dashboards. And what happens when the monitoring itself is down. Only the last one adds boxes.

Cardinality

Interviewerminute 20

A team adds a user_id label to their request counter. What happens, and how do you stop it?

What they are testing

Whether you can do series math, know that it multiplies, and put limits where they protect everyone else. “Tell teams not to do that” is the red flag.

A series exists for every combination of label values that occurs. So the count can reach the product of the number of values of each label. One latency histogram:

Label	Distinct values	Running series count
bucket (`le`)	12	12
route	40	480
method	4	1,920
status	8	15,360
pod	200	3.1 million

Real combinations are sparser, perhaps a third of that, but one metric on one service is still a million series. Add user_id, and in practice it is every user active in the last two hours: tens of millions of new series on one deploy. Without limits, ingester memory grows until the processes are killed.

Churn is the quieter version. Every new pod name is a new set of series, and the old ones stay in memory until the next block is cut. A team deploying 20 times a day on 200 pods creates 4,000 pods’ worth of series a day.

The defense is layered. Collectors drop bad labels before they leave. Distributors limit each team’s sample rate and label sizes. And ingesters enforce a limit on active series per team, say 5 million, because only they know which series exist.

When user_id lands, samples for series that already exist keep flowing, so the team’s dashboards and alerts still work. Only new series are refused, with a 400 that names the limit. Nobody else notices. Ingesters can also list a team’s top metrics by series count, so the team is warned at 80% and told which metric grew.

Go deeperEvery layer of limits1 min

Layer	Mechanism	Protects
Collector	Relabel rules drop bad labels before they leave	Everything downstream
Distributor	Sample rate limit per team, label count and length checks	Every other team’s ingest
Ingester, per team	Active series limit, split across its shards	Every other team’s memory
Ingester, per metric	Series limit per metric name	The team’s other metrics
Ingester, global	A ceiling on series per process, as a last resort	The process, from running out of memory
Query	Series and sample limits per query	Queriers

Each ingester enforces its share: the limit × 3 replicas ÷ the ingesters holding the team’s series. An operator can push a drop rule to the distributors while the team rolls back.

Storing a sample in 1.4 bytes

The numbers said 1.4 bytes a sample. But a timestamp and a float are 16. Where did the rest go?

Samples next to each other in one series are alike: timestamps come on a fixed interval, and values change slowly. Facebook’s Gorilla paper turned that into two encodings that work one sample at a time, so even the open chunk in memory is compressed.

Timestamps: the delta of the delta. Store the first timestamp in full. After that, the gap between samples is about 10,000 ms, and the change in that gap is almost always 0. A 0 costs one bit. A small change costs 9 bits, and larger ones a few more.

TimestampEncoder.javatimestamp_encoder.pyTimestampEncoder.cstimestamp_encoder.gotimestamp-encoder.ts

// Gorilla's timestamp encoding (Facebook, VLDB 2015). Scrapes arrive on a fixed interval, so the
// change in the gap between samples, the delta of delta, is usually 0 and costs 1 bit.
class TimestampEncoder {
    // {prefix, prefix bits, payload bits}. A dod in [-(2^(n-1) - 1), 2^(n-1)] fits n payload
    // bits as two's complement; the decoder reads a payload above 2^(n-1) as negative.
    private static final int[][] BUCKETS = {{0b10, 2, 7}, {0b110, 3, 9}, {0b1110, 4, 12}};

    private final BitWriter out = new BitWriter();
    private int count = 0;
    private long prevTime;
    private long prevDelta = 0; // so the first delta goes out as a dod: Gorilla gives it 14 bits instead

    // Timestamps are in milliseconds, as in Prometheus. A few ms of scrape jitter makes many dods
    // small but not zero, so real chunks spend about 9 bits here, not 1: still 7 times under 64.
    void append(long timeMillis) {
        if (count++ == 0) {
            out.writeBits(timeMillis, 64); // once per chunk of about 120 samples
            prevTime = timeMillis;
            return;
        }
        // Ingesters reject out-of-order samples. That keeps every delta positive and lets the
        // gaps inside a 2 hour chunk always fit the 32 bit bucket.
        if (timeMillis <= prevTime) throw new IllegalArgumentException("out-of-order sample");
        long delta = timeMillis - prevTime;
        long dod = delta - prevDelta;
        prevTime = timeMillis;
        prevDelta = delta;

        if (dod == 0) {
            out.writeBit(false); // the steady state: one bit per timestamp
            return;
        }
        for (int[] b : BUCKETS) {
            long half = 1L << (b[2] - 1);
            if (dod >= -(half - 1) && dod <= half) {
                out.writeBits(b[0], b[1]);
                out.writeBits(dod, b[2]); // low n bits of the two's complement
                return;
            }
        }
        out.writeBits(0b1111, 4);
        out.writeBits(dod, 32); // a missed scrape or a restart: rare, so 36 bits is fine
    }

    int bitLength() { return out.bitLength(); }
}

// Appends bits, most significant first, to a growing byte array. The chunk is this array.
class BitWriter {
    private byte[] buf = new byte[16];
    private int bits = 0;

    void writeBit(boolean one) {
        if (bits == buf.length * 8) buf = Arrays.copyOf(buf, buf.length * 2);
        if (one) buf[bits >> 3] |= (byte) (0x80 >>> (bits & 7));
        bits++;
    }

    void writeBits(long value, int n) {
        for (int i = n - 1; i >= 0; i--) writeBit(((value >>> i) & 1) == 1);
    }

    int bitLength() { return bits; }
}

Values: XOR with the previous value. XOR two nearby floats, and most bits are 0, because the sign, exponent and top of the number match. An unchanged value costs one 0 bit. Otherwise, write only the bits between the leading and trailing zeros. Gorilla reports an average of 1.37 bytes a sample on Facebook’s data, twelve times smaller.

A chunk closes at about 120 samples, 20 minutes at a 10-second interval. A chunk decodes only from its start, so reading one sample decodes up to 120: the price of the compression.

Go deeperThe timestamp buckets1 min

Delta of delta	Bits written
0	`0`: 1 bit
−63 to 64	`10` + 7 bits: 9 bits
−255 to 256	`110` + 9 bits: 12 bits
−2,047 to 2,048	`1110` + 12 bits: 16 bits
anything else	`1111` + 32 bits: 36 bits

The code uses Gorilla’s buckets exactly, and simplifies one thing. Gorilla stores the first delta in 14 bits, and the code stores it as an ordinary delta of delta. With millisecond timestamps, a scrape 3 ms late gives a delta of delta of 3. That takes 9 bits, not 1. Prometheus widened the buckets for this reason.

The write-ahead log is flushed in pages, not synced per sample, so replication, not the log, protects the last few seconds.

From a rule to a phone ringing

Follow “error rate goes above 5%” all the way to a phone ringing. Then: how do we avoid paging someone 400 times?

Checking. A rule is an expression and a threshold, such as “the 5-minute error rate is above 5%”, with for: 5m, checked every 30 seconds. Each label set it returns is a separate alert. It is pending when the condition first holds, firing once it has held for the whole 5 minutes, and resolved when it stops. A pending alert that clears disappears silently, so short blips never page.

AlertRule.javaalert_rule.pyAlertRule.csalert_rule.goalert-rule.ts

// One alerting rule, such as "error ratio above 5% for 5 minutes", evaluated every 30 s. The
// expression returns one value per label set (one per service), and each is its own alert.
class AlertRule {
    enum State { PENDING, FIRING }

    static final class Alert {
        State state = State.PENDING;
        final long activeSince; // persisted as a series, so a restart does not reset the for: timer
        long lastSentAt = Long.MIN_VALUE; // never sent
        Alert(long activeSince) { this.activeSince = activeSince; }
    }

    record Notification(String series, boolean resolved, long activeSince) {}

    private final double threshold;
    private final long forMillis;              // must stay true this long before anyone is paged
    private final long resendMillis = 60_000;  // re-send while firing, so the router never times it out
    private final Map<String, Alert> active = new HashMap<>();

    AlertRule(double threshold, long forMillis) {
        this.threshold = threshold;
        this.forMillis = forMillis;
    }

    // results: series label set -> value of the expression at `now`. Called only for a query
    // that succeeded completely: on an error or a partial answer the engine skips the call, because
    // an empty map there would resolve every firing alert mid-outage.
    List<Notification> evaluate(Map<String, Double> results, long now) {
        List<Notification> out = new ArrayList<>();
        for (var e : results.entrySet()) {
            if (e.getValue() <= threshold) continue;
            Alert a = active.computeIfAbsent(e.getKey(), k -> new Alert(now));
            // Pending until the condition has held for the whole duration: a blip that clears
            // inside one evaluation or two never wakes anyone.
            if (a.state == State.PENDING && now - a.activeSince >= forMillis) a.state = State.FIRING;
            if (a.state == State.FIRING
                    && (a.lastSentAt == Long.MIN_VALUE || now - a.lastSentAt >= resendMillis)) {
                out.add(new Notification(e.getKey(), false, a.activeSince));
                a.lastSentAt = now;
            }
        }
        // Anything active that is now below the threshold, or missing from the results, ends.
        // Missing data resolves the alert: a dead exporter looks healthy unless a separate
        // absent() rule watches for it.
        for (var it = active.entrySet().iterator(); it.hasNext(); ) {
            var e = it.next();
            Double v = results.get(e.getKey());
            if (v != null && v > threshold) continue;
            if (e.getValue().state == State.FIRING) {
                out.add(new Notification(e.getKey(), true, e.getValue().activeSince));
            }
            it.remove(); // a pending alert disappears silently: nobody was told it existed
        }
        return out;
    }
}

A failed query is not an empty result. If the ingesters time out, the engine skips that check and keeps its state. Treating the error as “nothing is above the threshold” would resolve every firing alert at the exact moment the outage got worse.

Routing. The router turns alerts into pages. It merges duplicates, and groups alerts with the same name and service into one page. It waits 30 seconds so related alerts join that page. A firing “cluster down” alert suppresses the per-service alerts it causes. Silences mute alerts during maintenance. And a tree of label rules picks who gets paged.

Two rule engine replicas, three routers, one page

Scroll or pinch to zoom, drag to pan

Color key
Service

Each router replica waits a little longer than the one before it. So the first normally sends, and the others hear about it. If that message is lost, two pages go out. A duplicate page costs annoyance. A missed page costs an outage.

Go deeperThe timeline, and alerts that survive restarts1 min

Time	What happens
0:00	The error rate for `checkout` crosses 5%
0:15	The sample is in the ingesters
0:30	Next check: the alert goes pending
5:30	Held for 5 minutes: firing, sent to all 3 router replicas
6:00	The 30-second wait ends: one page to the payments on-call
6:05	The phone rings, after the paging provider’s delivery

Firing alerts are sent again every minute, with an end time a few minutes ahead. So if a rule engine dies, the router resolves them by timeout. Each alert’s start time is saved as a series. So a rule engine that restarts mid-incident restores its timers, instead of waiting 5 more minutes. A page also repeats every 4 hours while it’s still firing.

Dashboards at ten times, and rollups that don’t lie

The query front end makes repeated dashboards cheap. Long ranges are the other half of the read problem: someone asks for p99 latency from last March.

Rollups. Each 1-hour rollup point stores min, max, sum and count. For counters, it also stores the increase within the hour. An average alone would hide a 30-second spike, and would break any later re-averaging. Percentiles can’t be averaged. The p99 of each hour, averaged, is not the day’s p99. So teams export histograms. Each bucket is a counter, which rolls up correctly, and the p99 is computed from the summed buckets at query time. A query’s step picks the tier. A 1-hour step over 13 months reads 1-hour rollups: about 9,000 points per series, instead of 3 million.

Go deeperWhat a rollup must store1 min

Stored per window	Answers correctly	Gets wrong
Average only	The average	Max and min (a 30 s spike vanishes), sums, and any later re-averaging
Last value only	Gauges read at a moment	Everything that happened inside the window
min, max, sum, count	Average (sum ÷ count), true min and max	Nothing for gauges; counters need one more value
+ counter increase	`rate` and `increase` across resets	Nothing for counters

The frontend aligns start and end to the step, so each refresh produces the same cache keys. A backfill into old blocks must clear the cached days, or dashboards show old numbers. At a billion series, two more changes come in. First, each team gets a random subset of about 24 ingesters. This is called shuffle sharding. It limits both query fan-out and the damage one bad team can do. Second, each region runs its own copy of this design, with a thin global query layer, as Google’s Monarch does.

Who watches the watchers

Your ingesters are down. Do any alerts fire? Who finds out?

Every alert so far runs through distributors, ingesters, rule engines and the router. If any of them is down, no alert fires, and silence looks like health. Three additions make that failure loud:

A meta-monitor. A small, separate deployment with its own servers, disk, rules and paging setup. It scrapes every part of the main stack and pages on ingest errors and rule-check lag.
A critical copy. Collectors also send the meta-monitor about 50,000 key series: every target’s up, and each service’s request and error counts. It runs the few dozen alerts that must fire even when the main stack is down.
A dead-man switch. One rule, Watchdog, is always firing while the whole pipeline works. The router sends it every minute to a heartbeat check at the paging provider. Five minutes without a ping, and the provider pages. This catches what no rule can: a crashed router, an expired API key, or everything at once.

Metrics and monitoring · architecture

Stage 6: Watching the watchers

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 6 of 6

Watching the watchers

Why this stage

Not load: a failure domain.

If the ingesters or the rule engines are down, no alert fires, and nothing tells anyone that no alert can fire.

What we add

A small, separate meta-monitoring stack that gets a critical copy of the data straight from the collectors, and a dead-man switch: a Watchdog alert that fires only while a canary series flows through ingest, rules and router, and whose silence pages.

What it costs

A second stack to run and keep separate: different machines, storage and paging integration.

At 15M samples ingested per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

The dead-man switch when the alert pipeline dies

Scroll or pinch to zoom, drag to pan

Color key
Service

The meta-monitor pings its own heartbeat check, and each stack alerts on the other. Anything the two stacks share, a deploy pipeline or a credential, fails with both, so they share nothing.

When things break

Interviewerminute 40

An ingester dies at peak. What do on-call engineers see, and do alerts still fire?

What they are testing

Whether you know each series has three copies, so data and alerts survive, and what the restart costs.

An ingester dies. Every series it held has two more copies, one in each other zone. Writes need 2 of 3, so they carry on. Queries and rule checks read the two survivors and merge their answers, so graphs show no gap and alerts keep firing. When the ingester comes back, it replays its write-ahead log before taking traffic. If the disk is gone too, a new ingester joins empty and fills up over the next two hours, while the other copies cover it.

Losing one copy is the easy case. The worst failure is silence that looks like health. So for every part of the main stack, the question is: if it dies, who pages? The meta-monitor from the stress tests answers it:

What is down	Main alerts	Critical alerts	Who knows
Object storage, store gateways	Fire normally	Fire	Dashboards show partial history
Ingesters or distributors	Stop	Fire	The meta-monitor pages “ingest down”
Rule engines	Stop	Fire	The meta-monitor pages on check lag
Alert router	Not delivered	Fire	The dead-man heartbeat pages

Under stress, shed load in this order: keep alert checks first, then recent ingest, then dashboards, then long-range queries. The query frontend is the first thing throttled.

Click any box to break it and see what users would notice:

Metrics and monitoring · architecture

Break it: what fails, who notices, what users see

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Break

Failure: click any box to break it

An ingester goes down

What breaks

Its share of the in-memory head is gone until it restarts and replays its write-ahead log.

How we notice

Ring heartbeat missing for 1 minute, and replay progress.

What users see

Nothing while 2 of the 3 replicas of every series are up.

Losing a second ingester in another zone fails writes for the series both held.

What we do

Replication 3 across zones, quorum 2.

On restart it replays the log (minutes for 50 GB), then rejoins the ring.

Roll out one zone at a time.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.

Go deeperEvery failure, and what users see1 min

Part	What users see	What we do
A service	Its team’s new series refused; nobody else	Limit per team; drop the label
Collector	Gaps for one cluster; absence alerts fire	Second collector; replay its buffer
Ingester	Nothing while 2 of 3 copies are up	Replay the log; roll out zone by zone
Object storage	Queries older than a day fail or are partial	Blocks kept locally until uploaded
Store gateway	Long queries slow for minutes	Each block on 2 gateways
Querier, frontend	Panels error and retry; alerts unaffected	Query limits, splitting, replicas
Rule engine	Its rules delayed about a minute	Others take over; state restored
Alert router	All down: main alerts not delivered	3 replicas; the provider pages

Operation	Guarantee	What users can see
Ingest a sample	2 of 3 copies, at least once from the collector	Retries are harmless: same series and timestamp, one sample
Read recent data	Reads merge all 3 copies	The newest 1 to 2 seconds may be missing; never wrong values
Alert notification	At least once, duplicates merged	A split between routers can send two pages, never zero
Silences	Shared between routers within seconds	A page can slip through just after a silence is created

Exactly-once delivery to a phone doesn’t exist: the provider can accept a page and fail to say so. Designing for exactly once tends to cause the opposite failure: a page nobody sent, because each replica thought another had.

Go deeperRunning it: alerts, deploys and cost1 min

Wake someone up when (the meta-monitor checks all of these):

a synthetic alert, fired every 5 minutes, doesn’t reach a test receiver within 90 seconds, twice in a row
more than 1% of rule groups miss their checks
ingest success drops below 99.9%
the newest sample from a collector is over 60 seconds old

Slow dashboards are a ticket, not a page.

Deploys. Ingesters roll out one zone at a time, so every series keeps 2 copies. The meta-monitor deploys from its own pipeline. Rules are code: reviewed, checked for label names that don’t exist, and tested against recorded series. A rule that misspells a label never fires, the most common silent failure in alerting.

Cost, largest first: ingester memory (set by series), traffic between zones (about $20,000 a month), query compute, then storage at about $1,000 a month. The cheapest saving is fewer series.

Wrap up

The whole design

Here is everything we built, in the order we built it. Use the buttons to step through it.

Metrics and monitoring · architecture

The full design, stage by stage

Color keyServiceDatabaseCacheBackground workerNew in this stageAsync or occasional

Stage

Samples ingested per second: 15,000,000

Stage 6 of 6

Watching the watchers

Why this stage

Not load: a failure domain.

If the ingesters or the rule engines are down, no alert fires, and nothing tells anyone that no alert can fire.

What we add

What it costs

A second stack to run and keep separate: different machines, storage and paging integration.

At 15M samples ingested per second, every part has headroom. Raise the traffic until something turns red.

Pinch or tap the zoom buttons to read small labels.Drag any box to move it. Reset puts every box back.The whole design. Step through it: one monitoring server, collectors and sharded ingesters, long-term storage, a separate alert path, a query frontend, then a meta-monitor that watches the watchers.

The numbers behind the busy bars

What one machine can do

One monitoring server ingests about 1 million samples a second.
A distributor forwards about 400,000 samples a second.
An ingester holds at most about 6 million series, about 8 KB each in a 50 GB process. At a 10-second interval, that is 600,000 samples a second.
Every sample is written to 3 ingesters.

Queries

The querier pool is 32 queriers of about 500 query units a second each. A query that reads object storage costs about 6 units.
For every 10 million samples a second, there are about 6,700 rule evaluations and 2,000 dashboard queries a second. 30% of the dashboard queries cover more than 2 hours.
The results cache answers about 80% of dashboard work.

Headroom

Scaled-out tiers are kept near 60% busy.

The decisions, and what each one costs

Decision	We chose	We gave up	We’d choose differently if
Collection	Pull locally, push centrally	One fewer part per cluster	Workloads were serverless (push)
Sharding	Series hash, 3 copies across zones	Queries that touch one shard	Queries were always per team
Durability	2-of-3 writes plus a local log	Replay after a double failure	Metrics drove billing (a log in front)
Storage	Gorilla chunks in blocks on object storage	Cheap deletes, out-of-order writes	Data were high-cardinality events (columnar)
Queries	A frontend that splits, caches and queues	Freshness of cached ranges	Dashboards were rare
Alerting	Own rule engines on ingesters, a router cluster	One system to run	Rules were few
Meta-monitoring	A separate stack and a dead-man switch	The cost of a second stack	Never: this one is not optional

How to spend 45 minutes

0 to 4. Series count, sample interval, retention, alert speed.
4 to 8. 100 million series, 2.4 TB of memory, 1.2 TB a day, 6,700 rule checks a second.
8 to 20. Build it: collectors, ingesters by series hash, object storage, the alert path. A mid-level answer can stop here.
20 to 40. The stress tests: cardinality first, then alerting, then rollups or the meta-monitor, then compression if asked.
40 to 45. An ingester dying, then your own questions.

Never skip cardinality, the alert path, or the meta-monitor.

Follow-up questions

Interviewer asksA collector was cut off for an hour and now wants to send its backlog. What happens?

Ingesters refuse its samples as out of order, because newer ones exist. There are two fixes. Allow a bounded out-of-order window in memory (Prometheus supports this, at some cost). Or have the collector write its backlog as a block, for the compactor to merge. Either way, cached results for that hour must be cleared.

Interviewer asksAn alert flaps between firing and resolved every few minutes. How do you fix it?

for: only delays firing. Flapping comes from resolving too eagerly. Use two thresholds (fire above 5%, resolve below 3%). Or keep it firing for a few minutes after it clears. Or alert on a longer window, as SLO burn-rate alerts do.

Go deeperMore follow-up questions1 min

Legal asks you to delete every series with a certain label. How? Blocks never change, so a delete writes a tombstone per block. Queries hide the data at once, and the compactor rewrites the blocks, and the rollups, without it.

A database outage fires 600 alerts across 40 services. What does on-call see? Grouping by alert name and service makes it 40 pages. Inhibition makes it one: a firing “database down” alert suppresses alerts from services labelled as depending on it.

This is now a product sold to customers. What changes? Tenancy becomes a security boundary: the tenant comes from authentication, not a header, and every cache key includes it. Limits become plan tiers billed by active series.

How do you test that an alert would actually fire? Unit-test rules against synthetic series (promtool test rules). In production, the synthetic alert proves the pipeline, and game days break a real dependency on purpose.

Design a metrics and monitoring system

The answer in one minute

Scope it

The numbers that decide it

Build the design

Step 1. One monitoring server

Stage 1: One monitoring server

Why this stage

What we add

What it costs

Step 2. Collectors and sharded ingesters

Collectors: pull nearby, push to the center

Ingesters, sharded by series

Stage 2: Collectors and sharded ingesters

Why this stage

What we add

What it costs

Trace a request: One scrape, ingested

Step 3. History on object storage

Stage 3: Long-term storage

Why this stage

What we add

What it costs

Step 4. Alerts on their own path

Stage 4: A separate alerting pipeline

Why this stage

What we add

What it costs

Trace a request: One rule evaluation

Step 5. A query front end

Stage 5: A query frontend

Why this stage

What we add

What it costs

Trace a request: A 7 day dashboard panel

Stress it

Cardinality

Storing a sample in 1.4 bytes

From a rule to a phone ringing

Dashboards at ten times, and rollups that don’t lie

Who watches the watchers

Stage 6: Watching the watchers

Why this stage

What we add

What it costs

When things break

Break it: what fails, who notices, what users see

What breaks

How we notice

What users see

What we do

Wrap up

The whole design

The full design, stage by stage

Why this stage

What we add

What it costs

The decisions, and what each one costs

How to spend 45 minutes

Follow-up questions

Further reading

Keep reading

The designbeyondcode digest, every Saturday