Design a Metrics Monitoring and Alerting System
Problem
Design a system that collects numeric metrics (CPU usage, request latency, error counts) from a fleet of 100,000 hosts, stores them as time series, and lets engineers query and graph them on dashboards. Users can define alert rules such as "error rate above 5% for 5 minutes" and receive notifications through email, chat, or paging. Each host emits about 1,000 series every 10 seconds, and recent data must be queryable within a few seconds of being emitted.
- Do agents push metrics to us, or do we pull (scrape) them from hosts?
- How long must raw data be kept, and is lower resolution acceptable for older data?
- What query language do users expect: simple filters by tag, or a full expression language with aggregations?
- How many unique tag combinations (cardinality) should we plan for, and do we need to protect against cardinality explosions?
- How quickly must an alert fire after the condition becomes true?
- Should alerting keep working if the dashboard query path is degraded?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Ingest metrics (name, tags, timestamp, value) from 100,000 hosts.
- Query time series by name and tags with aggregations (sum, avg, percentiles) over a time range.
- Render dashboards of recent and historical data.
- Define alert rules over metric queries and send notifications when they fire and resolve.
- Silence alerts during maintenance and group related alerts into one notification.
- Downsample old data to lower resolution automatically.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Ingest 100,000 hosts x 1,000 series / 10 s = 10 million data points per second.
- New data visible to queries within 10 seconds.
- About 100 million active series. At roughly 1.5 bytes per point after compression, raw data is about 1.3 TB per day.
- Keep 10-second data for 15 days (20 TB), 1-minute rollups for 90 days, and 1-hour rollups for 2 years.
- Dashboard queries over the last hour return in under 500 ms at p99.
- Alert evaluation must be more available than dashboards: alerts fire within 1 minute of the condition being met.
Lightweight agents batch metrics and push them to a stateless ingestion tier, which writes to a durable log partitioned by series. Storage nodes consume the log, keep the most recent two hours of each series in memory using delta and XOR compression, and flush immutable blocks to object storage, where a compactor builds downsampled rollups. A query service fans out across storage nodes and object storage, while an independent alert evaluator runs rules on a schedule and hands firing alerts to a notification manager.
Loading diagram...
Ingest. Each agent collects local metrics, batches about 10 seconds of points, compresses the batch, and sends it to the ingestion gateway. The gateway authenticates the tenant, enforces per-tenant series limits, and appends each point to the metrics log partition chosen by hashing the series key (metric name plus sorted tags). It acknowledges once the log write is durable.
Store. Each storage node consumes a set of log partitions and appends points to an in-memory head block per series. Every two hours it seals the head block, writes it to object storage as an immutable, compressed block with its own index, and truncates the log up to that offset.
Query. A dashboard sends sum(rate(http_errors{service="checkout"}[5m])) by (region) for the last 6 hours. The query service resolves the matching series IDs from the series index, picks the cheapest resolution that satisfies the time range and step, fetches chunks from storage nodes (recent data) and object storage (older blocks) in parallel, and merges and aggregates the results.
Alert. The alert evaluator runs each rule every 30 seconds through the query service. When a rule's condition holds for its for duration, it sends a firing alert to the notification manager, which deduplicates, groups by labels, applies silences, and routes to the configured channel.
Stateless HTTP and gRPC tier that accepts batches from agents and writes them to the log.
- Go service behind an L4 load balancer; Kafka as the metrics log.
API
POST /v1/writewith a compressed protobuf batch of{ series: [{ name, tags, samples: [[ts, value]] }] }.
Scalability
At 10 million points per second, batching per agent keeps the request rate to about 10,000 per second. The log is partitioned into a few thousand partitions by series hash so each storage node receives a stable subset of series.
Tradeoffs
Push is chosen over pull scraping because 100,000 hosts behind many networks are easier to reach outbound than inbound, and short-lived batch jobs can push before they exit. Pull gives a free health signal (a failed scrape means the target is down), so agents also send a heartbeat series to recover that signal. The durable log decouples ingestion from storage, so a storage node restart replays from the log instead of losing data.
Hold recent data in memory for fast queries and turn it into compressed, immutable blocks.
- Custom time series engine in the style of Prometheus TSDB or M3, with Gorilla-style compression.
Diagram
Loading diagram...
Data Model
- Series key
name{k1=v1,k2=v2}hashed to a 64-bit series ID. - Chunk: timestamps encoded as delta of deltas, values encoded with XOR against the previous value, about 1.5 bytes per point.
Scalability
With 100 million active series and about 4 KB of head memory per series, the cluster needs about 400 GB of RAM, spread across around 50 nodes with replication factor 2. Series are assigned to nodes by consistent hashing so adding nodes moves only a fraction of series.
Tradeoffs
A general purpose database such as PostgreSQL stores each point as a row of 30 bytes or more and cannot absorb 10 million inserts per second economically. Columnar compression specialized for regular timestamps and slowly changing values gives a 10x to 20x size reduction, at the cost of being append-only: late or out-of-order points older than the head block are rejected or written to a separate out-of-order block.
The index maps tag matchers to series IDs; the compactor merges blocks and builds rollups.
- Inverted index per block (tag value to sorted list of series IDs), cached in memory for recent blocks.
- Compactor jobs that run on object storage blocks.
Data Model
postings[tag=value] -> [series_id...], intersected to resolveservice="checkout" AND region="us-east".- Rollup blocks store
min, max, sum, countper series per 1-minute and 1-hour bucket.
Scalability
Cardinality is the main threat: one tag holding user IDs can create millions of series. The gateway enforces per-tenant and per-metric series limits and reports the offending tag, which protects the index from blowing up.
Tradeoffs
Keeping min, max, sum, count rather than a single average lets rollups answer averages, extremes, and totals correctly. Percentiles cannot be rolled up from these, so latency metrics are emitted as histograms whose bucket counts sum correctly across hosts and time.
Parses queries, plans which stores to read, and aggregates results.
- PromQL-compatible engine in Go, with a results cache keyed by query and aligned time step.
API
GET /api/query_range?q=&start=&end=&step=returns series of[timestamp, value].
Scalability
Queries are split by time range: the last two hours go to storage nodes, older ranges go to object storage, and each is processed in parallel shards. Dashboard refreshes hit the results cache for all but the newest step, so a dashboard viewed by 50 people costs little more than one viewer.
Tradeoffs
Automatically choosing the rollup resolution from the time range keeps a 1-year graph from scanning 3 million raw points per series. The cost is that very short spikes disappear at low resolution, which is why rollups keep max.
Evaluates rules on a schedule and turns firing alerts into deduplicated, routed notifications.
- Rule evaluators sharded by rule ID, each running in two replicas.
- Notification manager in the style of Alertmanager with a gossip cluster for deduplication.
Data Model
alert_rules(rule_id, tenant, query, condition, for_duration, labels, severity)alert_state(rule_id, label_set_hash, state, active_since, last_notified)
Scalability
Rules are partitioned across evaluators; both replicas of a shard evaluate every rule and send identical alerts, and the notification manager deduplicates by alert fingerprint. This tolerates the loss of an evaluator without missing an alert.
Tradeoffs
Alerting reads through a dedicated query pool that dashboards cannot exhaust, because an expensive ad hoc query must never delay a page. Duplicate evaluation doubles rule cost, which is cheaper than a missed outage. Grouping and inhibition (suppressing per-host alerts when the whole cluster alert fires) prevent an outage from producing thousands of pages.