Design a Log Aggregation System
Problem
Design a system that collects application and system logs from 50,000 servers into one place so engineers can search them by keyword, service, host, and time range and tail live logs while debugging. The fleet produces about 2 TB of logs per day, with bursts of 5x during incidents. Logs must be searchable within 30 seconds of being written and retained for 30 days, with older logs archived cheaply for compliance.
- Are logs structured (JSON) or free text, and should we parse free text into fields?
- What query patterns matter most: full-text keyword search, field filters, or aggregations like counts per minute?
- How long must logs stay searchable versus merely retained for compliance?
- Is it acceptable to drop or sample debug-level logs during a burst?
- Do logs contain sensitive data that must be redacted before storage?
- How many engineers query at once, and how far back does a typical query reach?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Collect logs from all servers and applications into a central store.
- Search logs by keyword and by fields (service, host, level) over a time range.
- Live tail logs for a service or host.
- Retain logs for 30 days and archive older logs.
- Parse common formats into structured fields and redact secrets on ingestion.
- Show log volume histograms and counts by field for a query.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Ingest 2 TB per day: about 23 MB per second average and 115 MB per second during 5x bursts.
- Logs searchable within 30 seconds of emission.
- At 200 bytes per line, about 10 billion lines per day, or 115,000 lines per second on average.
- 30 days searchable is 60 TB raw; with replication and index overhead, plan for about 150 TB of hot and warm storage.
- Searches over the last hour return in under 2 seconds; searches over 30 days may take tens of seconds.
- Ingestion must never block or slow the applications that produce logs.
A lightweight agent on every server tails log files and ships batches to a durable message queue, which absorbs bursts and decouples producers from indexing. Indexer workers parse, enrich, and redact each line, then write it to a search cluster organized as time-based indices that move from fast SSD nodes to cheaper nodes as they age. After 30 days, compressed raw logs remain only in object storage, and a query service fans out searches and serves live tail directly from the queue.
Loading diagram...
Ship. The agent tails log files and container stdout, tracking a file offset on local disk so a restart resumes where it left off. It batches lines for up to one second or 1 MB, compresses the batch, and sends it to the ingest gateway. If the gateway is unreachable, the agent buffers to a bounded local disk spool, then drops the oldest debug logs first.
Buffer. The ingest gateway authenticates the agent, attaches host metadata, and produces the batch to the log queue, partitioned by service. It acknowledges once the queue write is replicated.
Index. Indexer workers consume batches, parse each line (JSON, or a configured pattern for free text), redact values that match secret patterns, and bulk-write documents to the current daily index for that service group. The search cluster refreshes every 5 to 10 seconds, making new documents searchable well within 30 seconds.
Search. An engineer queries service:checkout AND level:error AND "timeout" for the last 2 hours. The query service routes the request only to indices whose time range overlaps the query, runs it in parallel across shards, and merges the newest results first.
Live tail. The tail service runs a consumer on the log queue, filters lines that match the user's query, and streams them to the browser over a WebSocket.
Runs on every server, tails logs, and ships them reliably without hurting the host.
- Fluent Bit or Vector, chosen for a small memory footprint and built-in disk buffering.
Scalability
Each agent caps its CPU and memory use and applies backpressure by buffering to disk instead of blocking the application. Batching and compression (typically 5x to 10x for logs) cut network traffic and request count.
Tradeoffs
Writing to local files and tailing them, rather than having applications send logs over the network directly, means a slow or failed log pipeline never stalls the application. The cost is a small delay and the need to track file offsets carefully across log rotation.
Durable buffer between agents and indexers that absorbs bursts and allows replay.
- Kafka with 3x replication and 3-day retention.
Scalability
At a 115 MB per second peak, a few hundred partitions spread across a modest broker cluster handle the load. Three days of retention at 2 TB per day compressed to about 400 GB fits easily, and lets indexers fall behind during an incident and catch up without losing data.
Tradeoffs
Writing straight from agents to the search cluster removes a hop but couples ingestion to indexing speed: when the cluster slows during an incident (exactly when logs spike), agents would back up or drop data. The queue costs extra infrastructure but makes indexing failures recoverable by replay.
Parse, enrich, redact, and bulk-index log lines.
- Stateless consumers in Go or Logstash, scaled by consumer lag.
Data Model
- Document:
{ timestamp, service, host, level, message, trace_id, fields: {...} }. - One index per service group per day, for example
logs-payments-2026.11.06.
Scalability
Bulk writes of a few thousand documents per request keep indexing overhead low. Workers autoscale on queue lag, and each consumer group can be sized independently from the archiver and live tail.
Tradeoffs
Daily indices per service group make retention a cheap whole-index delete instead of millions of row deletes, and let queries skip indices outside the time range. Very small services are grouped together to avoid thousands of tiny indices, which waste cluster memory.
Stores indexed logs and answers search queries, tiered by age.
- Elasticsearch or OpenSearch with index lifecycle management.
Diagram
Loading diagram...
Scalability
Most queries target the last few hours, so the hot tier gets fast SSDs and one replica, while the warm tier uses dense, cheaper disks. Shards are sized at 20 to 50 GB so each daily index uses a predictable number of shards.
Tradeoffs
A full-text inverted index gives fast arbitrary keyword search, but indexing every field can double storage. An alternative like Loki indexes only labels (service, host) and brute-force scans compressed chunks, which is far cheaper to store but slower for needle-in-a-haystack searches. Indexing only commonly searched fields plus the message text is a middle ground.
The archiver writes compressed raw logs to object storage; the query service routes searches and handles rehydration.
- Archiver: Kafka consumer writing hourly compressed files to S3, moving to a cold storage class after 30 days.
- Query service: stateless API in front of the cluster.
API
POST /searchwith{ query, from, to, limit, cursor }returns matching lines and a histogram.GET /tail?query=upgrades to a WebSocket that streams matching lines.POST /rehydratewith{ service, from, to }restores archived logs into a temporary index.
Tradeoffs
Archiving from the queue rather than exporting from the search cluster keeps the archive complete even if indexing fails. Storing a year of compressed logs in cold object storage costs a small fraction of keeping it indexed, and compliance requests are rare enough that a rehydration delay of minutes to hours is acceptable.