Design a Web Analytics Platform
Problem
Design a hosted web analytics product. Website owners add a small JavaScript snippet to their pages, which reports page views and custom events. Owners open a dashboard showing page views, unique visitors, top pages, referrers, countries, and devices for any date range, with data appearing within a few minutes. The platform serves 1 million websites that together send 20 billion events per day, and the largest customers have more than 100 million page views per day.
- Which metrics must be exact, and can unique visitor counts be approximate within a percent or two?
- How long is raw event data retained, and how long are aggregates kept?
- Do customers need custom events with properties and funnel reports, or only page views?
- Must the product work without cookies to respect privacy rules such as GDPR?
- Do customers need real-time "visitors right now" views?
- Is there an export API or raw data access for large customers?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Provide a lightweight tracking script that sends page views and custom events.
- Show page views and unique visitors over any date range.
- Break down traffic by page, referrer, country, and device.
- Make new events visible on the dashboard within a few minutes.
- Group events into sessions and report bounce rate and session duration.
- Show a live count of current visitors.
- Let owners filter any report by another dimension, for example top pages for visitors from one country.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- 20 billion events per day: about 230,000 per second average and 1 million per second at peak.
- Dashboard data no more than a few minutes old.
- Dashboard queries under 2 seconds at p95 for a 30 day range, even for the largest sites.
- The tracking script must not slow page loads: under 5 KB and fully asynchronous.
- Events at about 300 bytes raw and 50 bytes compressed in columnar form are about 1 TB per day compressed, with 2 years of raw retention for paid plans.
- Ingestion is highly available; a short dashboard outage is acceptable, but losing events is not.
The tracking script sends events with a beacon request to a fleet of collectors behind a global load balancer, which enrich each event and append it to Kafka. A stream processor sessionizes events per visitor and writes them into a columnar OLAP database, where both raw events and rollups are stored per site and time. Dashboards query the OLAP store directly, using pre-aggregated rollups for common reports and HyperLogLog sketches for unique visitor counts across arbitrary ranges.
Loading diagram...
Track. The script loads asynchronously and, on each page view, sends navigator.sendBeacon to POST /e with the site id, URL, referrer, screen size, and an anonymous visitor id. The collector responds 202 immediately, adds the country from the IP address and the device from the user agent, drops the raw IP, and appends the event to Kafka partitioned by site id.
Process. The sessionizer consumes events keyed by (site_id, visitor_id). It starts a new session when a visitor has been idle for 30 minutes, assigns a session id, and marks the first and last page of the session. Enriched events are written in micro-batches to the columnar store. A small state update per site also maintains the count of visitors seen in the last 5 minutes for the live view.
Query. The dashboard calls GET /sites/:id/stats?from=&to=&metric=&groupBy=. The query service chooses daily rollups for long ranges and raw events for short or heavily filtered ranges, merges HyperLogLog sketches to compute unique visitors, and caches the response for completed days.
Receives events from browsers at very high volume and gets them into Kafka as fast as possible.
- Go service deployed in many regions behind an anycast load balancer.
- Kafka producer with batching and compression.
API
POST /ewith{ siteId, url, referrer, visitorId, event, props? }returns202.
Scalability
Collectors are stateless and do minimal work per event, so 1 million events per second needs only a few hundred instances. Each region writes to a local Kafka cluster, replicated to the processing region.
Tradeoffs
Responding 202 before processing keeps the browser request short. A visitor id derived from a daily rotating hash of site, IP, and user agent is offered as a cookie-free option, trading cross-day visitor tracking for privacy compliance.
Groups events into sessions and writes enriched events to storage.
- Apache Flink with keyed state and session windows.
Diagram
Loading diagram...
Scalability
State is keyed by visitor, which spreads load evenly even for the largest sites. Session state is small and expires after 30 minutes of inactivity, so total state stays bounded by the number of concurrently active visitors.
Tradeoffs
Sessionizing in the stream rather than at query time makes session metrics cheap to query, at the cost of reprocessing if the session definition changes. Raw events in the archive make that reprocessing possible.
Stores every enriched event and the rollups that power dashboards.
- ClickHouse, chosen for fast scans, compression, and built-in HyperLogLog aggregate functions.
Data Model
events(site_id, timestamp, visitor_id, session_id, url_path, referrer_domain, country, device, browser, event_name, props)ordered by(site_id, timestamp)and partitioned by month.daily_rollup(site_id, day, url_path, referrer_domain, country, device, pageviews, visitors_hll)maintained by a materialized view.
Scalability
Data is sharded by site_id across nodes, with very large sites spread over several shards by a secondary hash. Ordering by site and time means a dashboard query reads only one site's contiguous range. Compression of about 6x keeps 2 years of data manageable on a few hundred nodes.
Tradeoffs
A columnar store is chosen over a row store because analytics queries read a few columns across millions of rows. Storing HyperLogLog sketches instead of visitor id lists lets unique counts be merged across any days with about 1% error and a few kilobytes per row; exact unique counts over arbitrary ranges would need a full scan of visitor ids.
Translates dashboard requests into OLAP queries, enforces tenant isolation, and caches results.
API
GET /sites/:id/stats?from=&to=&metrics=pageviews,visitors&groupBy=url&filter=country:DEreturns rows.GET /sites/:id/livereturns the current visitor count.
Scalability
Responses for fully completed days never change, so they are cached in Redis keyed by query. Per-tenant query quotas stop one large customer's heavy report from slowing everyone else.
Tradeoffs
Routing queries to rollups when possible gives fast dashboards for common views, while falling back to raw events keeps arbitrary filters possible. The service, not the client, decides which table to use, so storage can evolve without breaking the dashboard.
Long-term copy of every raw event for reprocessing, export, and backfills.
- Kafka Connect writing Parquet files to object storage, partitioned by date and site.
Tradeoffs
Object storage is much cheaper than OLAP nodes for data that is rarely read. Keeping it means a bug in enrichment or sessionization can be fixed and the affected days rebuilt without asking customers to resend data.