Design a Notification System
Problem
Design a shared service that other teams call to notify users. It must send notifications over mobile push (iOS and Android), SMS, and email, through third-party providers such as APNs, FCM, an SMS gateway, and an email service. Users can opt out of categories and channels, and the system must not spam users with too many messages. It should handle 10 million notifications per day on average, with spikes of 1 million in a few minutes when a marketing campaign or breaking news alert goes out. Transactional messages, such as login codes, must arrive within seconds even during a campaign.
- Which notification types exist: transactional (password reset, order shipped), social, and marketing?
- Do callers send fully rendered messages, or a template name and variables?
- Is delivery at least once acceptable, or must duplicates be avoided?
- Do we need scheduled sends, such as delivering in the user's local morning?
- Should we track delivery, opens, and clicks?
- Are there regulatory requirements such as quiet hours or unsubscribe links in email?
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Send notifications via push, SMS, and email.
- Respect user opt-outs per category and channel.
- Limit how many notifications a user receives in a time window.
- Deliver transactional notifications ahead of bulk campaigns.
- Render messages from templates with variables and localization.
- Support scheduled sends and track delivery status per message.
Items in bold are existing requirements. The rest are suggestions. Confirm with the interviewer whether they are relevant first.
- Handle 10 million notifications per day (about 115 per second) and bursts of 1 million in a few minutes (about 5,000 per second).
- Transactional notifications are handed to the provider within 5 seconds at p99, even during a bulk send.
- At-least-once delivery, with deduplication to keep duplicates rare.
- High availability (99.9%); a provider outage must not lose messages.
- Store delivery logs for 90 days: at about 500 bytes per event and three events per notification, roughly 1.4 TB.
- Stay within provider rate limits and costs (SMS is expensive, so prefer push when allowed).
Callers submit notification requests to an API that validates them, stores them, and enqueues them. A processing tier applies user preferences, rate limits, and deduplication, renders the template, and fans out to per-channel queues split by priority. Channel workers deliver to third-party providers with retries and backoff, and delivery events feed a status store that callers and dashboards can query.
Loading diagram...
Submit. A service calls POST /notifications with the user ID, a category, a template name, variables, and an idempotency key. The API rejects a repeated idempotency key, writes the request to the notification store with status accepted, and publishes it to the intake queue (high or low priority topic based on category). Scheduled requests are held by the scheduler until their send time.
Process. The processor loads the user's preferences and devices. It drops channels the user opted out of, checks quiet hours and the per-user rate limit for the category, and picks channels by policy (for example push first, SMS only for transactional messages when no device is registered). It renders the template in the user's language and publishes one message per channel and device to the matching channel queue.
Deliver. A channel worker sends the message to the provider. On success it records sent with the provider's message ID. On a retryable error (timeout, 429, 5xx) it republishes with exponential backoff; after the maximum attempts it moves the message to a dead letter queue. Permanent errors such as an invalid device token mark the token inactive. Provider callbacks (delivered, bounced, opened) arrive at a webhook and are written to the notification store as further status events.
Entry point for internal services that want to notify users.
- Stateless HTTP or gRPC service with service-to-service authentication.
API
POST /notificationswith{ userId, category, template, variables, channels?, sendAt?, idempotencyKey }returns{ notificationId }.POST /notifications/bulkwith a template and a user segment or list, for campaigns.GET /notifications/:idreturns status per channel.PUT /users/:id/preferencesupdates opt-outs, used by the settings page.
Scalability
The API only validates, writes one row, and publishes to the queue, so it handles bursts easily. Bulk campaign requests are expanded into individual notifications by a separate job, keeping a single API call from blocking.
Tradeoffs
Accepting a template name and variables instead of rendered text keeps branding, localization, and unsubscribe links consistent and lets the system re-render for each channel. The cost is that callers must register templates first, which is a small process overhead worth the consistency.
Applies business rules that decide whether, where, and what to send.
- Consumer group on Kafka or SQS, horizontally scaled.
- Redis for per-user rate limit counters and recent deduplication keys.
Diagram
Loading diagram...
Scalability
Processing is stateless apart from Redis lookups, so adding consumers scales throughput linearly up to the partition count. Preferences are cached in memory for a short time, since they change rarely and are read for every notification.
Tradeoffs
Applying preferences and rate limits centrally, rather than in each calling service, guarantees that no team can accidentally spam users or ignore an opt-out. The processor becomes a shared dependency, so it is kept simple and isolated from slow provider calls by the channel queues.
Per-channel, per-priority queues with workers that call third-party providers.
- Separate topics such as
push-high,push-low,sms-high,email-low, with worker pools per topic. - Provider SDKs (APNs over HTTP/2, FCM HTTP v1, SMS and email REST APIs).
Scalability
Separate high and low priority queues with dedicated workers mean a campaign of 1 million marketing emails never delays a login code. Workers enforce each provider's rate limit with a token bucket and scale out to drain a backlog within minutes.
Tradeoffs
A single shared queue is simpler, but one slow provider or one huge campaign would block every other channel. Isolating queues by channel and priority contains failures. When a provider is down, its messages pile up in its queue and are retried later rather than lost; for SMS, a second provider can be configured as a failover.
Stores what each user wants to receive and where.
- A relational database (PostgreSQL) for preferences, read through a cache.
Data Model
user_preferences(user_id, category, channel, enabled, updated_at)devices(device_id, user_id, platform, push_token, app_version, last_seen, is_active)user_contact(user_id, email, phone, timezone, locale, quiet_start, quiet_end)
Scalability
Reads dominate and are served from cache; writes happen only when a user changes settings or registers a device. Partitioning by user_id is straightforward if the table outgrows one primary.
Tradeoffs
Defaulting new categories to enabled maximizes reach but risks annoying users, so marketing categories default to off where regulation requires consent. Stale push tokens are deactivated when providers report them invalid, which keeps send costs and error rates down.
Records the lifecycle of every notification for debugging, analytics, and caller queries.
- Delivery events on Kafka, written to a wide-column store such as Cassandra (or DynamoDB) with a 90 day TTL.
- Provider webhooks for delivered, bounced, opened, and clicked events.
Data Model
notification_events(notification_id, channel, event_time, status, provider_message_id, error_code)
Scalability
Events are append only and partitioned by notification_id, which matches the main query (status of one notification). Aggregate metrics such as delivery rate per template are computed by a stream job and stored separately.
Tradeoffs
Exactly-once delivery is not possible across third-party providers, because a timeout leaves it unknown whether the provider accepted the message. The system chooses at-least-once with a deduplication key per notification and channel, kept in Redis for 24 hours, so most retries are caught. A rare duplicate push is far better than a missed login code.