Distributed Notification System
System Design Interview Document — 50,000 Requests/Second
1. Introduction
This document presents the design of a highly available, distributed notification system capable of handling
50,000 requests per second. The system supports Email, SMS, Push, and In-App notifications, with user
preference management, channel selection, and eventual delivery semantics for non-real-time channels. The
design prioritises scalability, fault tolerance, and low latency.
2. Requirements
Functional Requirements
• Send notifications via Email, SMS, Push (mobile/web), and In-App channels
• Support user preferences: Do-Not-Disturb windows, channel opt-in/opt-out, frequency caps
• Guaranteed at-least-once delivery with deduplication (idempotency keys)
• Support templated and dynamic notification content
• Notification scheduling (send now or at a future time)
• Delivery status tracking per notification and per channel
Non-Functional Requirements
• Throughput: 50,000 notifications/sec sustained; 100k/sec peak
• Availability: 99.99% uptime (< 52 min downtime/year)
• Latency: Push/In-App < 500 ms p99; Email/SMS eventual (< 30 sec)
• Horizontal scalability — scale each component independently
• Durability: No notification lost once accepted by the system
• Observability: Metrics, logs, and distributed tracing
3. Capacity Estimation & Assumptions
Assumptions
• 500M registered users, 50M daily active users
• Average notification payload: 1 KB
• Read:Write ratio for preferences: 10:1
• Retention of notification logs: 30 days
• Peak traffic is 2× average (100k req/sec)
• Third-party providers (SendGrid, Twilio, FCM/APNs) have their own rate limits
Back-of-Envelope Calculations
Throughput : 50,000 notifications/sec
Storage/day : 50,000 × 86,400 × 1KB ≈ 4.3 TB/day (log storage)
30-day logs : ~130 TB (compressed ~40 TB with 3× compression)
Queue depth : 50,000 msg/sec × 5s lag ≈ 250,000 messages in flight
Worker pods : ~200 pods (each handling ~250 req/sec with headroom)
DB write rate: ~50k writes/sec → Cassandra / sharded DB needed
4. High-Level Architecture (HLD)
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ CLIENTS ■
■ (Web App, Mobile App, Internal Services, Cron Jobs) ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ HTTPS
■■■■■■■■■■■■■■■■■■■■■■■■■■▼■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ API GATEWAY (Load Balanced) ■
■ Auth · Rate Limiting · Request Validation ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■
■■■■■■■■■■■■■■■■■■■■■■■■■■▼■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ NOTIFICATION SERVICE (Stateless) ■
■ Preference Lookup · Channel Routing · Idempotency ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
■ ■ ■ ■
■■■▼■■■ ■■■■■▼■■■■ ■■■■▼■■■■ ■■■■■▼■■■■
■Email■ ■ SMS ■ ■ Push ■ ■In-App ■ ← Kafka Topics
■Queue■ ■ Queue ■ ■ Queue ■ ■ Queue ■
■■■■■■■ ■■■■■■■■■■ ■■■■■■■■■ ■■■■■■■■■■
■ ■ ■ ■
■■■▼■■■ ■■■■■▼■■■■ ■■■■▼■■■■ ■■■■■▼■■■■
■Email■ ■ SMS ■ ■ Push ■ ■In-App ■ ← Worker Pods
■Wrkr ■ ■ Worker ■ ■Worker ■ ■Worker ■
■■■■■■■ ■■■■■■■■■■ ■■■■■■■■■ ■■■■■■■■■■
■ ■ ■ ■
■■■▼■■■ ■■■■■▼■■■■ ■■■■▼■■■■■■■ ■■▼■■■■■■■■■■■■■
■Send ■ ■Twilio/ ■ ■FCM/APNs/ ■ ■WebSocket / ■
■Grid ■ ■Vonage ■ ■OneSignal ■ ■SSE / DB Poll ■
■■■■■■■ ■■■■■■■■■■ ■■■■■■■■■■■■ ■■■■■■■■■■■■■■■■
All components are stateless and horizontally scalable. Kafka acts as the central message bus, decoupling
producers from consumers and buffering traffic spikes.
5. Component Deep-Dive
5.1 API Gateway
• Handles authentication (JWT/OAuth2) and authorisation
• Rate limits per tenant/user to prevent abuse
• Validates payload schema and rejects malformed requests early
• Routes traffic to Notification Service via internal load balancer
5.2 Notification Service
• Stateless microservice — scales horizontally behind a load balancer
• Fetches user preferences from Redis cache (< 2 ms) with DB fallback
• Determines eligible channels per user (respects DND, opt-outs, frequency caps)
• Assigns idempotency key and publishes to appropriate Kafka topic(s)
• Writes initial record to Notification Store (status: PENDING)
5.3 Message Queue (Apache Kafka)
• Separate Kafka topics per channel: [Link], [Link], [Link],
[Link]
• Partitioned by user_id or notification_id for ordering and parallelism
• Retention: 24 hours (enough for retry + replay on worker failure)
• Consumer groups per channel worker type for independent scaling
• Dead Letter Queue (DLQ) topic for messages that fail after max retries
5.4 Channel Workers
• Dedicated worker pool per channel — scale independently based on queue depth
• Templating engine renders final notification content from template + user data
• Circuit breaker around third-party API calls (fail fast, avoid cascade failures)
• Retry with exponential backoff: 1s → 2s → 4s → 8s (max 3 retries)
• On final failure, publishes to DLQ for alerting and manual review
• Updates Notification Store with final status (DELIVERED / FAILED)
5.5 User Preference Store
• Primary store: Cassandra (wide-column, high write throughput, multi-region)
• Cache layer: Redis (TTL 5 min) to serve hot preference lookups in < 2 ms
• Schema: user_id → {channel: {enabled, dnd_start, dnd_end, frequency_cap}}
• Write-through cache on preference update to keep cache consistent
5.6 Notification Store
• Stores all notifications with status, channel, timestamps, and delivery receipt
• Cassandra: partitioned by user_id, clustered by created_at for time-range queries
• Used for In-App notification inbox (users poll or subscribe via WebSocket/SSE)
• TTL-based expiry: auto-delete after 30 days
6. Data Flow — Send a Notification
1. Client → POST /v1/notifications (API Gateway)
2. Gateway → Auth, rate-limit, validate → Notification Service
3. Svc → Fetch user prefs (Redis/Cassandra)
4. Svc → Evaluate channels (DND? opted-out? freq cap?)
5. Svc → Write PENDING record to Notification Store
6. Svc → Publish message(s) to Kafka topic(s)
7. Worker → Consume from Kafka, render template
8. Worker → Call 3rd party API (SendGrid / Twilio / FCM)
9. Worker → On success: update store → DELIVERED
10. Worker → On failure: retry (exp backoff) → DLQ if exhausted
7. High Availability & Fault Tolerance
• Multi-region active-active deployment — traffic routed via global load balancer (e.g. AWS
Route53/Cloudflare). Any region can serve requests.
• No single point of failure — every component (API GW, Notification Svc, Kafka, Workers, DBs) runs with
≥ 3 replicas across availability zones.
• Kafka replication factor = 3 — messages survive loss of up to 2 brokers.
• Circuit breakers around all external calls — prevents cascade failures when a 3rd party provider is
degraded.
• Graceful degradation — if Email provider is down, fall back to SMS or In-App if user preference allows.
• Idempotency keys — clients retry safely; duplicate notifications are deduplicated at ingestion.
• Dead Letter Queue — failed messages captured for alerting and replay without data loss.
8. User Preferences & Channel Selection
Preference Decision Flow:
For each target channel:
1. Is user opted-in to this channel? → No → Skip
2. Is current time within DND window? → Yes → Delay or Skip
3. Has frequency cap been reached today? → Yes → Skip
4. Is notification type suppressed? → Yes → Skip
5. ✓ Eligible → Enqueue to channel topic
Preference evaluation is done at ingestion time in the Notification Service. If the primary channel is ineligible,
fallback channels are tried in priority order (configured per user). Scheduled notifications are re-evaluated at
delivery time to respect last-minute preference changes.
9. Monitoring & Observability
• Metrics (Prometheus + Grafana): Throughput, latency (p50/p95/p99), success/failure rates per channel,
Kafka consumer lag, DLQ depth
• Distributed Tracing (Jaeger/Zipkin): End-to-end trace per notification from API ingestion to delivery
• Centralized Logging (ELK Stack): Structured logs from all services; searchable by notification_id,
user_id, channel
• Alerting: PagerDuty alerts on DLQ spike, consumer lag > 10k, error rate > 1%, p99 latency > 2s
• Delivery Dashboard: Real-time view of channel health, 3rd party provider status, and notification funnel
10. Security & Compliance
• All data encrypted in transit (TLS 1.3) and at rest (AES-256)
• API keys and secrets stored in Vault / AWS Secrets Manager — never in code
• PII (phone numbers, emails) encrypted at the field level in the DB
• Audit log for every notification event (who sent what, when, to whom)
• GDPR compliance: user data deletion cascades to notification records
• Rate limiting and DDoS protection at the API Gateway layer
11. Trade-offs & Design Alternatives
Decision Chosen Alternative Reason
■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
Message Queue Kafka RabbitMQ/SQS Throughput, replay
Preference DB Cassandra DynamoDB Multi-region writes
Preference Cache Redis Memcached Rich data structures
In-App delivery WebSocket/SSE Long polling Lower latency
Worker scaling K8s HPA Manual/VM Auto-scale on lag
Deduplication Redis SET+TTL DB unique index Speed (< 1 ms)
12. Open Questions & Future Enhancements
• SLA commitments per channel — should Email have a guaranteed delivery window?
• Multi-tenancy — should different clients (teams/products) be isolated at the queue level?
• Analytics pipeline — track open rates, click rates, and unsubscribe signals
• Smart routing — ML model to predict best channel/time for each user
• Notification batching — group multiple events into a single digest notification
End of Document — Distributed Notification System Design