Scribd
Upload
2 views2 pages

Message Queue

The document outlines the design of a distributed message queue similar to Kafka, focusing on reliable asynchronous communication with a scale estimate of 100 million daily active users and peak requests of 50,000 per second. It details a high-level architecture, core components including databases and a worker pool, as well as API design and non-functional requirements such as 99.99% availability. Additionally, it discusses key trade-offs, failure scenarios with mitigations, and strategies for scaling the system effectively.

Uploaded by

gregory.borodin
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views2 pages

Message Queue

The document outlines the design of a distributed message queue similar to Kafka, focusing on reliable asynchronous communication with a scale estimate of 100 million daily active users and peak requests of 50,000 per second. It details a high-level architecture, core components including databases and a worker pool, as well as API design and non-functional requirements such as 99.99% availability. Additionally, it discusses key trade-offs, failure scenarios with mitigations, and strategies for scaling the system effectively.

Uploaded by

gregory.borodin
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

System Design: Message Queue

Problem Statement
Design a distributed message queue like Kafka for reliable asynchronous communication.

Scale Estimates
Metric Estimate

Daily Active Users 100M

Peak Requests/sec 50K

Total Storage 100TB

High-Level Architecture

Client --> CDN --> Load Balancer --> API Gateway --> Service Layer --> Data Layer
| |
Rate Limiter Cache Layer

Core Components
Primary Database (PostgreSQL)
Cassandra Ring
Load Balancer (Nginx/HAProxy)
Worker Pool (Celery)
CockroachDB
TimescaleDB

Data Model
Key entities and their relationships:

Entity Storage Access Pattern

message_queue_record Primary DB Read-heavy, indexed by ID

message_queue_metadata Cache + DB Write-through caching

message_queue_event Event Store Append-only, time-partitioned

message_queue_audit Cold Storage Write-once, query by date range

API Design

POST /api/v1/message-queue - Create resource


GET /api/v1/message-queue/{id} - Get by ID
PUT /api/v1/message-queue/{id} - Update resource
GET /api/v1/message-queue?page=1 - List with pagination
DELETE /api/v1/message-queue/{id} - Soft delete
GET /api/v1/message-queue/health - Health check endpoint

Non-Functional Requirements
Availability: 99.99% uptime (< 52 min downtime/year)
Throughput: 10,000 req/s per node
Observability: Distributed tracing, < 1% overhead

Key Trade-offs
Storage vs Compute: Pre-computed aggregations reduce query time but increase storage 3x
Push vs Pull: Push model gives lower latency at higher server resource cost

Failure Scenarios & Mitigations


1. Node Failure: Automatic failover with health checks every 10s
2. Network Partition: Graceful degradation returning cached/stale data
3. Data Corruption: Checksums on write, periodic integrity audits
4. Cascading Failure: Circuit breakers with exponential backoff
5. Hot Partition: Consistent hashing with virtual nodes for even distribution

Scaling Strategy
Read scaling: Add read replicas and cache layers
Write scaling: Partition/shard by tenant or entity ID
Compute scaling: Auto-scale worker pools based on queue depth
Storage scaling: Tiered storage (hot/warm/cold) with lifecycle policies

System Design Document | Generated for study and reference purposes

You might also like