System Design Topics

Topics Covered
Low-Latency Systems | High-Throughput Services | Concurrent Programming
Performance Profiling & Tuning | SQL/NoSQL Databases | Distributed Caching
Multi-Threading | Real-Time Programming

Page 1
Table of Contents

1. Low-Latency Systems3
2. High-Throughput Services7
3. Concurrent Programming11
4. Performance Profiling and Tuning16
5. SQL vs NoSQL Databases20
6. Distributed Caching25
7. Multi-Threading30
8. Real-Time Programming35
9. Reference Materials & Resources39

Page 2
1. Low-Latency Systems

1.1 What is Latency?

Latency is the time elapsed between initiating a request and receiving the first byte of the
response. In system design, low latency means the system responds to user requests in the
shortest time possible, typically measured in milliseconds or microseconds. At FAANG-scale
companies, even single-digit millisecond improvements can translate to significant revenue
gains and better user experience.

1.2 Why Low Latency Matters

•​ Amazon found that every 100ms of added latency cost them 1% of sales.
•​ Google discovered that a 500ms delay in search results caused a 20% drop in traffic.
•​ In high-frequency trading, microsecond differences determine profit or loss.
•​ Mobile users expect page loads under 3 seconds; anything slower increases bounce
rates dramatically.

1.3 Key Latency Numbers Every Engineer Should Know

Operation Approximate Latency
L1 Cache Reference 0.5 ns
L2 Cache Reference 7 ns
Main Memory Reference (RAM) 100 ns
SSD Random Read 150 µs
HDD Disk Seek 10 ms
Send Packet CA to NL and back 150 ms
Read 1 MB sequentially from Memory 250 µs
Read 1 MB sequentially from SSD 1 ms
Round trip within same data center 0.5 ms

1.4 Techniques for Achieving Low Latency

A. Caching

Page 3
Caching stores frequently accessed data in fast-access memory (like RAM) so it does not need
to be fetched from slower storage or recomputed. This is the single most impactful latency
reduction technique.
In-Memory Caches: Redis and Memcached keep hot data in RAM, providing sub-millisecond
read times. Use cache-aside pattern (check cache first, fall back to DB, then populate cache) or
write-through (write to cache and DB simultaneously).
CDN Caching: Content Delivery Networks like CloudFront or Akamai cache static assets at
edge locations geographically close to users, dramatically reducing round-trip time for static
content.
Application-Level Caching: Use local in-process caches (like Guava Cache in Java or
lru_cache in Python) for data that changes infrequently and is read often.

# Python example: LRU cache for expensive computation

from functools import lru_cache

@lru_cache(maxsize=1024)
def get_user_profile(user_id: str):
# This DB call only happens on cache miss
return [Link](f'SELECT * FROM users WHERE id = {user_id}')

B. Connection Pooling
Creating a new TCP connection or database connection for every request is expensive (TCP
handshake alone is ~1.5 round trips). Connection pooling pre-creates and reuses connections,
eliminating this overhead.
// Java: HikariCP Connection Pool Configuration
HikariConfig config = new HikariConfig();
[Link]("jdbc:postgresql://localhost:5432/mydb");
[Link](20);
[Link](5);
[Link](3000); // 3s max wait
HikariDataSource ds = new HikariDataSource(config);

C. Asynchronous and Non-Blocking I/O

Instead of blocking a thread while waiting for I/O (network call, disk read), non-blocking I/O
allows the thread to handle other work. This is critical in services handling thousands of
concurrent requests.
// [Link]: Non-blocking async I/O
const express = require('express');
const app = express();

[Link]('/user/:id', async (req, res) => {

// Non-blocking: thread is freed while waiting for DB

Page 4
 const user = await [Link]([Link]);
const orders = await [Link](`orders:${[Link]}`);
[Link]({ user, orders });
});

D. Data Locality and Proximity

Place data as close to the computation as possible. This includes co-locating services that
communicate frequently in the same data center or availability zone, using edge computing for
latency-sensitive operations, and partitioning data so that related records are on the same
shard.

E. Protocol Optimization
•​ Use gRPC instead of REST for internal service communication (binary serialization is
5-10x faster than JSON).
•​ Use HTTP/2 or HTTP/3 (QUIC) for multiplexed connections, avoiding head-of-line
blocking.
•​ Use Protocol Buffers or FlatBuffers instead of JSON for serialization to reduce parsing
overhead.

F. Kernel Bypass and Zero-Copy

For ultra-low-latency applications (trading systems, real-time gaming), bypass the kernel
network stack using technologies like DPDK (Data Plane Development Kit) or io_uring in Linux.
Zero-copy techniques like sendfile() avoid copying data between kernel and user space.

1.5 Measuring and Monitoring Latency

•​ P50 (median): 50% of requests are faster than this. Tells you the typical experience.
•​ P95: 95% of requests are faster. This is the most commonly used SLA metric.
•​ P99: 99% of requests are faster. Captures tail latency issues.
•​ P99.9: Used for the most critical services. Even 0.1% of slow requests at scale affects
thousands of users.

FAANG Interview Tip

Always discuss latency in terms of percentiles, not averages. Averages hide tail latency spikes
that affect real users. Mention P95/P99 specifically.

1.6 Common Interview Questions

Interview Questions

Page 5
1. Design a system that serves API responses in under 50ms at P99. What architecture choices
would you make?
2. How would you reduce the latency of a service that currently takes 500ms to respond?
3. Explain the latency implications of synchronous vs asynchronous processing.
4. How does data replication strategy affect read latency vs write latency?
5. You have a microservice that calls three downstream services sequentially. How do you
reduce overall latency?
6. What is tail latency and why does it matter at scale? How do you mitigate it?
7. Design a low-latency notification system for a social media platform.

Page 6
2. High-Throughput Services

2.1 What is Throughput?

Throughput is the number of operations a system can handle per unit of time, commonly
measured in requests per second (RPS), transactions per second (TPS), or data volume per
second (MB/s, GB/s). While latency measures speed of individual requests, throughput
measures total capacity.

2.2 Throughput vs Latency: The Key Tradeoff

Latency and throughput are often in tension. Batching requests improves throughput but
increases latency for individual items. The goal in system design is to maximize throughput
while keeping latency within acceptable bounds.

Aspect Latency Throughput

Measures Time per request Requests per time unit
Unit ms, µs RPS, TPS, MB/s
Optimization Reduce hops, cache Batch, parallelize, scale out
Analogy Speed of one car Cars per hour on highway

2.3 Strategies for High Throughput

A. Horizontal Scaling
Add more machines to distribute load. This is the most fundamental approach for throughput at
FAANG scale. Stateless services are easiest to scale horizontally because any instance can
serve any request.
# Kubernetes HPA: Auto-scale based on CPU/RPS
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: api-server
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: api-server
minReplicas: 3
maxReplicas: 50

Page 7
 metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70

B. Batching
Instead of processing items one at a time, group them into batches. This amortizes per-request
overhead (network round trips, context switches, disk seeks) across many items.
// Java: Batch database inserts
PreparedStatement ps = [Link](
"INSERT INTO events(user_id, type, ts) VALUES(?,?,?)");

for (Event e : eventBatch) {

[Link](1, [Link]);
[Link](2, [Link]);
[Link](3, [Link]);
[Link]();
}
[Link](); // One round trip instead of N

C. Load Balancing
Distribute incoming requests evenly across servers. Common strategies include Round Robin,
Least Connections, Consistent Hashing (for cache-friendly routing), and Weighted distribution
(send more traffic to powerful machines). Layer 4 (TCP) load balancing is faster but less flexible;
Layer 7 (HTTP) allows content-based routing.

D. Asynchronous Processing with Message Queues

Decouple producers from consumers using message queues like Kafka, RabbitMQ, or SQS.
Producers push work into the queue without waiting for processing to complete. Consumers pull
and process at their own rate. This absorbs traffic spikes and allows independent scaling of
producers and consumers.
# Python + Kafka: High-throughput event producer
from kafka import KafkaProducer
import json

producer = KafkaProducer(
bootstrap_servers=['kafka1:9092', 'kafka2:9092'],
batch_size=65536, # 64KB batch

Page 8
 linger_ms=10, # Wait up to 10ms to fill batch
compression_type='lz4', # Compress batches
acks=1 # Leader ack only for throughput
)

for event in event_stream:

[Link]('user-events', [Link](event).encode())

E. Data Partitioning / Sharding

Split data across multiple database nodes so each handles only a fraction of total traffic.
Partition by user ID (hash-based), geography, or time range. Each shard handles its own
read/write load independently.

F. Compression
Compress data in transit and at rest. Use gzip or Brotli for HTTP responses, Snappy or LZ4 for
internal communication (fast compression, lower ratio), and columnar formats like Parquet for
analytical workloads (10x or more compression for repetitive data).

2.4 Back Pressure and Rate Limiting

When a system is overwhelmed, it must protect itself. Back pressure means slowing down
upstream producers when downstream consumers cannot keep up. Rate limiting caps the
number of requests per client per time window (e.g., 1000 RPS per API key). These prevent
cascading failures and ensure fair resource usage.
// Token Bucket Rate Limiter (conceptual)
class TokenBucket {
constructor(capacity, refillRate) {
[Link] = capacity;
[Link] = capacity;
[Link] = refillRate; // tokens per second
[Link] = [Link]();
}
tryConsume() {
[Link]();
if ([Link] >= 1) { [Link]--; return true; }
return false; // Rate limited
}
refill() {
const now = [Link]();
const elapsed = (now - [Link]) / 1000;
[Link] = [Link]([Link],
[Link] + elapsed * [Link]);

Page 9
 [Link] = now;
}
}

2.5 Common Interview Questions

Interview Questions
1. Design a system that handles 1 million requests per second. Walk through your architecture.
2. How would you design a log aggregation pipeline processing 10TB of logs per day?
3. Explain how Kafka achieves high throughput. What are partitions and consumer groups?
4. How do you handle a sudden 10x traffic spike without dropping requests?
5. Design a URL shortener that handles 100K reads/sec and 1K writes/sec.
6. What is back pressure and how do you implement it in a distributed system?
7. Compare horizontal vs vertical scaling. When would you choose each?

Page 10
3. Concurrent Programming

3.1 Concurrency vs Parallelism

Concurrency is about dealing with multiple things at once (structuring code to handle multiple
tasks), while parallelism is about doing multiple things at once (executing simultaneously on
multiple cores). Concurrency is a design pattern; parallelism is an execution model. You can
have concurrency without parallelism (single-core time-slicing) and parallelism without
concurrency (SIMD instructions).

Aspect Concurrency Parallelism

Definition Multiple tasks make progress Multiple tasks execute
simultaneously
Hardware Possible on single core Requires multiple cores/CPUs
Goal Structure and responsiveness Speed and throughput
Example Web server handling many clients Matrix multiplication on GPU

3.2 Concurrency Models

A. Shared Memory (Threads)

Multiple threads access the same memory space. This is fast but requires careful
synchronization to avoid race conditions, deadlocks, and data corruption. Java, C++, and C#
primarily use this model.
// Java: Thread-safe counter with synchronized
public class SafeCounter {
private int count = 0;

public synchronized void increment() {

count++; // Atomic within synchronized block
}

public synchronized int getCount() {

return count;
}
}

B. Message Passing (Actors)

Page 11
Instead of sharing memory, processes/actors communicate by sending immutable messages.
Each actor has its own state and processes messages sequentially. This eliminates
shared-state bugs by design. Erlang, Akka (Scala/Java), and Go channels use this model.
// Go: Channel-based message passing
func producer(ch chan<- int) {
for i := 0; i < 100; i++ {
ch <- i // Send message
}
close(ch)
}

func consumer(ch <-chan int, done chan<- bool) {

for val := range ch {
[Link]("Received:", val)
}
done <- true
}

C. Event Loop / Async-Await

A single thread runs an event loop that dispatches callbacks or coroutines when I/O completes.
Ideal for I/O-bound workloads with many concurrent connections. [Link], Python asyncio, and
JavaScript Promises use this model.
# Python: asyncio for concurrent I/O
import asyncio
import aiohttp

async def fetch_url(session, url):

async with [Link](url) as resp:
return await [Link]()

async def main():

urls = ['[Link] '[Link] '[Link]
async with [Link]() as session:
# All three requests run concurrently
results = await [Link](
*[fetch_url(session, u) for u in urls]
)
return results

3.3 Synchronization Primitives

Page 12
 Primitive Purpose Use Case
Mutex / Lock Mutual exclusion; only one Protecting shared counters,
thread at a time maps
Semaphore Allow N concurrent accesses Connection pool with max size
Read-Write Lock Multiple readers OR one writer Config that is read often,
updated rarely
Condition Variable Wait until a condition is true Producer-consumer queue
Atomic Variables Lock-free single-variable Counters, flags, CAS operations
operations
Barrier Wait until N threads reach a Phased parallel computation
point

3.4 Common Concurrency Problems

Race Conditions
Two threads access shared data simultaneously, and the result depends on timing. The classic
example is a check-then-act sequence: checking a condition and acting on it are not atomic, so
another thread can change the state between the check and the act.
// RACE CONDITION EXAMPLE (Java)
// Thread A and Thread B both execute this:
if (balance >= amount) { // CHECK
balance -= amount; // ACT
}
// If both threads check before either acts,
// balance can go negative!

// FIX: Make check-and-act atomic

synchronized(account) {
if (balance >= amount) {
balance -= amount;
}
}

Deadlocks
Two or more threads each hold a resource the other needs, and neither can proceed. The four
necessary conditions are: mutual exclusion, hold and wait, no preemption, and circular wait.
Break any one condition to prevent deadlocks.
// DEADLOCK EXAMPLE
// Thread 1: lock(A) -> lock(B)

Page 13
 // Thread 2: lock(B) -> lock(A)

// FIX: Always acquire locks in the same global order

// Thread 1: lock(A) -> lock(B)
// Thread 2: lock(A) -> lock(B) // Same order!

Starvation and Livelock

Starvation occurs when a thread never gets access to a resource because other threads always
take priority. Livelock occurs when threads keep changing state in response to each other but
make no progress, like two people in a hallway both stepping aside in the same direction
repeatedly.

3.5 Common Interview Questions

Interview Questions
1. What is the difference between concurrency and parallelism? Give real-world examples.
2. Implement a thread-safe bounded blocking queue.
3. How would you design a concurrent web crawler that respects rate limits?
4. Explain the dining philosophers problem and your solution.
5. What are the four conditions for deadlock? How do you prevent each?
6. Compare mutexes, semaphores, and monitors. When would you use each?
7. Design a concurrent LRU cache with O(1) operations.
8. How does Go's concurrency model differ from Java's? Which would you choose and why?

Page 14
4. Performance Profiling and Tuning

4.1 The Performance Engineering Cycle

Performance tuning follows a disciplined cycle: Measure (profile the system to find bottlenecks),
Analyze (identify the root cause of the bottleneck), Optimize (apply targeted fixes), and Verify
(confirm the improvement without regressions). Never optimize without profiling first. Premature
optimization based on guesswork often makes things worse.

Amdahl's Law
If 80% of execution time is in function X, even making X infinitely fast only gives a 5x overall
speedup. Always optimize the biggest bottleneck first.

4.2 Types of Profiling

A. CPU Profiling
Identifies which functions consume the most CPU time. Sampling profilers (like Linux perf,
async-profiler for Java) periodically record the call stack, creating flame graphs. Instrumentation
profilers (like gprof) add timing code to every function call but have higher overhead.
# Linux perf: CPU flame graph generation
perf record -F 99 -g -p <PID> -- sleep 30
perf script | [Link] | [Link] > [Link]

# Python: cProfile for CPU profiling

import cProfile
[Link]('process_data(large_dataset)', sort='cumulative')

B. Memory Profiling
Tracks heap allocations, identifies memory leaks, and finds objects that are not being garbage
collected. Tools include Valgrind (C/C++), JVisualVM / Eclipse MAT (Java), memory_profiler
(Python), and Chrome DevTools Heap Snapshots (JavaScript).
# Python: memory_profiler
from memory_profiler import profile

@profile
def process_data():
data = [i ** 2 for i in range(1_000_000)] # 8MB allocation
filtered = [x for x in data if x % 2 == 0] # Another 4MB
return sum(filtered)

Page 15
C. I/O Profiling
Identifies slow disk reads, network calls, and database queries. Tools include strace/ltrace
(system call tracing), tcpdump/Wireshark (network), and slow query logs (database). The most
common bottleneck in web services is I/O, not CPU.

D. Distributed Tracing
In microservices architectures, a single user request may traverse dozens of services.
Distributed tracing tools like Jaeger, Zipkin, and AWS X-Ray propagate trace IDs across service
boundaries, allowing you to see the full request path and identify which service is the bottleneck.

4.3 Key Optimization Techniques

Algorithm and Data Structure Optimization

The highest-impact optimizations come from choosing better algorithms. Replacing O(n²) with
O(n log n) sorting, using hash maps instead of linear search, or using a trie instead of repeated
string comparisons can deliver orders-of-magnitude improvements.

Database Query Optimization

•​ Add indexes on columns used in WHERE, JOIN, and ORDER BY clauses.
•​ Use EXPLAIN ANALYZE to understand query execution plans.
•​ Avoid N+1 query problems: use JOINs or batch loading instead of looping queries.
•​ Denormalize read-heavy tables to avoid expensive JOINs.
•​ Use connection pooling (PgBouncer, HikariCP) to avoid connection creation overhead.

-- PostgreSQL: EXPLAIN ANALYZE to find slow queries

EXPLAIN ANALYZE
SELECT [Link], COUNT([Link]) as order_count
FROM users u
JOIN orders o ON [Link] = o.user_id
WHERE o.created_at > '2025-01-01'
GROUP BY [Link]
ORDER BY order_count DESC
LIMIT 10;

-- Add index for the bottleneck

CREATE INDEX idx_orders_user_created ON orders(user_id, created_at);

Page 16
JVM Tuning (for Java/Scala/Kotlin)
•​ Heap sizing: Set -Xms and -Xmx to the same value to avoid resize pauses.
•​ GC selection: Use G1GC for balanced latency/throughput; ZGC or Shenandoah for
ultra-low pause times.
•​ Thread pool sizing: For CPU-bound tasks, use N threads (N = number of cores). For
I/O-bound, use N * (1 + wait_time/compute_time).

4.4 Common Interview Questions

Interview Questions
1. A production service's P99 latency spiked from 50ms to 2s. How do you diagnose and fix it?
2. Explain what a flame graph shows and how you would use it to optimize a service.
3. How do you identify and fix N+1 query problems?
4. What are the different garbage collection algorithms and their trade-offs?
5. How would you optimize a service that is CPU-bound vs one that is I/O-bound?
6. Explain Amdahl's Law and its implications for performance optimization.
7. How do you load test a system? What tools would you use and what metrics would you track?

Page 17
5. SQL vs NoSQL Databases

5.1 SQL (Relational) Databases

SQL databases store data in structured tables with predefined schemas. They enforce ACID
properties (Atomicity, Consistency, Isolation, Durability) and use SQL as the query language.
Examples include PostgreSQL, MySQL, Oracle, and SQL Server.

ACID Properties Explained

•​ Atomicity: A transaction either completes fully or not at all. If transferring money, both the
debit and credit happen, or neither does.
•​ Consistency: The database moves from one valid state to another. Constraints (foreign
keys, unique indexes) are always enforced.
•​ Isolation: Concurrent transactions do not interfere with each other. Different isolation
levels (Read Uncommitted, Read Committed, Repeatable Read, Serializable) trade
correctness for performance.
•​ Durability: Once a transaction commits, the data survives crashes (written to disk /
WAL).

-- SQL: Complex query with JOINs, aggregation, transactions

BEGIN TRANSACTION;

-- Transfer $100 from account A to account B

UPDATE accounts SET balance = balance - 100 WHERE id = 'A';
UPDATE accounts SET balance = balance + 100 WHERE id = 'B';

-- Verify no negative balance

SELECT balance FROM accounts WHERE id = 'A';
-- If balance < 0, ROLLBACK; else COMMIT;

COMMIT;

5.2 NoSQL Databases

NoSQL databases sacrifice some ACID guarantees for scalability, flexibility, and performance.
They come in several categories, each optimized for different access patterns.

Type Examples Best For

Key-Value Store Redis, DynamoDB, Riak Session data, caching, simple
lookups

Page 18
 Document Store MongoDB, Couchbase, Semi-structured data, CMS,
Firestore catalogs
Wide-Column Store Cassandra, HBase, ScyllaDB Time-series, IoT, write-heavy
workloads
Graph Database Neo4j, Amazon Neptune, Social networks,
ArangoDB recommendations, fraud
detection

The CAP Theorem

In a distributed system, you can only guarantee two of three properties: Consistency (every read
returns the most recent write), Availability (every request gets a response), and Partition
Tolerance (system works despite network failures). Since network partitions are unavoidable,
the real choice is between CP (consistent but may reject requests during partitions) and AP
(available but may return stale data during partitions).

Database CAP Choice Reasoning

PostgreSQL (single node) CA No partitions on single node
MongoDB (replica set) CP Rejects writes if primary is unreachable
Cassandra AP Returns data even during partitions
(eventual consistency)
DynamoDB AP (tunable) Configurable consistency per request

5.3 When to Choose SQL vs NoSQL

Choose SQL When Choose NoSQL When
Data has clear relationships (joins needed) Data is denormalized or document-shaped
ACID transactions are critical (banking, Eventual consistency is acceptable
inventory)
Schema is stable and well-defined Schema evolves rapidly
Complex queries with aggregations needed Simple key-based lookups dominate
Data volume fits on a few nodes Massive scale requires easy sharding
Regulatory compliance requires strong Availability is more important than consistency
consistency

5.4 Indexing Deep Dive

Page 19
Indexes are the single most important performance feature in any database. A B-Tree index
turns an O(n) table scan into O(log n) lookup. However, each index adds write overhead (the
index must be updated on every insert/update/delete) and storage cost.

Index Type Structure Best For

B-Tree (default) Balanced tree Range queries, sorting, equality
Hash Index Hash table Exact equality lookups only
GIN (Inverted) Posting lists Full-text search, JSONB, arrays
GiST Generalized search tree Geospatial, range types
Bitmap Bit arrays Low-cardinality columns in
analytics

-- Composite index: covers both WHERE and ORDER BY

CREATE INDEX idx_orders_composite
ON orders(user_id, created_at DESC, status);

-- Partial index: only index active users (saves space)

CREATE INDEX idx_active_users
ON users(email) WHERE is_active = true;

-- Covering index: includes all queried columns

CREATE INDEX idx_covering
ON products(category_id) INCLUDE (name, price);

5.5 Common Interview Questions

Interview Questions
1. You are designing a social media platform. Would you use SQL, NoSQL, or both? Justify your
choice.
2. Explain the CAP theorem with concrete examples. What does your system sacrifice?
3. How does database sharding work? What are the challenges of resharding?
4. Design the database schema for an e-commerce platform (users, products, orders, reviews).
5. What are the isolation levels in SQL? Explain phantom reads and how to prevent them.
6. When would you choose Cassandra over PostgreSQL? What about DynamoDB?
7. How do indexes work internally? When would adding an index actually hurt performance?
8. Explain eventual consistency. How do you handle it in application code?

Page 20
Page 21
6. Distributed Caching

6.1 Why Caching?

Caching stores copies of frequently accessed data in faster storage (typically RAM) to reduce
load on slower backends (databases, APIs, disk). A well-designed cache can reduce database
load by 80-90%, cut response times from hundreds of milliseconds to sub-millisecond, and allow
systems to handle orders of magnitude more traffic.

6.2 Cache Architectures

A. Local (In-Process) Cache

Data cached within the application process memory. Fastest access (no network hop) but
limited by process memory and not shared across instances. Examples: Guava Cache (Java),
lru_cache (Python), node-cache ([Link]). Best for configuration data, reference data, or data
that is expensive to compute but small.

B. Distributed Cache
Dedicated caching servers shared by all application instances. Redis and Memcached are the
dominant choices. Adds a network hop (~0.5ms) but provides shared state, larger capacity, and
persistence options.

Feature Redis Memcached

Data Structures Strings, Lists, Sets, Sorted Sets, Strings only
Hashes, Streams
Persistence RDB snapshots + AOF log None (pure cache)
Replication Master-replica with automatic None built-in
failover
Max Value Size 512 MB 1 MB
Multi-threading Single-threaded (I/O threads in Multi-threaded
6.0+)
Use Case Cache + data store + pub/sub + Simple, high-throughput caching
queues

6.3 Caching Strategies (Patterns)

Page 22
Cache-Aside (Lazy Loading)
The application checks the cache first. On cache miss, it fetches from the database, stores in
cache, then returns. This is the most common pattern. Pros: Only requested data is cached (no
wasted space). Cons: Cache miss penalty (three round trips: cache check, DB fetch, cache
write). Data can become stale.
# Python: Cache-Aside Pattern
def get_user(user_id):
# 1. Check cache
cached = [Link](f'user:{user_id}')
if cached:
return [Link](cached)

# 2. Cache miss - fetch from DB

user = [Link]('SELECT * FROM users WHERE id = %s', user_id)

# 3. Populate cache with TTL

[Link](f'user:{user_id}', 3600, [Link](user)) # 1hr TTL
return user

Write-Through
Every write goes to both the cache and database. Ensures cache is always consistent with the
database. Pros: Cache is never stale. Cons: Higher write latency (two writes per operation). May
cache data that is never read.

Write-Behind (Write-Back)
Writes go to cache first, then asynchronously to database in batches. Pros: Lowest write
latency, batching reduces DB load. Cons: Risk of data loss if cache fails before flush,
complexity. Used in systems where write performance is critical and some data loss is tolerable.

Read-Through
The cache itself is responsible for loading data from the database on a miss (vs the application
doing it in cache-aside). Simplifies application code but requires cache infrastructure to support
it.

6.4 Cache Invalidation

Cache invalidation is one of the two hardest problems in computer science (along with naming
things). When the underlying data changes, the cache must be updated or removed. Strategies
include TTL-based expiration (simplest but allows stale reads up to TTL), event-driven
invalidation (database triggers or CDC publish changes), and versioned keys (append version
number to cache key and increment on update).

Page 23
 Cache Stampede Prevention
When a popular cache key expires, hundreds of concurrent requests may all miss cache and hit
the database simultaneously. Solutions include: lock/mutex (only one thread fetches, others
wait), early/probabilistic refresh (refresh before expiration), and request coalescing (deduplicate
identical in-flight requests).

6.5 Consistent Hashing

When scaling a distributed cache from N to N+1 servers, simple modulo hashing (key % N)
remaps almost all keys. Consistent hashing maps both keys and servers onto a hash ring, so
adding or removing a server only remaps ~1/N of the keys. This is essential for distributed
caches like Memcached clusters and is used in DynamoDB, Cassandra, and CDNs.

6.6 Common Interview Questions

Interview Questions
1. Design a distributed cache for a social media feed. How do you handle cache invalidation?
2. Explain the differences between cache-aside, write-through, and write-behind patterns. When
would you use each?
3. How does consistent hashing work? What are virtual nodes and why are they needed?
4. How would you prevent a cache stampede on a popular key?
5. Redis vs Memcached: when would you choose each?
6. Design a multi-level caching architecture (L1 local, L2 distributed, L3 CDN).
7. How do you handle cache consistency in a microservices architecture where multiple services
modify the same data?
8. What is cache warming and when would you use it?

Page 24
7. Multi-Threading

7.1 Fundamentals
A thread is the smallest unit of execution within a process. Multiple threads within a process
share the same memory space (heap, global variables) but each has its own stack, program
counter, and registers. Multi-threading allows a program to perform multiple operations
concurrently, utilizing multiple CPU cores for parallel execution.

Concept Process Thread

Memory Separate address space Shared address space
Creation Cost Heavy (fork, copy page tables) Light (shared memory)
Communication IPC (pipes, sockets, shared Direct memory access
memory)
Isolation Full (crash does not affect others) None (crash kills all threads)
Context Switch Expensive (TLB flush) Cheaper (same address space)

7.2 Thread Lifecycle

A thread moves through these states: New (created but not started), Runnable (ready to run,
waiting for CPU), Running (actively executing on a CPU core), Blocked/Waiting (waiting for a
lock, I/O, or condition), and Terminated (execution completed). Understanding these states is
critical for debugging thread-related issues.

7.3 Thread Pools

Creating a new thread for every task is expensive (~1MB stack allocation, OS scheduling
overhead). Thread pools maintain a fixed set of reusable threads. Tasks are submitted to a
queue; worker threads pick tasks from the queue and execute them. This amortizes thread
creation cost and bounds resource usage.
// Java: ThreadPoolExecutor configuration
ExecutorService pool = new ThreadPoolExecutor(
10, // Core pool size
50, // Maximum pool size
60, [Link], // Keep-alive for idle threads
new LinkedBlockingQueue<>(1000), // Task queue capacity
new [Link]() // Rejection policy
);

// Submit tasks

Page 25
 Future<Result> future = [Link](() -> {
return processRequest(request);
});
Result result = [Link](5, [Link]); // With timeout

7.4 Lock-Free Programming

Lock-free data structures use atomic operations (Compare-And-Swap / CAS) instead of locks.
They guarantee system-wide progress: at least one thread makes progress even if others are
suspended. This avoids deadlocks and reduces contention but is much harder to implement
correctly.
// Java: AtomicInteger with CAS
AtomicInteger counter = new AtomicInteger(0);

// Lock-free increment
int oldVal, newVal;
do {
oldVal = [Link]();
newVal = oldVal + 1;
} while ();

// ConcurrentHashMap: lock-free reads, segment-locked writes

ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();
[Link]("key", k -> expensiveComputation(k));

7.5 The Java Memory Model

In multi-threaded programs, threads may see stale values because of CPU caches, compiler
reordering, and instruction pipelining. The Java Memory Model (JMM) defines when changes
made by one thread are guaranteed to be visible to other threads. The volatile keyword ensures
visibility (all threads see the latest value), and synchronized blocks ensure both visibility and
atomicity.

7.6 Thread Safety Patterns

•​ Immutable Objects: Objects that cannot be changed after creation are inherently
thread-safe. Use final fields, defensive copies, and unmodifiable collections.
•​ Thread-Local Storage: Each thread gets its own copy of a variable (ThreadLocal in
Java). Useful for per-request context, database connections, or date formatters.
•​ Copy-On-Write: Shared data is copied when modified. CopyOnWriteArrayList in Java is
ideal for read-heavy, write-rare scenarios.
•​ Confinement: Restrict data access to a single thread. If only one thread can access data,
no synchronization is needed.

Page 26
7.7 Common Interview Questions

Interview Questions
1. Implement a producer-consumer pattern using threads, locks, and condition variables.
2. What is a thread pool and how do you size it? What happens if the task queue overflows?
3. Explain the difference between volatile and synchronized in Java.
4. What is a ConcurrentHashMap and how does it achieve thread safety without locking the
whole map?
5. Implement a read-write lock from scratch.
6. What is the ForkJoinPool and when would you use it over a regular thread pool?
7. How would you debug a thread leak in a production Java application?
8. Explain false sharing and how it impacts multi-threaded performance on multi-core CPUs.

Page 27
8. Real-Time Programming

8.1 What is Real-Time Programming?

Real-time programming involves systems where correctness depends not only on the logical
result but also on the time at which the result is produced. A correct answer delivered too late is
as bad as a wrong answer. Real-time systems are classified by how strictly they enforce timing
requirements.

Type Deadline Violation Examples

Hard Real-Time Causes system failure or safety Aircraft flight control, anti-lock
hazard brakes, pacemakers
Firm Real-Time Result is useless but no Video frame rendering, radar
catastrophe tracking
Soft Real-Time Quality degrades but system still Video streaming, online gaming,
works VoIP calls

8.2 Real-Time in Web Systems (Soft Real-Time)

In the context of FAANG interviews, real-time usually means soft real-time web systems:
delivering updates to users with minimal delay. This includes live notifications, chat messages,
collaborative editing, live sports scores, stock tickers, and real-time dashboards.

WebSockets
WebSockets provide full-duplex communication over a single TCP connection. After an HTTP
handshake upgrades the connection, both client and server can send messages at any time.
This eliminates the overhead of repeated HTTP requests and enables true push-based
communication.
// [Link]: WebSocket server with [Link]
const io = require('[Link]')(httpServer, {
cors: { origin: '*' }
});

[Link]('connection', (socket) => {

[Link]('Client connected:', [Link]);

// Join a room (e.g., chat room, game lobby)

[Link]('join-room', (roomId) => {
[Link](roomId);
});

Page 28
 // Broadcast message to room
[Link]('message', (data) => {
[Link]([Link]).emit('message', {
sender: [Link],
text: [Link],
timestamp: [Link]()
});
});
});

Server-Sent Events (SSE)

SSE is a simpler, one-directional protocol where the server pushes updates to the client over a
long-lived HTTP connection. The client uses the EventSource API. SSE is simpler than
WebSockets, works with standard HTTP infrastructure (proxies, load balancers), and supports
automatic reconnection. Use SSE when you only need server-to-client updates (notifications,
feeds, dashboards).
// [Link]: Server-Sent Events endpoint
[Link]('/events', (req, res) => {
[Link]('Content-Type', 'text/event-stream');
[Link]('Cache-Control', 'no-cache');
[Link]('Connection', 'keep-alive');

const sendEvent = (data) => {

[Link](`data: ${[Link](data)}\n\n`);
};

// Subscribe to updates
const unsubscribe = [Link](sendEvent);
[Link]('close', () => unsubscribe());
});

Long Polling
The client sends an HTTP request and the server holds it open until new data is available (or a
timeout). Once the response is sent, the client immediately sends a new request. This is the
simplest approach and works through all proxies/firewalls but has higher overhead than
WebSockets due to repeated connection setup.

8.3 Real-Time Architecture Patterns

Page 29
Pub/Sub for Real-Time Fanout
When a user posts a message, it must be delivered to potentially millions of followers. Redis
Pub/Sub, Kafka, or a dedicated message broker fans out the message to all connected
WebSocket servers, which then push to their connected clients. This decouples message
production from delivery.
# Architecture: Real-time notification fanout
#
# User posts message
# -> API Server publishes to Kafka topic 'notifications'
# -> WebSocket Server 1 (consumer) pushes to its connected clients
# -> WebSocket Server 2 (consumer) pushes to its connected clients
# -> WebSocket Server N (consumer) pushes to its connected clients

Event Sourcing + CQRS

Instead of storing current state, store a sequence of events. The read model (query side)
subscribes to events and maintains a materialized view optimized for reads. This naturally
supports real-time updates because any subscriber can react to events as they happen.

8.4 Challenges in Real-Time Systems

•​ Connection Management: Each WebSocket consumes a file descriptor and memory. A
single server can typically handle 50K-100K concurrent connections. Use connection
limits and heartbeats to clean up dead connections.
•​ Ordering Guarantees: Messages may arrive out of order, especially across partitions or
servers. Use sequence numbers or vector clocks to detect and reorder.
•​ Exactly-Once Delivery: Achieving exactly-once is nearly impossible in distributed
systems. Most real-time systems use at-least-once delivery with client-side deduplication
(idempotency keys).
•​ Presence Detection: Detecting whether a user is online requires heartbeats with
timeouts. Set aggressive timeouts (30-60s) to avoid showing stale presence.

8.5 Hard Real-Time Concepts

While less common in FAANG interviews, understanding hard real-time systems shows depth.
These systems use Real-Time Operating Systems (RTOS like VxWorks, FreeRTOS) that
guarantee deterministic scheduling. Key concepts include worst-case execution time (WCET)
analysis, priority inversion (when a high-priority task waits for a low-priority task holding a lock),
and rate-monotonic scheduling (assigning higher priority to tasks with shorter periods).

8.6 Common Interview Questions

Page 30
Interview Questions
1. Design a real-time chat application supporting 10 million concurrent users.
2. Compare WebSockets, SSE, and Long Polling. When would you use each?
3. How would you design a real-time collaborative document editor like Google Docs?
4. Design a live sports scoreboard that updates all connected clients within 1 second.
5. How do you handle message ordering in a distributed real-time system?
6. Design a real-time notification system for a social media platform.
7. What is the Operational Transformation (OT) algorithm? How does it enable real-time
collaboration?
8. How would you scale WebSocket servers horizontally?

Page 31
9. Reference Materials & Resources

9.1 Recommended YouTube Channels & Videos

1. System Design by Gaurav Sen

Gaurav Sen's YouTube channel covers all major system design topics with clear diagrams and
explanations. His videos on consistent hashing, caching strategies, and database sharding are
particularly relevant for the topics in this guide.
Channel: [Link]/@gaborSen

2. ByteByteGo by Alex Xu
Alex Xu (author of the System Design Interview books) runs the ByteByteGo channel with
animated, visual explanations of system design concepts. Excellent for understanding
distributed caching, CAP theorem, and high-throughput architectures.
Channel: [Link]/@ByteByteGo

3. Hussein Nasser - Backend Engineering

Hussein Nasser provides deep-dive videos on networking, databases, concurrency, and
real-time communication protocols. His videos on WebSockets vs SSE, database indexing
internals, and connection pooling are directly relevant.
Channel: [Link]/@haborNasser

9.2 Recommended Blogs & Reading

1. Martin Kleppmann's Blog & Book

'Designing Data-Intensive Applications' (DDIA) is the gold standard reference for distributed
systems, databases, and data processing. Covers consistency models, replication, partitioning,
and stream processing in depth.

2. The Morning Paper (Adrian Colyer)

Summarizes key computer science papers. Excellent for understanding the theoretical
foundations behind distributed caching, consensus algorithms, and concurrent data structures.

3. High Scalability Blog

Real-world architecture case studies from companies like Netflix, Instagram, Twitter, and Uber.
Shows how the concepts in this guide are applied at massive scale in production systems.
URL: [Link]

Page 32
9.3 Practice Platforms
•​ LeetCode (Concurrency section): Practice multi-threading problems like Print in Order,
Building H2O, Dining Philosophers.
•​ [Link] - Grokking the System Design Interview: Structured course covering all
major system design patterns.
•​ System Design Primer (GitHub): Free, comprehensive guide with diagrams and code
examples. [Link]/donnemartin/system-design-primer

9.4 Quick Reference: Topics to System Design Mapping

Topic Key System Design Applications
Low Latency CDN design, edge computing, search autocomplete, ad serving
High Throughput Log aggregation, analytics pipelines, feed generation
Concurrency Web servers, database connection pools, task schedulers
Profiling & Tuning SRE interviews, production debugging, capacity planning
SQL/NoSQL Any data-heavy design: e-commerce, social media, analytics
Distributed Caching Feed ranking, session management, rate limiting
Multi-Threading Web servers, parallel data processing, game servers
Real-Time Chat, notifications, collaborative editing, live dashboards

Good luck with your interviews!

Page 33