Distributed Systems
Table of Contents
• Overview
• Characteristics and Challenges
• Communication and Networking
• Time, Ordering, and Clocks
• Consistency Models
• Replication and Fault Tolerance
• Consensus Algorithms
• Distributed Transactions and Coordination
• Scalability Patterns: Sharding, Caching, Load Balancing
• Monitoring, Observability, and Debugging
• Security in Distributed Systems
• Case Studies and Architectures
• Exercises and Further Reading
Overview
Distributed systems coordinate multiple independent machines to provide services
with improved scalability, reliability, and performance. They must handle partial
failures, network unreliability, and concurrency.
Characteristics and Challenges
• Partial failures: some components may fail while others continue.
• Concurrency: multiple actors operate simultaneously.
• Latency and bandwidth constraints.
• Heterogeneity: different hardware, OS, and software stacks.
Communication and Networking
• RPC and RESTful APIs for inter-service communication.
• Serialization formats: JSON, Protocol Buffers, Thrift, Avro.
• Message-oriented middleware and messaging patterns: pub/sub, queues.
Time, Ordering, and Clocks
• Physical clocks: NTP for synchronization, but cannot be perfect.
• Logical clocks: Lamport timestamps and vector clocks for ordering events.
• Causality and happens-before relation.
Consistency Models
• Strong consistency: linearizability and sequential consistency.
• Eventual consistency: replicas converge over time.
1
• Causal consistency and read-your-writes guarantees.
Replication and Fault Tolerance
• Replication strategies: leader-based, leaderless, multi-leader.
• Quorum systems for reads/writes (e.g., R + W > N).
• Failure detectors and health checking.
Consensus Algorithms
• Problem: agree on a value across unreliable nodes.
• Paxos: classic consensus algorithm (conceptual complexity).
• Raft: consensus algorithm designed for understandability; leader election,
log replication.
Distributed Transactions and Coordination
• Two-Phase Commit (2PC): blocking commit protocol.
• Three-Phase Commit (3PC): attempts to avoid blocking.
• Saga pattern: long-running transactions implemented as sequences of
compensating actions.
• Coordination services: ZooKeeper, etcd for leader election and configura-
tion.
Scalability Patterns: Sharding, Caching, Load Balancing
• Partitioning data across nodes by key.
• Caching layers (CDNs, in-memory caches like Redis) to reduce latency.
• Load balancers and service discovery systems.
Monitoring, Observability, and Debugging
• Metrics, logs, distributed tracing (e.g., OpenTelemetry, Jaeger).
• Health checks and automated remediation.
• Challenges in reproducing distributed failures and techniques for debugging
(record-and-replay).
Security in Distributed Systems
• Mutual TLS, service-to-service authentication and authorization.
• Secure key distribution and rotation in multi-node environments.
• Least privilege and network segmentation.
Case Studies and Architectures
• CAP theorem trade-offs in real systems.
2
• Storage systems: Spanner, Cassandra, Dynamo, Bigtable—design choices
and trade-offs.
• Microservices vs monoliths: operational considerations.
Exercises and Further Reading
1. Implement a simple leader election using a shared coordination service
(e.g., etcd).
2. Build a replicated log using a simplified Raft algorithm and demonstrate
leader changes.
3. Design a sharding strategy for a social feed and analyze hotspot mitigation
techniques.
Further reading: - Tanenbaum & van Steen: Distributed Systems. - Chandy and
Lamport: Distributed snapshots. - Papers on Paxos, Raft, Dynamo, Spanner.