Understanding Distributed

Systems PDF
Roberto Vitillo
Understanding Distributed Systems
Master the essentials of building scalable distributed
systems efficiently.
Written by Bookey
Check more about Understanding Distributed Systems
Summary
Listen Understanding Distributed Systems Audiobook
About the book
Understanding Distributed Systems by Roberto Vitillo serves
as an essential guide for anyone looking to master the
complexities of large-scale distributed system development.
While a wealth of resources exists—ranging from academic
studies to engineering blogs—the information is often
fragmented, leaving aspiring developers struggling to see how
it all fits together. This book bridges the gap between theory
and practice, distilling key concepts drawn from extensive
real-world experience in constructing systems capable of
handling millions of requests per second and billions of
devices. Perfect for back-end developers or those eager to
delve into this field, this comprehensive resource covers vital
topics such as network architecture, data consistency models,
scalability, and reliability patterns. By equipping readers with
foundational knowledge, Vitillo aims to help them avoid the
common pitfalls of debugging and redesigning, enabling a
smoother path to success in building robust distributed
systems.
About the author
Roberto Vitillo is a distinguished computer scientist and
educator known for his extensive expertise in distributed
systems and software architecture. With a robust academic
background and years of industry experience, Vitillo has
contributed significantly to advancing the field through
research, teaching, and practical applications. His work often
emphasizes the design and implementation of scalable, reliable
systems that meet the demands of modern computing
environments. As a passionate advocate for the dissemination
of knowledge, he shares his insights not only through
scholarly publications but also in accessible formats, helping
both students and professionals navigate the complexities of
distributed technologies. His latest work, "Understanding
Distributed Systems," reflects his commitment to demystifying
the intricacies of these systems for a broad audience.
Summary Content List
Chapter 1 : Understanding Distributed Systems

Chapter 2 : 1 Introduction

Chapter 3 : 2 Reliable links

Chapter 4 : 3 Secure links

Chapter 5 : 4 Discovery

Chapter 6 : 5 APIs

Chapter 7 : 6 System models

Chapter 8 : 7 Failure detection

Chapter 9 : 8 Time

Chapter 10 : 9 Leader election

Chapter 11 : 10 Replication

Chapter 12 : 11 Transactions

Chapter 13 : 12 Functional decomposition

Chapter 14 : 13 Partitioning

Chapter 15 : 14 Duplication
Chapter 16 : 15 Common failure causes

Chapter 17 : 16 Downstream resiliency

Chapter 18 : 17 Upstream resiliency

Chapter 19 : 18 Testing

Chapter 20 : 19 Continuous delivery and deployment

Chapter 21 : 20 Monitoring

Chapter 22 : 21 Observability

Chapter 23 : 22 Final words

 Chapter 1 Summary : Understanding
Distributed Systems

Section Content

Preface
The demand for distributed systems expertise is increasing, as seen in the Stack Overflow 2020 developer
survey. Modern applications are distributed, making learning challenging due to scattered resources. The author
shares personal experiences to provide an accessible introduction, with regular updates encouraged through reader
feedback.

Who
should Targeted at back-end web or mobile developers interested in building distributed systems. Topics include
read network stack, data consistency models, and scalability. It aims to reduce debugging time and address knowledge
this gaps for both new and seasoned engineers, as well as serve as a resource for system design interviews in large
book companies.

Understanding Distributed Systems

Preface
The demand for expertise in distributed systems is
highlighted by the Stack Overflow’s 2020 developer survey,
indicating that top-paying engineering roles require such
skills. Modern applications are fundamentally distributed
systems, making the learning process challenging,
particularly at scale. While a wealth of resources exists, they
tend to be scattered across a spectrum of theory and
practice—with little available in the middle ground for
practical application. The author shares their own struggles in
finding a comprehensive resource and aims to provide an
accessible introduction to cover the fundamentals of
distributed systems, based on personal experience in building
large-scale systems.
Regular updates will be made to the book, and the author
encourages reader feedback for continual improvement.

Who should read this book

This book is targeted at developers involved in back-end web

or mobile applications, especially those interested in building
distributed systems. Key topics include understanding the
network stack, data consistency models, and scalability and
reliability patterns. Familiarity with these concepts can
reduce debugging time and architectural redesigns. Even
seasoned engineers will find knowledge gaps addressed,
improving their skills as practitioners and system architects.
Furthermore, this book serves as a useful resource for
preparing for system design interviews with large companies
that manage extensive distributed systems, where proficiency
in designing complex networked services is crucial for
success in senior roles.
Example
Key Point:Importance of practical knowledge in
distributed systems
Example:Imagine you're developing a mobile
application that needs to handle thousands of users
simultaneously and you start to see performance issues.
This is where understanding distributed systems
becomes crucial; by applying the concepts of scalability
and reliability, you can design a system that efficiently
manages user requests, ensuring smooth operation even
under stress. This knowledge not only enhances the
application’s performance but also reduces
troubleshooting time, making it an invaluable asset for
your development skills.
 Chapter 2 Summary : 1 Introduction

Section Description

1 Introduction A distributed system is a network of cooperating nodes (physical machines or software processes) that
collaborate to perform tasks. Key motivations include inherent distribution, high availability, large
workloads, and performance requirements.

1.1 Nodes must communicate over networks, facing challenges like message fidelity, network outages, and
Communication security. Understanding communication is vital due to potential leaks in abstractions.

1.2 Coordinating nodes is complex, especially during failures, illustrated by the "two generals" problem.
Coordination Effective coordination algorithms will be detailed in a later section.

1.3 Scalability Performance is gauged by load handling (throughput and response time). Scalability can be achieved by
upgrading hardware (scaling up) or adding machines (scaling out), each with specific architectural patterns.

1.4 Resiliency Resilience allows a distributed system to function during failures; strategies include redundancy and
self-healing. Availability is key, with higher percentages indicating better performance despite failures.

1.5 Operations Testing, deployment, and maintenance of distributed systems are integrated practices, especially with
microservices and DevOps. Continuous deployment with minimal disruption is emphasized.

1.6 Anatomy of The architecture of distributed systems varies based on perspectives, including physical machines and
a Distributed software processes. Each service provides capabilities via interfaces, and the server-client dynamic allows
System processes to play both roles.

1 Introduction

A distributed system is a network of cooperating nodes that

exchange messages to accomplish tasks. These nodes can be
either physical machines or software processes. Key
motivations for building distributed systems include:
-
Inherent Distribution:
Some applications, like the web, are inherently distributed
across multiple devices.
-
High Availability:
Systems like Dropbox replicate data across nodes to prevent
data loss from single-node failures.
-
Large Workloads:
Applications like Google need distributed systems to handle
massive incoming requests.
-
Performance Requirements:
Services like Netflix depend on proximity of data centers for
efficient streaming.

1.1 Communication

Nodes in a distributed system must communicate over

networks, posing challenges such as ensuring message
fidelity during transmission, handling network outages, and
maintaining security against eavesdropping. Understanding
communication intricacies is crucial as abstractions may
leak.

1.2 Coordination

Coordinating nodes to function as a coherent system is

complex, especially during failures. The "two generals"
problem exemplifies these challenges, where uncertainty
arises from potential communication failures. Effective
coordination algorithms will be explored in a dedicated
section of the book.

1.3 Scalability

A distributed system's performance is defined by its

efficiency in handling load, measured by throughput and
response time. As load increases, systems will eventually
reach their capacity, causing performance to decline.
Scalability can be achieved by upgrading hardware (scaling
up) or by adding more machines (scaling out), each with its
own architectural patterns discussed in a later section.
1.4 Resiliency

Resilience enables a distributed system to maintain

functionality amidst failures. Every component's failure
probability increases with scale, necessitating strategies like
redundancy and self-healing mechanisms. The concept of
availability—expressed in percentages—is critical: higher
availability indicates better performance despite failures.

1.5 Operations

Testing, deploying, and maintaining distributed systems have

become integrated responsibilities, especially with the rise of
microservices and DevOps. Continuous deployment with
minimal disruption is essential. The final section of the book
focuses on operational best practices.

1.6 Anatomy of a Distributed System

Distributed systems vary in architecture based on

perspectives—physical machines, software processes, and
loosely-coupled services. Each service provides specific
capabilities via inbound and outbound interfaces, requiring
adapters for communication. The server-client dynamic is
defined, illustrating that a process can play both roles. The
discussion incorporates various architectural perspectives
throughout the book, adaptable to relevant topics.
Chapter 3 Summary : 2 Reliable links

Reliable Links in Distributed Systems

1. Introduction to TCP

TCP (Transmission Control Protocol) is a transport-layer

protocol that provides a reliable communication channel
between processes over IP. It ensures ordered delivery of byte
streams without gaps, duplication, or corruption, while
implementing stability patterns to manage network load.

2. Reliability

TCP creates a reliable channel by dividing the byte stream

into numbered segments. Each segment requires
acknowledgment from the receiver. If an acknowledgment is
not received, the segment is retransmitted. A checksum is
used by the receiver to ensure segment integrity.

3. Connection Lifecycle
Before data transmission, a TCP connection must be
established using a three-way handshake:
1. The sender initiates with a SYN segment.
2. The receiver responds with a SYN/ACK segment,
choosing a sequence number.
3. The sender acknowledges with an ACK and possible
application data.
The connection transitions through states: opening,
established, and closing, with resource release needed after
data transmission.

4. Flow Control

Flow control prevents the sender from overwhelming the

receiver, which manages a receive buffer for incoming
segments. The receiver communicates buffer size to the
sender during acknowledgment, ensuring that the sender does
not exceed buffer capacity.

5. Congestion Control

TCPInstall Bookey
prevents networkApp to Unlock
flooding Fullavailable
by estimating Text and
bandwidth and maintainingAudio
a congestion window which
limits unacknowledged segments. The window size increases
Chapter 4 Summary : 3 Secure links

Secure Links

We now know how to reliably send bytes from one process to

another over the network. The problem is these bytes are sent
in the clear, and any middle-man can intercept our
communication. To protect against that, we can use the
Transport Layer Security (TLS) protocol. TLS runs on top of
TCP and encrypts the communication channel so that
application layer protocols, like HTTP, can leverage it to
communicate securely. In a nutshell, TLS provides
encryption, authentication, and integrity.

Encryption

Encryption guarantees that the data transmitted between a

client and a server is obfuscated and can only be read by the
communicating processes. When a TLS connection is first
opened, the client and the server negotiate a shared
encryption secret using asymmetric encryption for the
key-pair generation. Once the shared secret is established,
symmetric encryption is used for efficiency. The shared key
is regularly renegotiated to minimize the impact of potential
compromises and protect in-flight data.

Authentication

To ensure the authenticity of the communicating parties, TLS

implements authentication using digital signatures based on
asymmetric cryptography. The server shares its public key in
a digital certificate, which is verified by the client. This is
reinforced by a certificate authority (CA) that issues
certificates, creating a chain of trust. Clients must trust either
the server's certificate or a root CA certificate stored locally.
Automation to monitor and renew certificates is crucial for
maintaining service connectivity.

Integrity

To guard against data tampering, TLS verifies message

integrity using a HMAC (Hash-based Message
Authentication Code). This process ensures that if a received
message's digest does not match the expected digest, it is
either corrupted or tampered with, and such messages are
discarded. This layer of protection is essential, despite TCP’s
checksum, which is not fail-proof.
Handshake

When establishing a new TLS connection, a handshake

occurs between the client and server, which includes the
following steps:
1. Agreement on the cipher suite, specifying algorithms for
key exchange, signing, encryption, and HMAC.
2. Generation of a shared secret using the negotiated key
exchange algorithm.
3. Verification of the server’s certificate by the client to
confirm the server's identity.
In modern implementations, optimizations can reduce
handshake round trips, with TLS 1.2 typically requiring two,
and TLS 1.3 requiring just one. Due to the expense of
creating new connections, minimizing round trips and
reusing connections are recommended strategies.
Chapter 5 Summary : 4 Discovery

Discovery

In this chapter, the process of discovering remote process IP

addresses is explored, focusing on the Domain Name System
(DNS), which serves as the Internet's phone book.

DNS Resolution Process

When a user enters a URL (e.g., [Link]) in a

browser, the following steps occur to resolve the hostname to
an IP address:
1. The browser first checks its local cache for the hostname.
If found, it uses the cached IP address. If not, it sends a
request to a DNS resolver, usually managed by the Internet
Service Provider (ISP).
2. The DNS resolver checks its cache. If there is no entry, it
queries a root name server.
3. The root name server maps the request's top-level domain
(TLD) to the relevant TLD name server's address.
4. The resolver then queries the TLD name server for the
specific domain (e.g., .com).
5. The TLD name server directs the resolver to the
authoritative name server for the domain, which holds the
records mapping hostnames to IP addresses.
6. Finally, the resolver asks the authoritative name server for
the IP address of [Link], which is returned. If a
subdomain was queried, the authoritative server would return
the details of the subdomain's name server instead.

Optimization through Caching

To minimize the overhead of multiple server lookups, DNS

leverages caching, as hostname-to-IP mappings rarely
change. Different entities (browser, operating system, DNS
resolver) all maintain caches, which have an associated time
to live (TTL) indicating how long a record is valid.

TTL Considerations

Setting TTL involves trade-offs; a long TTL can result in

outdated entries being accessed, while a short TTL increases
server load and response times. A balance is necessary to
ensure clients can connect reliably and quickly, without
overwhelming name servers.
Risk of DNS as a Single Point of Failure

The DNS infrastructure can present a vulnerability, as a

failure of the DNS name server can prevent clients from
reaching services, potentially leading to large-scale outages.
Example
Key Point:The critical role of DNS in discovering
remote services reliably and efficiently.
Example:Imagine you’re trying to access your favorite
online store but instead of simply typing the name, you
reflect on how that name is transformed into a
functional connection. As you hit enter, your browser
doesn’t just stumble into the digital unknown; it
meticulously checks its memory for the store’s address.
If it can’t remember, it invokes your ISP’s DNS
resolver, like a knowledgeable librarian fetching books
from various shelves. This intricate dance ensures
you’re connected swiftly to the online shop you love,
highlighting how DNS resolves domain names to IP
addresses, making your browsing experience faster and
more reliable.
 Chapter 6 Summary : 5 APIs
Section Summary

5 APIs APIs act as adapters for service operations, facilitating direct request-response communication. They use various
serialization formats and IPC technologies like gRPC for internal APIs and REST for external ones.

5.1 HTTP HTTP is a stateless request-response protocol where clients send requests and receive responses. It has evolved
from HTTP 1.1 (sequential handling) to HTTP 2 (binary protocol with multiplexing) and is gradually moving to
HTTP 3 (UDP-based).

5.2 Resources in an e-commerce catalog service, such as products, are identified via URLs. The design follows
Resources REST principles for simplicity and efficient caching.

5.3 HTTP methods (POST, GET, PUT, DELETE) enable CRUD operations on resources. Safety and idempotency of
Request methods are important for request retrying.
methods

5.4 Status codes in HTTP responses indicate request outcomes: 200-299 success, 300-399 redirection, 400-499 client
Response errors, and 500-599 server errors.
status
codes

5.5 The OpenAPI specification formalizes RESTful API definitions, aiding in documentation and client SDK
OpenAPI generation based on endpoint and response schemas.

5.6 APIs must evolve without breaking compatibility, using versioning strategies like URL prefixes or custom
Evolution headers and ensuring backward compatibility to minimize disruption.

5 APIs

A service exposes its operations to consumers through APIs,

which act as adapters translating messages from IPC
mechanisms to service interface calls. Communication can be
direct or indirect; this chapter focuses on direct
request-response communication. Messages are serialized in
various formats that impact performance and readability,
leading to synchronous or asynchronous communication
styles. Common IPC technologies include gRPC for internal
APIs and REST for external ones. The chapter details
creating a RESTful HTTP API.

5.1 HTTP

HTTP is a request-response protocol where clients send

requests to server endpoints and receive responses. It is
stateless, requiring all information for processing to be
included in each request. HTTP 1.1 establishes a connection
to the server to handle requests sequentially, which can be
inefficient. HTTP 2 improves throughput with a binary
protocol allowing multiplexing while HTTP 3, based on
UDP, is in gradual rollout.

5.2 Resources

In building an e-commerce catalog service, resources like

products are identified via URLs. An HTTP server hosts
resources, with the product catalog accessed via a specific
URL format. REST principles guide API design to keep
Install Bookey
relationships and URLs App to Unlock
simple, promotingFull Textcaching.
efficient and
Audio
5.3 Request methods
Chapter 7 Summary : 6 System models

System Models in Distributed Systems

To understand distributed systems, it's essential to define the

possible and impossible behaviors within them. A system
model encapsulates the assumptions regarding nodes,
communication links, and timing, serving as a framework to
analyze distributed systems without delving into the
intricacies of actual technologies.

Communication Link Models

1.
Fair-Loss Link Model
: Allows for message loss and duplication. Messages
eventually reach their destination if the sender keeps
retransmitting.
2.
Reliable Link Model
: Guarantees that messages are delivered exactly once
without loss or duplication. This can be achieved by
de-duplicating messages at the receiver end.
3.
Authenticated Reliable Link Model
: Similar to the reliable link model, but the receiver can
authenticate the message sender.
These models, though abstract, aid in verifying algorithm
correctness. Reliable and authenticated communication links
can be built on fair-loss models, exemplified by TCP and
TLS protocols.

Node Failure Models

1.
Arbitrary-Fault Model
: Accounts for nodes that may behave unpredictably, leading
to crashes or abnormal behavior, often referenced as the
"Byzantine" model. It can be proven that systems can tolerate
a certain number of Byzantine nodes.
2.
Crash-Recovery Model
: Nodes do not deviate from their logic but can crash and
restart, losing in-memory state.
3.
Crash-Stop Model
: Similar to crash-recovery, but once a node crashes, it never
returns.
Unlike communication links, it’s not feasible to convert
unreliable node behavior into reliable performance, leading
to distinctly different algorithms for various node models.
Byzantine models are particularly relevant in safety-critical
scenarios.

Timing Assumptions

1.
Synchronous Model
: Assumes that message sending or operation execution is
bound by a time limit - an unrealistic assumption in practice.
2.
Asynchronous Model
: Assumes that operations can take an unbounded amount of
time, leading to potential algorithmic stasis.
3.
Partially Synchronous Model
: A practical compromise where the system operates
synchronously most of the time but can occasionally revert to
asynchronous behavior.
The book primarily adopts a system model characterized by
fair-loss links, crash-recovery node behavior, and partial
synchrony, diverging from Byzantine considerations which
apply in specific high-stakes contexts. It emphasizes the
importance of questioning the assumptions associated with
these models and recognizing their limitations as abstractions
of reality. For further exploration, readers may refer to
“Introduction to Reliable and Secure Distributed
Programming” for insights into other system models.
Critical Thinking
Key Point:The abstraction of system models may
limit real-world applicability.
Critical Interpretation:Vitillo underscores that while
system models like fair-loss or crash-recovery provide a
structured way to analyze distributed systems, they
inherently abstract complexities that can arise in
practical applications. These models might lead
engineers to overlook critical factors, potentially
resulting in inadequacies when these systems encounter
unforeseen real-world conditions. Particularly in
scenarios demanding high reliability, such as financial
systems or healthcare applications, relying solely on
these models could yield shortcomings that affect
operational viability. It's vital for practitioners to remain
skeptical of these simplifications and consider
additional literature, such as 'Introduction to Reliable
and Secure Distributed Programming,' which expands
on the diverse implications of system design and
behavior.
Chapter 8 Summary : 7 Failure detection

Failure Detection

Introduction to Failure Detection

In distributed systems, when a client sends a request to a

server, several outcomes are possible. Ideally, the client
receives a response. However, if no response is received, it
becomes difficult to determine if the server is slow, crashed,
or if there is a network issue.

Timeout Configuration

To address the uncertainty, clients can configure a timeout. If

the server does not respond within a specified time, the client
assumes the server is unavailable and throws an error.
However, setting the appropriate timeout period is
challenging; too short a timeout can lead to false positives
(assuming the server is dead), while too long a timeout can
keep the client waiting indefinitely.
Limits of Failure Detection

Perfect failure detection is unattainable. A client can

proactively monitor the availability of a server by sending
pings or heartbeats.

Ping Mechanism

A ping is a periodic request sent by one process to another to

check its availability. If the server fails to respond within a
set time, it is marked as dead, but the client continues to send
pings. If the server comes back online, it will respond and be
marked as available again.

Heartbeat Mechanism

A heartbeat is a periodic message sent by one process to

another, indicating it is still operational. If a heartbeat is not
received within the designated timeframe, the recipient
considers the sender as dead. Once the sender resumes
sending heartbeats, it will be marked as available again.

Use Cases for Pings and Heartbeats

Pings and heartbeats are particularly useful in scenarios
where processes frequently interact, and quick actions are
necessary when a process becomes unreachable. In less
intensive communication scenarios, failure detection at the
time of communication may suffice.
Critical Thinking
Key Point:The challenge of timeout configuration in
failure detection systems.
Critical Interpretation:Vitillo highlights the difficulty of
configuring the appropriate timeout for failure detection
in distributed systems, noting that both overly optimistic
and overly pessimistic settings can lead to significant
issues in system reliability. While the author's
perspective sheds light on a key technical challenge, it is
important to consider alternative theories and solutions.
For instance, research by Birman and van Renesse
suggests that adaptive timeout mechanisms could offer
more robust real-time communication in distributed
systems, thus questioning the absoluteness of Vitillo’s
conclusions. Readers should weigh these insights
carefully to gain a more nuanced understanding of
failure detection mechanisms.
Chapter 9 Summary : 8 Time

Chapter 9: Time in Distributed Systems

Introduction to Time in Distributed Systems

Time is a crucial element in applications, particularly in

distributed systems where there is no shared global clock.
Understanding the order of operations is vital, especially
since processes can run concurrently. In this chapter, we
explore different clocks that help establish the order of
operations.

8.1 Physical Clocks

Physical clocks, such as quartz clocks, are commonly used,

but they are prone to inaccuracies, known as clock drift and
skew. Synchronization with more accurate clocks, like
atomic clocks, is necessary, typically achieved through the
Network Time Protocol (NTP). However, NTP can introduce
errors due to network latency, complicating the measurement
of elapsed time. To mitigate these issues, monotonic clocks
are introduced, which measure time from a fixed point and
only move forward, although they cannot synchronize across
nodes.

8.2 Logical Clocks

Logical clocks measure the passage of time in terms of

operations rather than physical time. The simplest form is a
counter that increments before an operation to ensure distinct
logical timestamps. While it accurately represents the order
of operations within a single process, establishing order
across multiple processes requires a more sophisticated
approach, leading to the development of Lamport clocks.
Lamport clocks allow the tracking of causal relationships but
do not guarantee the converse; therefore, they can be
misleading.

8.3 Vector Clocks

Vector clocks enhance the capabilities of logical clocks by

allowing a partial ordering of operations. Each process
InstallanBookey
maintains App to ensuring
array of counters, Unlockaccurate
Full Text andof
tracking
causality. When processes Audio
communicate, they update their
clocks by merging them, which helps establish a clear order.
Chapter 10 Summary : 9 Leader election

Leader Election

Leader election is essential in distributed systems when a

single process must hold special responsibilities, such as
managing resource access or assigning tasks. This involves
electing a leader from candidate processes who will maintain
authority until an unavailability occurs. The primary goal of
a leader election algorithm is to ensure that only one leader
exists at any time (safety) and that the election completes
successfully (liveness). This chapter focuses on the Raft
leader election algorithm.

Raft Leader Election

Raft employs a state machine with three states for each

process: follower, candidate, and leader. Election terms are
represented by a logical clock, which increases over time.
The election process begins when a follower does not receive
a heartbeat from the leader within a timeout period,
prompting it to become a candidate and propose itself as
leader.
-
Winning the Election
: A candidate wins if it receives votes from the majority of
processes. When this occurs, it transitions to the leader state
and starts sending heartbeats.

-
Accepting New Leadership
: If a candidate receives a heartbeat from another process
indicating it is the leader with a higher or equal term, the
candidate accepts this leadership and returns to the follower
state.

-
Split Votes
: If multiple candidates emerge and no majority is reached,
this scenario is known as a split vote. Each candidate will
eventually time out and initiate a new election, with timeouts
randomized to minimize subsequent split votes.

Practical Considerations

While Raft is a straightforward and understandable leader

election protocol, many existing systems like etcd or
ZooKeeper offer abstractions facilitating leader election
implementation. These systems typically provide atomic
operations, such as compare-and-swap, with expiration times
to prevent stale leaders.
However, merely implementing leader election does not
guarantee there will be no concurrent leaders. If multiple
processes update a shared resource, care must be taken to
ensure that only one process leads the update operation.
Using fencing tokens can help ensure updates are valid and
authorized by the current leader.

Potential Downsides of Leader Election

Leaders can introduce bottlenecks, as they handle all

operations, potentially affecting performance. They also
represent a single point of failure; if a leader malfunctions or
the election process fails, the system may be compromised.
Solutions such as partitioning the system and assigning
separate leaders can mitigate some issues but introduce
added complexity.
Alternatives to leader election include optimistic locking
methods or tolerating occasional failures with non-critical
systems. Leaders should minimize workload to prevent
performance issues and be prepared for situations where
more than one leader might exist if proper token systems
aren't implemented.
Example
Key Point:Understanding the critical role of leader
election in maintaining system integrity.
Example:Imagine you are part of a team tasked with
organizing a large-scale event. Your team needs a
coordinator to make decisions and manage tasks. If the
main coordinator goes silent or unresponsive, someone
needs to step up to propose themselves as the new
leader to keep things moving forward. This is crucial
because without a single point of authority,
misunderstandings and chaos could ensue, similar to
how Raft ensures that only one leader at a time manages
responsibilities in a distributed system. Just like in your
event team, where you must ensure smooth
communication and clear leadership to avoid confusion,
a distributed system relies on leader election to
designate authority and maintain order.
Critical Thinking
Key Point:Dependency on leader election
mechanisms in distributed systems.
Critical Interpretation:While leader election algorithms
like Raft aim for efficiency and safety, it is critical to
analyze their effectiveness in practical applications, as
centralized leadership can introduce bottlenecks and
points of failure. The reliance on a single leader for
operational authority may not always be suitable,
especially for highly dynamic systems. This perspective
emphasizes the need for evaluating alternatives and
recent developments in distributed systems, such as
those mentioned by other researchers like Leslie
Lamport in 'Paxos Made Simple,' which explores
consensus in distributed environments without a single
leader. Therefore, acknowledging potential drawbacks
and controversies in the efficiency of leader election
methodology is essential for a balanced understanding.
Chapter 11 Summary : 10 Replication

Replication

Replication of data is crucial in distributed systems,

enhancing availability, scalability, and performance. If a
node fails, replicas ensure data is still accessible. However,
keeping replicas consistent poses challenges. This chapter
discusses the Raft replication algorithm, which provides
strong consistency guarantees, making data appear
centralized despite being replicated.

State Machine Replication

The Raft algorithm initializes with a leader election, allowing

only the elected leader to make changes. The leader
maintains a log of operations that are replicated to followers.
The process includes:
1. Appending a new log entry for each operation.
2. Sending AppendEntries requests to followers.
3. Committing operations upon receiving acknowledgments
from a majority.
The algorithm ensures that if a leader fails, a new leader can
be elected without causing consistency issues.

Consensus

State machine replication also tackles the consensus problem,

requiring fault-tolerant agreement among processes.
Consensus is essential in applications like transaction
commitments. While implementing consensus can be
complex, existing solutions like etcd and ZooKeeper simplify
the process through APIs.

Consistency Models

When a client requests data from a replicated store, the actual

response time varies due to network delays. There are several
consistency models:
1.
Strong Consistency
: Clients interact only with the leader, ensuring a single
version of the data. This leads to a linearizability guarantee,
where all effects of operations are visible to all processes
once completed.
2.
Sequential Consistency
: Allows requests to be processed by followers, creating a
trade-off between consistency and throughput. All operations
are applied in order, but without real-time visibility of
updates.
3.
Eventual Consistency
: Enables queries to any follower, risking that clients see stale
data. However, eventual consistency ensures that all replicas
converge over time, which might be suitable for specific
applications.
4.
CAP Theorem
: Outlines the trade-off between consistency, availability, and
partition tolerance during network partitions. The PACELC
theorem further expands on this, emphasizing the balance
between latency and consistency during normal operations.

Practical Considerations

High-availability, distributed data stores often come with

trade-offs in consistency guarantees. Understanding these
trade-offs is essential for application design, as they affect
performance based on the required consistency level.
Solutions may allow users to prioritize performance or
guarantees like those found in systems such as Azure's
Cosmos DB and Cassandra.
Example
Key Point:Importance of Replication in Distributed
Systems
Example:Imagine you're running a popular online game.
When players join, they expect a seamless experience
even if one server crashes. By replicating data across
multiple servers, your game continues to function,
offering players uninterrupted access. This replication
not only ensures availability but enhances scalability,
allowing more players to connect simultaneously.
However, maintaining consistency across those
replicated data points requires careful management, as
the Raft algorithm demonstrates by electing a leader to
coordinate updates. Armed with this understanding, you
can design your system to balance performance and
consistency, delivering a robust gaming experience.
Critical Thinking
Key Point:Reliance on Replication for Data
Availability
Critical Interpretation:Vitillo emphasizes the importance
of data replication in distributed systems, yet this
perspective may overlook potential drawbacks such as
increased complexity and overhead costs. Critics argue
that while replication enhances availability and
performance, it can also lead to significant challenges in
maintaining consistency (Zhao et al., 2021). Therefore,
it's crucial for readers to recognize that the author's
insights, while valuable, may not encompass all aspects
of replication dynamics in practice.
 Chapter 12 Summary : 11 Transactions
Topic Summary

Transactions Allow a group of operations to modify data with exclusive access, ensuring all succeed or none do.
Complex to implement across multiple services and datastores.

ACID Acronym for Atomicity, Consistency, Isolation, and Durability in transactions, ensuring reliable
operation. Focus mostly on atomicity and isolation in distributed transactions.

Isolation Prevents race conditions (e.g., dirty writes/reads). Stronger isolation levels (e.g., serializability)
prevent interference but impact performance.

Concurrency Control Pessimistic Concurrency Control uses locks, while Optimistic Concurrency Control checks for
conflicts without blocking (e.g., MVCC).

Atomicity Ensured in distributed systems by 2-phase commit (2PC), managed by a coordinator, but can block
during failures.

Asynchronous Alternatives to synchronous 2PC include Log-based Transactions for asynchronous updates and
Transactions Sagas, which manage local transactions and compensations.

Isolation in Managed using semantic locks with dirty flags, but risks deadlocks in the process.
Asynchronous
Transactions

Transactions

Transactions allow a group of operations modifying data to

have exclusive access and ensure that all operations succeed
or none do. While they work well for data owned by a single
service in a single datastore, implementing them across
multiple services with different datastores is complex.

ACID

In the context of transactions, ACID stands for:

-
Atomicity
: All operations must succeed or none at all; if any operation
fails, the entire transaction is rolled back.
-
Consistency
: The data must adhere to certain invariants and rules,
although this is less relevant in current discussions.
-
Isolation
: Concurrent transactions should not interfere with each
other, preventing race conditions.
-
Durability
: Once committed, changes must persist even in case of a
failure, often ensured through write-ahead logs.
Distributed transactions differ from traditional ACID
transactions, focusing on ensuring atomicity and isolation.

Isolation

Install
Isolation Bookey
prevents raceApp to Unlock
conditions such as Full Text and
dirty writes, dirty
Audio
reads, fuzzy reads, and phantom reads in concurrently
running transactions. Different isolation levels are available,
Chapter 13 Summary : 12 Functional
decomposition

Functional Decomposition

Microservices

Initially, applications often begin as monoliths, composed of

various components implementing different business
capabilities. As feature teams grow, codebases become
complex, leading to decreased productivity and increased
coupling. Transitioning to a microservice
architecture—where services are split into independently
deployable entities that communicate via APIs—can alleviate
these growing pains. Despite the term "micro," the services
can vary in size and complexity.

Benefits

Adopting microservices can significantly enhance

development speed by enabling small, autonomous teams to
manage individual services. Key benefits include:
- Reduced communication overhead due to smaller teams.
- Improved codebase manageability, making it easier to
onboard new hires.
- Independent scalability and technology choices for each
service.
- Flexibility in using distinct data models for each
microservice.

Costs

However, embracing microservices incurs costs related to

complexity and overhead:
- Increased operational overhead if services utilize different
technologies, necessitating some standardization.
- Significant resource provisioning efforts to manage
numerous services.
- Challenges with remote communication and the associated
performance impacts.
- More complex continuous integration and deployment
processes to ensure smooth service interactions.
- Increased operational demands, as development teams often
need to manage their own operations.
Practical Considerations

It is advisable to start with a monolith and transition to

microservices only when necessary. Establishing clear
service boundaries can be challenging within a monolith,
thus making it easier to refine before splitting.

API Gateway

Once services are created, an API gateway becomes essential

for client communication, acting as a facade that abstracts
backend APIs. This gateway provides routing, data
composition, and API translation to meet diverse client
needs.

Routing

It maps external API requests to internal services, allowing

for backward compatibility as internal endpoints evolve.

Composition

It enables the aggregation of data from various services into a

single response for clients, reducing the load on them to
query multiple services directly.

Translation

The gateway can convert different IPC mechanisms and

tailor APIs based on client requirements, leading to the
implementation of graph APIs for flexibility.

Cross-Cutting Concerns

The API gateway handles essential cross-cutting functions

like caching and security, managing authentication and
authorization, and providing a single entry point to simplify
these concerns for internal services.

Caveats

API gateways can introduce development bottlenecks,

requiring maintenance and scalability considerations as they
grow to accommodate numerous services.

CQRS

Command Query Responsibility Segregation (CQRS)

involves separating read and write operations, allowing for
optimized data stores tailored to specific use cases. This
leads to potential complexity in maintaining synchronization
between the paths.

Messaging

As services communicate increasingly through network calls,

messaging systems allow for indirect communication, which
facilitates asynchronous operations and enhances robustness
against service unavailability. Through message brokers,
different communication styles can be achieved: one-way
messaging, request-response, and broadcast messaging.

Guarantees

Messaging services, like AWS SQS, offer varying guarantees

on message delivery and ordering, introducing trade-offs that
depend on scalability needs.

Exactly-Once Processing

Achieving exactly-once message processing is challenging,

requiring message idempotence to avoid data loss.
Failures

To guard against repeated processing failures, dead-letter

channels store messages that have been retried beyond a
certain threshold.

Backlogs

Monitoring message backlogs is essential to ensure systems

remain robust and scalable, addressing situations where
consumer rates lag behind producer throughputs.

Fault Isolation

Mitigation strategies should be employed for problematic

producers to prevent degradation of overall system
performance.

Reference Plus Blob

For transmitting large objects, the queue plus blob pattern

recommends using external storage for large data, enabling
efficient metadata management while recognizing the
complexities in doing so outside transactional contexts.
Chapter 14 Summary : 13 Partitioning

Partitioning in Distributed Systems

Partitioning, also known as sharding, is a technique used to

scale applications by distributing a dataset across multiple
nodes when it grows too large for a single node. This chapter
focuses on the implementation of a sharded key-value store.

Sharding Strategies

When handling requests in a partitioned data store, it is

crucial to route them to the correct node managing the
relevant partition. This can be achieved via a gateway service
that utilizes a mapping of keys to partitions stored in a
configuration service like etcd or Zookeeper. The primary
methods for mapping keys to partitions are:

Range Partitioning

In range partitioning, data is divided based on

lexicographical key ranges, with each partition containing a
contiguous range of keys. While this can enable efficient
range scans, it can lead to unbalanced partitions if key
distribution is not even and may cause performance hotspots
in certain access patterns.

Hash Partitioning

Hash partitioning employs a hash function to distribute keys

across partitions uniformly. This method helps in balancing
the number of entries across partitions but does not prevent
hotspots from certain frequently accessed keys. A strategy to
minimize the need for data reshuffling when adding
partitions is stable hashing, including approaches like ring
hashing and consistent hashing, which allow new partitions
to be added with minimal data movement.

Rebalancing

Rebalancing is the process of adjusting the number of nodes

serving partitions in response to changes in request volume
or dataset size. Effective rebalancing minimizes disruptions,
ensuring the data store remains operational during these
changes.

Static Partitioning
This method involves initializing more partitions than
immediately needed and distributing them across nodes.
While it helps maintain balance, it is inflexible as the number
of partitions cannot be easily altered.

Dynamic Partitioning

Dynamic partitioning creates partitions on-demand, starting

with a single partition that splits when it exceeds a certain
size or becomes overly accessed. This approach allows for
flexible adjustments but adds complexity to the system.

Practical Considerations

Partitioning introduces complexity, such as managing

partition imbalances and ensuring atomic updates across
multiple partitions. Maintaining efficiency while scaling
requires careful consideration, and readers are encouraged to
explore deeper topics in Martin Kleppmann's "Designing
Data-Intensive Applications" for further insights.
Chapter 15 Summary : 14 Duplication

Chapter 15 Summary: Duplication in Distributed

Systems

14 Duplication

This chapter explores duplication as a strategy for designing

horizontally scalable applications. It details the importance of
load balancing and replication to enhance system
performance and reliability.

14.1 Network Load Balancing

Network load balancing involves creating multiple instances

of a service to distribute requests efficiently, improving
capacity and availability. A load balancer (LB) acts as a
middleman between clients and servers, facilitating
transparent scaling and providing features such as service
discovery and health checks.

Load Balancing Algorithms

LBs can use various algorithms (e.g., round-robin,
health-based) to route requests. Load metrics are critical, but
they can lead to herding effects. Combining delayed load
metrics with randomness can mitigate this issue.

Service Discovery

This mechanism allows LBs to identify available servers.

Options include static configurations, DNS, or using a data
store for dynamic discovery, allowing for automatic scaling.

Health Checks

Health checks ensure that failing servers are removed from

active pools, with both passive and active checks being
utilized to monitor server statuses.

14.1.1 DNS Load Balancing

Basic load balancing can be achieved with DNS, but it is

Install
limited Bookey
by issues App todelays
like caching Unlock Full Text
and failure and
detection.
Audio
14.1.2 Transport Layer Load Balancing
Chapter 16 Summary : 15 Common
failure causes

15 Common Failure Causes

In distributed systems, it's crucial to understand common

causes of failure to effectively safeguard against them. Key
failure types include single points of failure, unreliable
networks, slow processes, unexpected load, and cascading
failures.

15.1 Single Point of Failure

A single point of failure can cripple a system if a critical

component fails. Multiple single points of failure may exist
within a system. For instance, a service dependent on a
non-replicated database exemplifies this risk; failure to
access the database means the service cannot start.
Identifying these weaknesses during the architectural phase is
essential, allowing for mitigation through redundancy or
minimizing potential damage.
15.2 Unreliable Network

Remote calls may not yield responses due to various network

issues such as dropped requests or server crashes. Clients
struggle to determine whether to wait for a response or fail
the request, leading to prolonged waiting and
difficult-to-trace delays. This unreliability poses a significant
threat to system performance, requiring protective measures
for the client-side as detailed in Chapter 16.

15.3 Slow Processes

Slow processes can be mistaken for non-responsive ones.

Resource leaks, particularly memory leaks, are common
culprits—leading to increased memory usage until the
system fails to allocate more, causing significant slowdowns.
Additionally, blocking threads within limited pools can
escalate this issue. Both synchronous and asynchronous calls,
if not managed with timeouts, can contribute to resource
exhaustion.

15.4 Unexpected Load

Systems can fail due to unexpected spikes in load, which can

arise from various stimuli including seasonal traffic or
malicious attacks like DDoS. Preparing in advance for these
spikes is critical for system resilience. Techniques to manage
unexpected load will be discussed in Chapter 17.

15.5 Cascading Failures

Failures can propagate through a system, creating cascading

failures where one component's failure increases the
likelihood of others failing. For instance, if one database
replica goes down, the load balancer redirects traffic to
another, potentially overloading it and leading to a
system-wide collapse. Preventing such cascading effects is
vital to maintaining system integrity.

15.6 Risk Management

Distributed systems must anticipate failures and plan

accordingly. Evaluating the probability and impact of failures
allows for prioritization of prevention and mitigation efforts.
By calculating a risk score, teams can effectively allocate
their resources, focusing first on high-impact and likely
failures while managing lower-priority risks separately.
Chapter 17 Summary : 16 Downstream
resiliency

16 Downstream Resiliency

This chapter discusses patterns that protect a service from

failures in its downstream dependencies.

16.1 Timeout

Setting a timeout for network calls is crucial to avoid

resource leaks when calls do not return. Timeouts help limit
and isolate failures, which is essential across different
contexts, including network requests and resource pool
accesses. For instance, JavaScript's XMLHttpRequest has a
default timeout of zero, while many other libraries default to
an infinite timeout, leading to potential issues. It is
recommended to set reasonable default timeouts and monitor
network call metrics. Libraries that handle timeouts and
monitoring can reduce the complexity for developers.

16.2 Retry
Upon failure or timeout of a network request, clients can
either fail fast or retry the request. Implementing a backoff
strategy, particularly exponential backoff, helps in managing
retries effectively, spreading out requests to avoid
overwhelming the downstream service. However, retries
should be approached cautiously, especially when the
possible errors are not transient or the operation is
non-idempotent. Additionally, care should be taken in
multi-level dependency scenarios to prevent retry
amplification, where retries at multiple levels could lead to
system overload.

16.3 Circuit Breaker

In cases of persistent downstream failures, a circuit breaker

pattern can mitigate cascading failures by blocking new
requests to the unresponsive service. The circuit breaker
operates as a state machine with three states: open, closed,
and half-open. It monitors the health of the downstream
dependency and allows graceful degradation of services
rather than total failure. Key considerations include
determining thresholds for failures and the timing for
transitioning between states based on historical data.
This chapter emphasizes the importance of resilience
patterns—timeouts, retries, and circuit breakers—when
dealing with failures in distributed systems, aiming to
maintain system integrity and performance.
 Chapter 18 Summary : 17 Upstream
resiliency
Section Description

Overview This chapter discusses mechanisms to protect against upstream pressure in distributed systems, focusing on
load handling strategies.

Load A technique where servers reject excess requests during overload to maintain performance, returning a 503
Shedding status code.

Load Introduces a messaging channel for clients to send requests without immediate expectations, smoothing out
Leveling traffic spikes.

Rate-Limiting Restricts the number of requests a client can make within a timeframe, returning status code 429 when limits
are exceeded.

Bulkhead Aims to isolate faults, minimizing impacts on the overall service by partitioning resources or instances.
Pattern

Health Informs load balancers of service status to route requests appropriately, utilizing various health checks.
Endpoint

Watchdog Monitors a process’s health and automatically restarts it if thresholds are breached, promoting self-healing
properties.

Chapter 18: Upstream Resiliency

Overview

This chapter discusses mechanisms to protect against

upstream pressure in distributed systems, focusing on load
handling strategies such as load shedding, load leveling,
rate-limiting, the bulkhead pattern, health endpoints, and
watchdog mechanisms.
Load Shedding

Load shedding refers to a technique where servers reject

excess requests during overload conditions to maintain
performance on already active requests. This is accomplished
by measuring system load, such as concurrent requests, and
returning a 503 (Service Unavailable) status code when
overloaded. The technique helps servers focus resources but
does not completely eliminate the costs associated with
rejecting requests.

Load Leveling

Load leveling introduces a messaging channel that allows

clients to send requests without immediate response
expectations, helping to smooth out spikes in traffic. This
approach decouples request rate from service capacity,
enabling services to pull requests at a manageable pace. Load
leveling can be combined with auto-scaling strategies to
handle increases in traffic adequately.
Install Bookey App to Unlock Full Text and
Rate-Limiting Audio
Chapter 19 Summary : 18 Testing

Testing in Distributed Systems

Testing is essential for early bug detection, enabling

confidence in code changes and overall system design
improvement. While testing cannot guarantee bug-free code,
it effectively uncovers known failure scenarios and validates
expected behaviors.

Scope of Tests

Testing can be categorized based on the scope of the system

under test (SUT):
-
Unit Tests
: Focus on small codebase components (e.g., individual
classes). They should change only with behavior alterations
of the SUT.
-
Integration Tests
: Verify service interactions with external dependencies; can
be narrow (specific to service dependencies) or broad (across
multiple services).
-
End-to-End Tests
: Validate user-facing scenarios over the entire application,
usually run in shared environments. They may be slower and
more complex to maintain due to their comprehensive scope.
While end-to-end tests are crucial, they should be minimized
by framing them as user journey tests to enhance efficiency.
The general guideline suggests a greater number of unit tests
compared to integration and end-to-end tests.

Size of Tests

Test size refers to the resources required, including the

number of nodes used:
-
Small Tests
: Run in a single process, fast, and less prone to intermittent
failures.
-
Intermediate Tests
: Run on a single node and may involve local I/O, increasing
non-determinism.
-
Large Tests
: Require multiple nodes, resulting in longer execution times
and higher flakiness.
To reduce test size without compromising scope, developers
can use test doubles, which include:
-
Fakes
: Lightweight implementations behaving similarly to real
ones.
-
Stubs
: Functions returning consistent values regardless of input.
-
Mocks
: Objects with expectations regarding their usage.
Using real implementations is preferable when feasible, with
fakes offering a useful compromise. Stubbing or mocking
should be last-resort options.

Practical Considerations

Testing requires trade-offs in scope and size. For example,

when testing a user-facing API endpoint, it may be efficient
to use mocks for dependencies that can be simulated (e.g., an
in-memory data store) while relying on playground
environments for external APIs to avoid real transactions.
In high-stakes scenarios, such as ensuring GDPR compliance
via user data purging, utilizing end-to-end testing is
warranted despite its challenges, as the cost of failure is
significant.
Critical Thinking
Key Point:The necessity and complexity of testing in
distributed systems.
Critical Interpretation:The chapter emphasizes that
testing is essential for early bug detection, suggesting it
as a critical practice in distributed systems. However,
while the author's assertion about the importance of
rigorous testing is valid, one must critically evaluate
whether the proposed categorization of tests effectively
covers the nuances and challenges unique to distributed
systems. Relying heavily on unit tests and minimizing
end-to-end tests may overlook the dynamic interactions
and potential failures that can arise in such
environments. As noted in various studies on software
testing, including "Testing Methods in Software
Development" (Myers et al.) and "Agile Testing"
(Kaner et al.), a more balanced approach to testing,
which includes extensive real-world simulations, may
yield better robustness in real-world operations,
counteracting the author's narrower practices.
Chapter 20 Summary : 19 Continuous
delivery and deployment

Continuous Delivery and Deployment

Continuous Delivery (CD) and Deployment is an automated

process ensuring that changes to a system can be released to
production safely and efficiently. This chapter addresses the
challenges of manual deployments, which can lead to
complications, especially in systems involving multiple
services. By automating deployments, developers can focus
on their tasks without needing to monitor releases.

Review and Build

The release process consists of a four-stage pipeline: review,

build, pre-production rollout, and production rollout. The
journey begins with a pull request (PR), requiring
compilation, static analysis, and validation through quick
tests. A checklist for reviewers ensures safety and correctness
before merging changes into the main repository.
Pre-production

In this stage, the artifact is deployed to a synthetic

pre-production environment for initial checks. This allows
for early bug detection without the full scale of production.
Multiple pre-production environments can simulate traffic
and various scenarios to ensure stability before the actual
rollout.

Production

After successful pre-production checks, the artifact is

released to production, starting with a limited number of
instances to identify potential issues early. The rollout is
conducted incrementally, allowing capacity management and
minimizing risk. The process can utilize stages to speed up
releases while maintaining safety.

Rollbacks

The CD pipeline continuously assesses the health of the

deployed artifact. If issues arise, the pipeline either rolls back
or alerts engineers for further evaluation. Monitoring
includes health metrics and the impact on related services.
Forwarding changes, especially backward-incompatible ones,
should be managed carefully to ensure smooth transitions. A
structured approach to making these changes includes
preparing, activating, and cleaning up to preserve system
integrity.
By investing in a well-structured CD pipeline, teams can
enhance deployment efficiency and reduce production
failures.
Critical Thinking
Key Point:The potential over-reliance on automation
in Continuous Delivery (CD) processes.
Critical Interpretation:While the author emphasizes the
efficiency of automation in deployment processes, it's
crucial to critically assess the reliance on such systems.
Automation can obscure underlying issues that might
require human judgment and expertise. For instance, the
complexity of distributed systems may lead to
unforeseen challenges during automated rollouts that a
human overseer could catch. Critics argue that
over-automation might reduce the rigor of manual
checks that are sometimes necessary for truly safe
deployments. As noted by experts like Jez Humble in
"Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment Automation," the
human element in deployment processes remains vital
for ensuring the health and correctness of changes,
challenging the notion that automation alone can
sufficiently safeguard against all pitfalls inherent in
complex systems.
Chapter 21 Summary : 20 Monitoring

Monitoring

Monitoring is crucial for detecting failures that affect users

and providing system health overviews through dashboards.
It has transitioned from a black-box approach (indicating
service status) to a white-box approach (using
application-level metrics). Blackbox monitoring focuses on
external dependencies, while white-box monitoring helps
identify root causes of failures.

Metrics

Metrics are numeric representations of information over

time, forming time-series data. Modern systems allow
tagging with labels for better analysis. Metrics can indicate
system load, internal states, and service performance,
requiring explicit instrumentation by developers for effective
monitoring.

Service-level Indicators (SLIs)

SLIs measure aspects of service quality, like response time
and error rates, typically presented as ratios of successful to
total events. They provide clarity for monitoring and alerting
interests.

Service-level Objectives (SLOs)

SLOs define acceptable ranges for SLIs to maintain service

health. They set user expectations and help prioritize repairs
over new features using an error budget to manage tolerable
failures. The balance between stringency and practicality in
SLOs is crucial.

Alerts

Alerts trigger actions upon certain conditions, focusing on

actionable items rather than vague metrics. Effective alerting
requires a balance between precision and recall, considering
the burn rate of error budgets.

Dashboards
Install Bookey App to Unlock Full Text and
Dashboards act as real-timeAudio
data visualizations representing
system health and should be designed with clear objectives
Chapter 22 Summary : 21 Observability

21 Observability

A distributed system is perpetually in a state of potential

failure, requiring operators to manage complexities
introduced by tolerant failure modes and resiliency
mechanisms. Human intervention remains crucial for
incident management and debugging, necessitating the use of
observability tools to understand emergent behaviors.

Observation Tools and Their Role

Observability enhances understanding through telemetry

sources such as metrics, logs, and traces. Metrics help with
system monitoring, while logs and traces serve debugging
purposes. Effective observability platforms reduce the time
required to validate hypotheses about system performance.

21.1 Logs

Logs record time-stamped events and can vary in format,

including free-form text, structured (JSON), or binary. They
originate from services and external dependencies, aiding
debugging and tracking down root causes. However, they can
lead to high noise levels if poorly managed, resulting in
expensive data storage issues. Best practices involve
consolidating data into single events, controlling logging
verbosity through dynamic levels, and prioritizing important
events.

21.2 Traces

Tracing captures the lifecycle of requests across services

using spans, which represent time intervals for operations.
Traces enable developers to debug specific requests, identify
bottlenecks, and manage resource attribution. A unique trace
ID is assigned upon request initiation, which is propagated
throughout the execution flow. Implementing tracing requires
modifying all components within the request path to support
trace propagation.

21.3 Integration of Logs, Metrics, and Traces

The specificity of logs makes them less useful for overall

system health assessment, leading to the need for metrics and
traces, which provide a higher-level view. Metrics
summarize statistics from various events, while traces
aggregate events into a cohesive request flow. Collectively,
these tools enhance the ability to monitor and debug
distributed systems effectively.
Chapter 23 Summary : 22 Final words

Final Words

Congratulations on reaching the end of the book! The

journey into understanding distributed systems begins here.

Learning Recommendations

To further enhance your knowledge, consider exploring key

industry papers that offer valuable insights:
1.
Windows Azure Storage

This paper discusses Azure's highly available cloud storage

service that emphasizes strong consistency, a crucial design
choice that simplifies developers' tasks compared to AWS
S3.
2.
Azure Data Explorer

This paper details a cloud-native event store built on

Azure's storage solution, showcasing how large-scale
systems can integrate and operate together.
3.
System Design Interview by Alex Xu

If you are preparing for system design interviews, this book

provides a structured framework along with over ten case
studies.
Embrace the learning journey ahead!
Best Quotes from Understanding
Distributed Systems by Roberto Vitillo
with Page Numbers
View on Bookey Website and Generate Beautiful Quote Images

Chapter 1 | Quotes From Pages 3-8

[Link] to build distributed systems is hard,
especially if they are large scale.
[Link] problem is that the available information is spread out
all over the place, and if you were to put it on a spectrum
from theory to practice, you would find a lot of material at
the two ends, but not much in the middle.
[Link] is why I decided to write a book to teach the
fundamentals of distributed systems so that you don’t have
to spend countless hours scratching your head to
understand how everything fits together.
[Link] you can build applications without knowing any
of that, you will end up spending hours debugging and
re-designing their architecture, learning lessons that you
could have acquired in a much faster and less painful way.
[Link] you are interviewing for a senior role, you are expected
to be able to design complex networked services and dive
deep into any vertical.
Chapter 2 | Quotes From Pages 9-17
1.A distributed system is one in which the failure of
a computer you didn’t even know existed can
render your own computer unusable. – Leslie
Lamport
[Link] applications are inherently distributed.
3.A fault is a component that stopped working, and a system
is fault-tolerant when it can continue to operate despite one
or more faults.
[Link] matter how small that probability is, the more
components there are, and the more operations the system
performs, the higher the absolute number of failures
becomes.
[Link] coordination is such a key topic, the second part of
this book is dedicated to distributed algorithms used to
implement coordination.
[Link] the system isn’t resilient to failures, which only increase
as the application scales out to handle more load, its
availability will inevitably drop.
[Link] same team that designs a system is also responsible for
its live-site operation.
Chapter 3 | Quotes From Pages -27
[Link] guarantees that a stream of bytes arrives in
order, without any gaps, duplication or corruption.
[Link] create the illusion of a reliable channel, TCP partitions a
byte stream into discrete packets called segments.
[Link] handshake introduces a full round-trip in which no
application data is sent.
[Link] lower the round trip time (RTT) is, the quicker the
sender can start utilizing the underlying network’s
bandwidth.
[Link]’s reliability and stability come at the price of lower
bandwidth and higher latencies than the underlying
network is actually capable of delivering.
[Link] a snapshot is lost in transmission, there is no value in
retransmitting it as the game evolves in real-time.
Chapter 4 | Quotes From Pages -32
[Link] provides encryption, authentication, and
integrity.
[Link] beauty of this approach is that the shared secret is
never communicated over the wire.
[Link] you have a very good reason, you should use TLS
for all communications.
[Link] can bring an entire service down as clients are no
longer able to connect with it.
[Link] negotiation checks several things, like the certificate's
expiration date and whether the digital signature was
actually signed by the issuing CA.
[Link] a new connection is expensive; yet another reason
to put your servers geographically closer to the clients and
reuse connections when possible.
Chapter 5 | Quotes From Pages 33-35
[Link] beauty is that the address of a root name
server is all that’s needed to resolve any hostname.
[Link] resolution process involves several round trips in the
 worst case, but its beauty is that the address of a root name
server is all that’s needed to resolve any hostname.
[Link] a TTL requires making a tradeoff.
[Link] can easily become a single point of failure.
Chapter 6 | Quotes From Pages 36-47
[Link] start out as beautifully-designed interfaces.
Slowly, but surely, they will need to change over
time to adapt to new use cases.
[Link] last thing you want to do when evolving your API is to
introduce a breaking change that requires modifying all the
clients in unison.
[Link]-compatible APIs tend to be not particularly
elegant, but they are a necessary evil.
Chapter 7 | Quotes From Pages 51-53
1.A system model encodes assumptions about the
behavior of nodes, communication links, and
timing; think of it as a set of assumptions that
allow us to reason about distributed systems by
ignoring the complexity of the actual technologies
used to implement them.
[Link] reliable link model assumes that a message is delivered
exactly once, without loss or duplication.
[Link], it can be theoretically proven that a system
with Byzantine nodes can tolerate up to of faulty nodes and
still operate correctly.
[Link] of that, algorithms for different node models look
very different from each other.
[Link] remember, models are just an abstraction of reality,
and sometimes abstractions leak.
[Link] you read along, question the models’ assumptions and
try to imagine how algorithms that rely on them could
break.
Chapter 8 | Quotes From Pages 54-55
[Link] bottom line is that it’s not possible to build a
perfect failure detector.
[Link] it’s too short and the server is reachable, the client will
wrongly consider the server dead; if it’s too long and the
server is not reachable, the client will block waiting for a
response.
3.A ping is a periodic request that a process sends to another
to check whether it’s still available.
4.A heartbeat is a message that a process periodically sends
to another to inform it that it’s still up and running.
Chapter 9 | Quotes From Pages -61
1.'Time is an essential concept in any application,
even more so in distributed ones.'
2.'There is no shared global clock that all processes agree on
and can be used to order their operations.'
3.'To solve this problem, we need to look at it from another
angle.'
4.'A logical clock measures the passing of time in terms of
 logical operations, not wall-clock time.'
5.'The beauty of vector clock timestamps is that they can be
partially ordered.'
Chapter 10 | Quotes From Pages -67
1.A leader election algorithm needs to guarantee
that there is at most one leader at any given time
and that an election eventually completes.
[Link] the candidate wins the election, it transitions to the leader
state and starts sending out heartbeats to the other
processes.
[Link] the system starts up, all processes begin their journey
as followers.
[Link] process can vote for at most one candidate in a term
on a first-come-first-served basis.
[Link] it doesn’t receive any heartbeat within a certain time
period, a timeout fires and the leader is presumed dead.
Chapter 11 | Quotes From Pages -78
[Link] Data replication is a fundamental
building block of distributed systems.
[Link] reason to replicate data is to increase availability.
[Link] is a fundamental problem studied in distributed
systems research, which requires a set of processes to agree
 on a value in a fault-tolerant way.
[Link] algorithm guarantees that an entry that is committed is
durable and will eventually be executed by all the
processes in the system, not just those that were part of the
original majority.
[Link] consistency model in which operations occur in the
same order for all observers, but doesn’t provide any
real-time guarantee about when an operation’s side-effect
becomes visible to them, is called sequential consistency.
[Link] eventually consistent store is perfectly fine if you want
to keep track of the number of users visiting your website,
as it doesn’t really matter if a read returns a number that is
slightly out of date.
[Link] consistency, availability and partition tolerance: pick
two out of three.
Chapter 12 | Quotes From Pages -93
[Link] relieve you from a whole range of
possible failure scenarios so that you can focus on
the actual application logic rather than all possible
 things that can go wrong.
[Link] achieve that, the withdrawal and deposit need to be
wrapped in an inseparable unit: a transaction.
[Link] application might not need serializability, but you
need to consciously decide which isolation level to use and
understand its implications, or your data store will silently
make the decision for you.
[Link] Saga guarantees that either all local transactions
succeed, or in case of failure, that the compensating local
transactions undo the partial execution of the transaction
altogether.
5.A message log is an append-only, totally ordered sequence
of messages, in which each message is assigned a unique
sequential index.
Chapter 13 | Quotes From Pages -121
[Link] down the backend by business
capabilities into a set of services with well-defined
boundaries allows each service to be developed
and operated by a single small team.
[Link] the number of feature teams contributing to the same
codebase increases, the components become increasingly
coupled over time.
[Link] term micro can be misleading, though — there doesn’t
have to be anything micro about the services.
[Link] teams can increase the application’s development
speed for a variety of reasons: They are more effective as
the communication overhead grows quadratically with the
team’s size.
[Link] the integration of all the microservices is an order
of magnitude harder.
[Link] logging and monitoring becomes crucial.
[Link] keep the read and write data models synchronized, the
write path pushes updates to the read path whenever the
 data changes.
[Link] provides several other benefits: It allows a
client to execute an operation on a service asynchronously.
[Link] of the drawbacks of using an API gateway is that it
can become a development bottleneck.
[Link] can upload a blob to an object storage service, like
AWS S3 or Azure Blob Storage, and then send the URL
of the blob via message.
Chapter 14 | Quotes From Pages -128
[Link] a dataset no longer fits on a single node, it
needs to be partitioned across multiple nodes.
[Link] idea behind hash partitioning is to use a hash function
to assign keys to partitions, which shuffles — or uniformly
distributes — keys across partitions.
[Link] needs to be implemented in such a way to
minimize disruption to the data store, which needs to
continue to serve requests.
[Link] partition is independent of the others, transactions are
required to update multiple partitions atomically.
[Link] you are interested to learn more about it, I recommend
reading Designing Data-Intensive Applications by Martin
Kleppmann.
Chapter 15 | Quotes From Pages -151
[Link] more service instances can be a fast and
cheap way to scale out a stateless service, as long
as you have taken into account the impact on its
dependencies.
[Link] clients are decoupled from servers and don’t need
to know their individual addresses, the number of servers
behind the LB can be increased or reduced transparently.
[Link] cached, and hence delayed, metrics to distribute
requests to servers can create a herding effect.
[Link] idea is to randomly pick two servers from the pool and
route the request to the least-loaded one of the two.
[Link] checks are used by the LB to detect when a server
can no longer serve requests and needs to be temporarily
removed from the pool.
[Link] the clients are internal to an organization, the L7 LB
 functionality can alternatively be bolted onto the clients
directly using the sidecar pattern.
[Link] mitigate these performance issues, you can distribute the
traffic to different data centers located in different regions.
[Link] is the process of storing a copy of the same
data in multiple nodes.
9.A cache can be introduced to reduce the load on the
dependency and improve the performance of accessing the
data.
[Link]’s a design smell if your system can’t cope at all without
a cache.
Chapter 16 | Quotes From Pages -161
1.A single point of failure is the most glaring cause
of failure in a distributed system; if it were to fail,
that one component would bring down the entire
system with it.
[Link] network calls are the silent killers of distributed
systems.
[Link] failures are very hard to get under control once
they have started.
[Link] a specific failure, you have to consider its
probability of happening and the impact it causes to your
system if it does happen.
Chapter 17 | Quotes From Pages -170
[Link] a rule of thumb, always set timeouts when
making network calls, and be wary of third-party
libraries that do network calls or use internal
resource pools but don’t expose settings for
timeouts.
[Link] fastest network call is the one you don’t have to make.
[Link] the failure or timeout was caused by a short-lived
connectivity issue, then retrying after some backoff time
has a high probability of succeeding.
[Link] are helpful when the expectation is that the next call
will succeed, while circuit breakers are helpful when the
expectation is that the next call will fail.
[Link] waiting and retrying failed network requests isn’t
the only way to implement retries. In batch applications
that don’t have strict real-time requirements, a process can
park failed requests into a retry queue.
Chapter 18 | Quotes From Pages -184
[Link] a server operates at capacity, there is no
good reason for it to keep accepting new requests
since that will only end up degrading it.
[Link] leveling is an alternative to load shedding, which can
be used when clients don’t expect a response within a short
time frame.
[Link]-limiting is also used to enforce pricing tiers; if a user
wants to use more resources, they also need to be prepared
 to pay more.
[Link] goal of the bulkhead pattern is to isolate a fault in one
part of a service from taking the entire service down with
it.
5.A smart load balancer instead detects that a large fraction
of the service instances is being reported as unhealthy and
considers the health check to no longer be reliable.
[Link] crashes are inevitable and your service is prepared
for them, you don’t have to come up with complex
recovery logic when a process gets into some weird
degraded state — you can just let it crash.
Chapter 19 | Quotes From Pages -193
[Link] longer it takes to detect a bug, the more
expensive it becomes to fix it.
[Link], because it’s impossible to predict all the
ways a complex distributed application can fail, testing
only provides best-effort guarantees that the code being
tested is correct and fault-tolerant.
[Link] a rule of thumb, if you want to be confident that your
implementation behaves in a certain way, you have to add a
test for it.
[Link]-to-end tests can be painful and expensive to maintain.
For example, when an end-to-end test fails, it’s not always
obvious which service is responsible and deeper
investigation is required.
[Link]-failing tests are nearly as bad as no tests at
all, as developers stop having any confidence in them and
eventually ignore their failures.
[Link] possible, prefer tests with smaller scope as they tend
to be more reliable, faster, and cheaper.
[Link] less the resemblance is, the less confidence you should
have that the test using the double is actually useful.
Chapter 20 | Quotes From Pages 194-199
[Link] a change has been merged to a repository, it
should automatically be rolled out to production
safely.
[Link] deployments are a terrible use of engineering time.
3.A good CD pipeline should strive to make a good trade-off
between the two [safety of a rollout and the time it takes to
release a change to production].
[Link] of the most common causes of production failures are
configuration changes applied globally without any prior
review or testing.
[Link] forward is much riskier than rolling back, any
change introduced should always be backward compatible
as a rule of thumb.
[Link] pipeline should allow enough time to pass between one
step and the next (bake time) to ensure that it was
successful, as some issues can appear only after some time
 has passed.
[Link] safely introduce a backward-incompatible change, it
needs to be broken down into multiple
backward-compatible changes.
Chapter 21 | Quotes From Pages 200-216
1.'A common approach is to periodically run scripts
that send test requests to external API endpoints
and monitor how long they took and whether they
were successful.'
2.'As a rule of thumb, if you can’t design away a hard-failure
mode, you should add monitoring for it.'
3.'Metrics are time-series and can be modeled and
manipulated with mathematical tools.'
4.'For example, if the 99th percentile is 1 second, then 99 %
of requests have a response time below or equal to 1
second.'
5.'An SLO defines a range of acceptable values for an SLI
within which the service is considered to be in a healthy
state.'
6.'Once you have decided what to measure, you need to
decide where to measure it.'
7.'Achieving a healthy on-call is only possible when alerts
are actionable.'
8.'The first step to address an alert is to mitigate it, not fix the
underlying root cause that created it.'
9.'Users can become over-reliant on the actual behavior of
your service rather than the published SLO.'
10.'Good dashboards don’t happen by coincidence.'
Chapter 22 | Quotes From Pages 217-224
1.A distributed system is never 100% healthy at any
given time as there can always be something
failing.
[Link] is a set of tools that provide granular insights
into a system in production, allowing us to understand its
emergent behaviours.
[Link] are mainly used for monitoring, while event logs
and traces mainly for debugging.
4.A good observability platform strives to minimize the time
it takes to validate hypotheses.
[Link] make the job of the engineer drilling into the logs less
painful, all the data about a specific work unit should be
stored in a single event.
[Link] main drawback of event logs is that they are
fine-grained and service-specific.
Chapter 23 | Quotes From Pages 225-226
[Link] this is the end of the book, it’s just the
beginning of your journey.
[Link] of the best ways to learn how to design large scale
systems is by standing on the shoulders of the giants.
[Link] papers provide a wealth of knowledge about
distributed systems that have stood the test of time.
[Link] of the key design decisions was to guarantee strong
consistency, unlike AWS S3, making the application
developers’ job much easier.
[Link], if you are preparing for the system design
interview, check out Alex Xu’s book ‘System Design
Interview.’
Understanding Distributed Systems
Questions
View on Bookey Website

Chapter 1 | Understanding Distributed Systems|

Q&A
[Link]
Why is expertise in distributed systems so important for
modern engineering roles?
Answer:Expertise in distributed systems is critical
because modern applications are fundamentally
built as distributed systems, enabling scalability,
reliability, and support for a vast number of users
and devices.

[Link]
What challenges did the author face when learning about
distributed systems?
Answer:The author struggled with finding accessible
resources that connected theoretical concepts with practical
applications, leading to a fragmented understanding that
required hours of effort to piece together.
[Link]
Who should consider reading this book?
Answer:This book is aimed at back-end developers of web or
mobile applications, experienced engineers seeking to
enhance their skills, and anyone preparing for system design
interviews at major tech companies.

[Link]
What benefits does the book offer to new learners in the
field of distributed systems?
Answer:The book provides a comprehensive guide that
bridges the gap between theory and practice, enabling
learners to understand essential concepts more rapidly and
effectively than they would through disjointed resources.

[Link]
How does the book intend to evolve over time?
Answer:The author plans to regularly update the book to
reflect new developments in the field and incorporate reader
feedback to continuously improve its content.

[Link]
Why is understanding the network stack and data
consistency important for engineers?
Answer:Understanding these concepts is essential because
they facilitate the design of robust and efficient distributed
systems, allowing engineers to prevent common pitfalls,
optimize performance, and ensure data integrity.

[Link]
What might happen if an engineer doesn't understand
distributed systems principles when designing
applications?
Answer:Without this understanding, an engineer may face
significant challenges, such as spending excessive time
debugging issues or having to redesign architectures,
ultimately leading to inefficient development processes and
potentially inferior applications.

[Link]
How can this book aid someone preparing for a technical
interview?
Answer:The book serves as a valuable study companion for
system design interviews, equipping candidates with the
knowledge needed to tackle complex distributed systems
questions that are often pivotal in landing a job with top-tier
tech companies.

[Link]
What is the overall aim of the book according to the
author?
Answer:The overall aim is to provide a structured, pragmatic
introduction to distributed systems, enabling readers to
quickly and effectively acquire the knowledge necessary to
succeed in developing and managing these complex systems.
Chapter 2 | 1 Introduction| Q&A
[Link]
What is a distributed system and why is it important?
Answer:A distributed system is a network of nodes
(physical devices or software processes) that
collaborate to perform tasks by exchanging
messages. It is important because many modern
applications, such as web services and cloud storage,
require high availability, resilience to failures, and
the ability to handle large workloads that a single
machine cannot manage. Examples include Dropbox
for data redundancy and Google for processing
extensive search queries.

[Link]
What challenges do engineers face when building
distributed systems?
Answer:Engineers face several challenges, including
communication issues, coordination among nodes despite
failures, scalability to handle increased loads, maintaining
resiliency in the face of inevitable failures, and effective
operations management. Communication involves ensuring
reliable message exchange, while coordination addresses
how to maintain consensus among nodes. Scalability pertains
to how well a system can manage increased demands, and
resiliency entails the system's ability to remain operational
during component failures. Finally, operations involve
deploying, testing, and maintaining the system.

[Link]
How does the 'two generals' problem illustrate the
challenges of coordination in distributed systems?
Answer:The 'two generals' problem highlights that it is
impossible for two parties to reach an absolute agreement
due to unreliable communication channels. Even with
multiple messages sent back and forth, neither general can be
sure that the other will agree to the attack time, as message
failures (like being captured) introduce uncertainty. This
reflects the complexities and limitations of achieving reliable
coordination in a distributed system.

[Link]
What is the significance of scalability in distributed
systems?
Answer:Scalability is crucial because it determines a system's
ability to handle increasing workloads efficiently. It is
measured in throughput (operations per second) and response
time. If a system cannot scale adequately, it will reach its
capacity, leading to failures or poor performance. Engineers
can scale systems by either increasing hardware capabilities
('scaling up') or adding more machines ('scaling out').
Understanding scalability patterns is essential for designing
robust systems.

[Link]
What does it mean for a distributed system to be
resilient?
Answer:Resilience in a distributed system means that it can
continue to function effectively despite failures of one or
more components. This is vital because as systems grow, the
likelihood of component failures increases. Resilient systems
incorporate techniques such as redundancy and self-healing
mechanisms to mitigate the impact of these failures, thus
maintaining availability and reliability.

[Link]
Why is operations management becoming integrated with
development in distributed systems?
Answer:Operations management is increasingly integrated
with development due to practices like DevOps, where the
same team that builds a system also monitors and maintains
it. This approach helps identify weaknesses in the system
through direct experience. Continuous deployment ensures
updates are rolled out safely without compromising
availability, while observability tools help teams understand
system behaviors and respond quickly to issues.

[Link]
How does the architecture of a distributed system impact
its functionality?
Answer:The architecture of a distributed system determines
how its components are organized and interact with each
other. Different architectural perspectives (physical
machines, software processes, and services) influence how
effectively a system can operate, scale, and maintain
communication. A well-designed architecture allows for
independent deployment of services, optimizes
communication through interfaces, and enhances overall
system resilience and performance.

[Link]
What role do adapters play in a distributed system?
Answer:Adapters act as intermediaries connecting different
parts of a distributed system by handling communication
between clients and services. Inbound adapters manage
incoming requests to a service's API, while outbound
adapters facilitate communication with external services like
databases. This separation helps streamline interactions and
maintain clarity in how components exchange data.
Chapter 3 | 2 Reliable links| Q&A
[Link]
What is the primary purpose of TCP and how does it
achieve reliability?
Answer:TCP (Transmission Control Protocol)
serves the purpose of providing a reliable
communication channel over the Internet Protocol
(IP). It achieves reliability by partitioning a byte
stream into numbered packets (segments), ensuring
they arrive in order, without gaps or duplication.
Each segment sent requires an acknowledgment
from the receiver, and if an acknowledgment is not
received in a certain timeframe, the packet is
retransmitted. Furthermore, TCP uses checksums to
validate the integrity of the data, thereby preventing
corruption during transmission.

[Link]
Can you explain the connection lifecycle in TCP?
Answer:The connection lifecycle in TCP consists of three
main states: opening, established, and closing. The
connection is first opened through a three-way handshake:
the sender sends a SYN segment to initiate the connection,
the receiver responds with a SYN/ACK segment, and finally,
the sender acknowledges by sending an ACK segment along
with application data. Once data transmission is complete,
the connection transitions to the closing state, where
resources on both ends are released, involving multiple
round-trips to ensure termination.

[Link]
What role does flow control play in TCP communication?
Answer:Flow control in TCP is a mechanism designed to
prevent the sender from overwhelming the receiver's
capacity. The receiver communicates the size of its available
buffer when acknowledging segments, which informs the
sender of how much data can be sent without exceeding the
buffer's limit. This process is similar to rate limiting but
operates on the connection level rather than the service level,
ensuring a balanced data flow that matches the receiver's
processing ability.

[Link]
How does TCP manage congestion control, and what
happens when a segment is lost?
Answer:TCP manages congestion control by monitoring the
available bandwidth and adjusting the size of the congestion
window, which limits the number of outstanding segments
that can be sent. When a segment is acknowledged, the
window size increases, allowing more data to be in-flight.
However, if a segment is lost and the sender detects a
timeout, congestion avoidance mechanisms reduce the
congestion window size to prevent flooding the network.
This adaptive approach helps maintain overall network
stability.

[Link]
What advantages does UDP offer compared to TCP, and
in what scenarios is it preferred?
Answer:UDP (User Datagram Protocol) offers several
advantages over TCP, mainly its simplicity and low
overhead. Unlike TCP, UDP does not guarantee delivery,
order, or packet integrity, which results in lower latency and
higher bandwidth utilization. This makes UDP preferable for
applications requiring real-time performance, such as online
gaming or live video streaming, where timely delivery of
data is more critical than ensuring every packet arrives
correctly.
Chapter 4 | 3 Secure links| Q&A
[Link]
What role does TLS play in secure communication in
distributed systems?
Answer:TLS (Transport Layer Security) is crucial in
ensuring secure communication over the network by
encrypting the data transmitted between clients and
servers, providing authentication to verify the
identities of the communicating parties, and
maintaining the integrity of the messages to prevent
tampering.

[Link]
How does TLS use encryption to secure data, and why is
symmetric encryption preferred over asymmetric
encryption after establishing a connection?
Answer:TLS employs asymmetric encryption during the
initial handshake to negotiate a shared secret between the
client and server, ensuring secure key exchange without
transmitting the secret itself. After the shared secret is
established, symmetric encryption is used for the bulk of the
data transfer due to its speed and efficiency over asymmetric
encryption, which is slower and resource-intensive.

[Link]
What is the significance of digital signatures in TLS, and
how do they help in authentication?
Answer:Digital signatures in TLS are vital for authentication
as they allow the client to verify the server's identity. The
server signs its messages using its private key, and the client
uses the corresponding public key to confirm the signature's
validity, ensuring that the server is indeed who it claims to
be.

[Link]
Why are certificates important in the TLS protocol, and
what is the role of a Certificate Authority (CA)?
Answer:Certificates are essential in the TLS protocol as they
validate the ownership of public keys, providing assurance to
clients about whom they are communicating with. A
Certificate Authority (CA) issues these certificates and
verifies their authenticity, thus forming a trusted chain that
establishes the credibility of the certificates presented by
servers.

[Link]
What potential issues can arise from an expired TLS
certificate, and how can they be mitigated?
Answer:An expired TLS certificate can prevent clients from
verifying the server's identity, resulting in failed connections
which may bring down services. To mitigate this risk, it's
crucial to implement automation for monitoring and
automatically renewing certificates before they expire.

[Link]
How does TLS ensure the integrity of transmitted data,
and what measures are taken to avoid data corruption?
Answer:TLS ensures data integrity through the use of
HMAC (Hash-based Message Authentication Code), which
verifies that the data has not been tampered with during
transmission. While TCP provides a checksum for error
detection, HMAC serves as an additional layer to catch
potential tampering and data corruption that TCP might miss.
[Link]
Can you explain the handshake process in the context of
TLS and its importance?
Answer:In the TLS handshake, the client and server agree on
encryption methods, generate shared secrets, and verify each
other's identities through certificates. This process is critical
as it establishes a secure communication channel and ensures
both parties are authenticated before data exchange begins.

[Link]
Why is it recommended to keep servers geographically
closer to clients in a distributed system using TLS?
Answer:Keeping servers geographically closer to clients
reduces the latency associated with establishing new TLS
connections, which can be resource-intensive. This practice
improves performance and enhances the efficiency of secure
communication.
Chapter 5 | 4 Discovery| Q&A
[Link]
How does the DNS resolution process work when you
enter a URL in a browser?
Answer:When you enter a URL, the browser first
checks its local cache for a saved IP address. If not
found, it queries a DNS resolver, usually hosted by
the Internet Service Provider (ISP). The resolver
checks its own cache, and if unsuccessful, sends a
request to a root name server, which identifies the
top-level domain's (TLD) server. Then, the resolver
asks the TLD server for the authoritative name
server of the domain, which finally provides the IP
address associated with the hostname.

[Link]
Why is the DNS resolution process described as iterative?
Answer:The DNS resolution is considered iterative because
the resolver makes a series of requests to different DNS
servers one after the other rather than receiving all the
necessary information at once. After the resolver queries one
server and receives a response, it uses that information to
query the next relevant server until it finds the authoritative
name server that holds the final answer.
[Link]
What role does caching play in the DNS resolution
process, and how does it improve efficiency?
Answer:Caching plays a crucial role in improving DNS
resolution efficiency by storing previously resolved domain
name-IP address mappings. Both the browser and DNS
resolvers cache these results, which can significantly reduce
lookup times. When a domain name is resolved, subsequent
requests can be answered more quickly using cached entries,
thereby alleviating the load on DNS servers and reducing
latency in accessing websites.

[Link]
How does the Time to Live (TTL) affect DNS records,
and what challenges does it introduce?
Answer:TTL indicates how long a DNS record stays valid in
the cache. A long TTL means clients might use outdated
records longer, while a short TTL increases server load and
response times as clients have to resolve frequently. The
challenge lies in balancing TTL; set it too long, and changes
may not propagate fast enough; too short, and server
performance and reliability may suffer if the name server
becomes unavailable.

[Link]
What potential risks does DNS pose when it comes to
system outages?
Answer:DNS can become a single point of failure; if the
DNS name server is down, clients cannot resolve addresses
and connect to services. This situation can lead to widespread
outages, as users depend on DNS to find the IP addresses of
services, and without it, access is completely blocked.
Ensuring high availability and resilience in DNS
infrastructure becomes critical to prevent these risks.

[Link]
Why is there a movement towards running DNS over
TLS, and what does it address?
Answer:There is a growing movement towards running DNS
over TLS (DoT) to enhance security by encrypting DNS
requests. This transition is aimed at preventing third parties
from intercepting or snooping on DNS traffic, which
currently transmits in clear text over the internet. By using
TLS, the integrity and privacy of the resolution process are
preserved, which is increasingly important in our
security-conscious digital landscape.
Chapter 6 | 5 APIs| Q&A
[Link]
What is the difference between direct and indirect
communication in distributed systems?
Answer:Direct communication occurs when a client
communicates directly with a service, requiring both
to be active for interaction. Indirect communication
involves a broker or intermediary, allowing for
more flexibility when one of the parties may not be
available.

[Link]
How does request-response communication work in the
context of distributed systems?
Answer:In request-response communication, a client sends a
request message to a service, which then replies with a
response message. This process is like a function call but
takes place across different processes over a network.

[Link]
What are the implications of synchronous vs
asynchronous communication?
Answer:Synchronous communication blocks the client until a
response is received, which can be inefficient. In contrast,
asynchronous communication allows the client to continue
processing and get notified when the response arrives,
improving resource utilization.

[Link]
What are the commonly used IPC technologies for
request-response interactions?
Answer:The most commonly used IPC technologies include
gRPC for internal service communications, and REST and
GraphQL for external APIs.

[Link]
Why is HTTP considered a stateless protocol?
Answer:HTTP is stateless because each request from a client
to a server must contain all the information needed to
understand and process the request, without context from
previous requests.

[Link]
How does HTTP 2 improve upon HTTP 1.1?
Answer:HTTP 2 uses a binary protocol that allows
multiplexing multiple concurrent requests over a single
connection, which enhances efficiency compared to the
serialized transaction model of HTTP 1.1.

[Link]
What are REST principles, and why are they important
in API design?
Answer:REST principles are a set of conventions for
structuring APIs to be stateless and resource-oriented,
facilitating scalability and maintainability. They guide how
resources are created, accessed, and managed.

[Link]
What is the role of status codes in HTTP responses?
Answer:Status codes in HTTP responses indicate whether a
request was successful or not, helping clients understand the
result of their request. They are categorized into ranges
indicating success, redirection, client errors, and server
errors.

[Link]
What is idempotency, and why is it critical in HTTP
requests?
Answer:Idempotency refers to the property that an operation
can be repeated multiple times without changing the result
beyond the initial application. It’s critical for safely retrying
requests that may have failed without a response.

[Link]
How can APIs evolve without breaking existing clients?
Answer:APIs can evolve by implementing versioning
through URL prefixes or headers, and by making changes in
a backward-compatible way whenever possible to minimize
disruption for clients.
Chapter 7 | 6 System models| Q&A
[Link]
What is the purpose of creating system models in
distributed systems?
Answer:System models serve to clearly define the
expected behavior of various elements in distributed
systems, like nodes, communication links, and
timing. By establishing these models, we can reason
about distributed systems while simplifying the
complex technologies that implement them.

[Link]
What are the three communication link models
mentioned, and how do they differ?
Answer:The three communication link models are: 1.
Fair-loss link model – messages may be lost or duplicated,
but a sender can eventually ensure delivery through
retransmission. 2. Reliable link model – messages are
delivered exactly once without loss or duplication. 3.
Authenticated reliable link model – similar to the reliable
link, but allows the receiver to verify the sender's identity,
enhancing security.

[Link]
Explain the differences in node failure models and their
implications on system design.
Answer:The node failure models are: 1. Arbitrary-fault
model (Byzantine) – a node can behave unpredictably,
causing crashes or unintended behavior; critical for
safety-critical applications. 2. Crash-recovery model – a node
may crash but will always return to its last known state when
it restarts. 3. Crash-stop model – a node that crashes does not
return. These differences impact how algorithms are
designed, particularly in how they handle unreliable nodes.

[Link]
Why is the asynchronous model challenging in distributed
systems?
Answer:The asynchronous model is challenging because it
allows for unbounded time delays in message sending or
operations, which can lead to scenarios where algorithms
cannot progress or reach a consensus, effectively getting
'stuck'.

[Link]
What does the partially synchronous model offer that is
beneficial for practical systems?
Answer:The partially synchronous model represents a
compromise, allowing for synchronous behavior most of the
time while accommodating occasional asynchronous delays.
This aligns with real-world conditions better than fully
synchronous or asynchronous models.

[Link]
How does the text suggest readers approach the
assumptions made in system models?
Answer:Readers are encouraged to critically evaluate the
assumptions of the models presented and consider how these
assumptions might fail or 'leak' in real scenarios, potentially
leading to algorithm breakdowns.

[Link]
What can we infer about the algorithms discussed in the
book based on the assumed system model?
Answer:The algorithms explored in the book primarily
assume fair-loss communication links, crash-recovery node
behavior, and partial synchrony. As such, they are likely
tailored to work effectively under these conditions, with
consideration for how they would mitigate issues like
message loss and node failures.
Chapter 8 | 7 Failure detection| Q&A
[Link]
What are the challenges associated with failure detection
in distributed systems?
Answer:The main challenges in failure detection
include determining whether a server is slow,
crashed, or if there is a network issue preventing
message delivery. Without a timely response, clients
cannot accurately assess the server's status, risking
unnecessary timeout errors if the timeout period is
not properly configured.

[Link]
How can clients manage the uncertainty in server
availability?
Answer:Clients can manage uncertainty by implementing a
timeout mechanism to trigger if no response is received
within a set time frame. However, this requires careful
consideration to avoid false positives (considering a server
dead when it is just slow) and false negatives (waiting too
long when the server is unreachable).

[Link]
What are pings and heartbeats, and how do they improve
failure detection?
Answer:Pings are periodic requests sent by a process to
check if another process is still available. If a response is not
received within a specific time frame, the process is marked
as dead. Heartbeats are similar, where a process sends
messages to confirm its operational status. If the recipient
does not receive a heartbeat, it marks the sender as dead.
Both mechanisms help quickly identify failures in frequently
interacting processes.

[Link]
Why is it impossible to build a perfect failure detector?
Answer:It's impossible to create a perfect failure detector
because there's inherent uncertainty in network
communication. Timing issues, transient failures, and the
variability in server response times complicate the detection
process, leading to either premature failure detection or
delayed notifications.

[Link]
How can the concepts of pings and heartbeats be applied
in real-world scenarios?
Answer:In real-world applications, pings and heartbeats can
be crucial for systems with critical uptime requirements, such
as financial transactions or real-time data processing systems.
For example, a heart-monitoring application may regularly
send heartbeats to ensure the system is still operational,
immediately recovering from any failures to maintain service
continuity.

[Link]
What considerations should be taken into account when
determining the timeout period for failure detection?
Answer:Determining the timeout period requires a balance
based on expected response times, network conditions, and
system requirements. Too short a timeout may flag a
functioning server as down, while too long a timeout could
result in inefficiencies and delayed error responses.

[Link]
In what situations is it sufficient to detect failures only
during communication time?
Answer:Detecting failures only at communication time is
sufficient in systems where interactions are not frequent or
critical. For instance, occasional data syncs between
distributed databases might not necessitate constant health
checks, as occasional delays can be tolerated without
immediate repercussions.

[Link]
How do pings and heartbeats contribute to systems with
high availability demands?
Answer:Pings and heartbeats contribute to high availability
by ensuring that processes can quickly adapt to changes in
status, allowing for timely failure recovery. This minimizes
downtime and maintains continuous service, which is
particularly vital for applications requiring constant
accessibility.
Chapter 9 | 8 Time| Q&A
[Link]
Why is time an essential concept in distributed systems
compared to single-threaded applications?
Answer:In single-threaded applications, operations
execute sequentially, making the flow of execution
easy to follow. In contrast, distributed systems lack
a shared global clock, leading to challenges in
ordering operations since processes can run
concurrently. This necessitates timestamps to
reconstruct the order of operations, making time
critical for ensuring correctness and consistency.

[Link]
What are the key differences between physical clocks and
logical clocks in distributed systems?
Answer:Physical clocks measure real-world time but can
suffer from issues like clock drift and skew, impacting the
ordering of operations. Logical clocks, on the other hand,
measure time based on the sequence of operations rather than
real-world time, allowing for the establishment of a causal
relationship between operations using constructs like
counters and arrays.

[Link]
How does a Lamport clock work, and what limitations
does it have?
Answer:A Lamport clock utilizes a numerical counter that
increments before each operation and updates based on
message passing between processes. While it guarantees that
if one operation happened-before another, the timestamps
will reflect this order, it does not guarantee the converse.
That is, if operation O1's timestamp is less than O2's, it
doesn’t necessarily mean O1 happened before O2, which can
lead to ambiguities in concurrent operations.

[Link]
What is the significance of vector clocks compared to
Lamport clocks?
Answer:Vector clocks enhance the concept of logical clocks
by providing a mechanism to not only capture the
happened-before relationship but also to support a partial
ordering of events. This means they can accurately determine
when two operations are concurrent and when one precedes
another, thus resolving some of the ambiguities inherent in
Lamport clocks.

[Link]
Can physical clocks ever be sufficient for time-stamping
in distributed systems?
Answer:Yes, physical clocks can be sufficient for
time-stamping in certain contexts, such as logging for
debugging purposes where perfect ordering isn't crucial.
However, for the accurate ordering of events across
distributed systems, logical clocks are generally preferred
due to their ability to capture causal relationships between
events.
[Link]
What practical applications of logical clocks might we
encounter later in the book?
Answer:While the content becomes more abstract, logical
clocks are widely used in many distributed systems,
including databases for transaction management, distributed
file systems for consistency, and coordination protocols.
Recognizing and understanding these applications will
highlight the importance of logical time in maintaining
system integrity.
Chapter 10 | 9 Leader election| Q&A
[Link]
What are the key properties needed for an effective leader
election algorithm?
Answer:An effective leader election algorithm must
guarantee two essential properties: safety and
liveness. Safety ensures that there is at most one
leader at any given time, preventing conflicts and
chaos within the system. Liveness guarantees that an
election will eventually complete, ensuring that a
leader is elected, allowing the system to continue
functioning.

[Link]
How does the Raft leader election algorithm start the
election process?
Answer:The Raft leader election algorithm initiates the
election process when a follower process does not receive a
periodic heartbeat from the leader within a specified timeout
period. Upon this timeout, the follower presumes the leader
is dead, increments the current election term, transitions to
the candidate state, votes for itself, and requests votes from
other processes to become the new leader.

[Link]
What scenarios can occur after a candidate starts an
election in Raft?
Answer:Three scenarios can unfold after a candidate starts an
election in Raft: (1) The candidate wins the election by
securing a majority of votes and transitions to the leader
state. (2) Another process, claiming to be the leader, sends a
heartbeat with a term greater than or equal to the candidate’s
term, leading the candidate to accept this new leader and
revert to the follower state. (3) A situation of no winner
arises, potentially due to split votes among candidates, at
which point the candidates will time out and initiate a new
election.

[Link]
How can leader election adaptation improve system
performance and reliability?
Answer:Leader election adaptation can enhance system
performance and reliability by distributing workloads
effectively and introducing redundancy. For instance, rather
than relying on a single leader, systems can implement
partitioning to have different leaders per partition, which
alleviates the single point of failure risk and can handle an
increased load. However, this adds complexity and
necessitates careful design to maintain consistency across
partitions.

[Link]
What complexities arise from using a leader in a
distributed system?
Answer:Using a leader introduces complexities such as
creating a potential bottleneck for operations, where the
leader might not be able to keep up due to high demand.
Furthermore, it presents a single point of failure risk—if the
leader fails or the election process fails, the entire system
could be compromised. It's essential to design around these
issues; for example, by using optimistic locking as an
alternative to reduce dependence on a single leader.

[Link]
What is a fencing token and why is it important in leader
election scenarios?
Answer:A fencing token is a logical clock that increases each
time a distributed lock is acquired. It serves to ensure that
only the current leader's requests are accepted by the
downstream data store, preventing race conditions that may
arise when multiple leaders perform operations
simultaneously. Employing fencing tokens adds necessary
checks and balances to maintain data integrity across
concurrent operations.

[Link]
What should one consider before implementing leader
election in a system?
Answer:Before implementing leader election, it is crucial to
assess whether the desired functionality can be achieved
through alternatives. Options such as optimistic locking
might suffice, minimizing computational overhead.
Additionally, the need for high availability must be
evaluated, as sometimes using a single process that may
occasionally fail may be acceptable. If leader election is
necessary, work performed by the leader should be
minimized to reduce risks.

[Link]
What happens when multiple candidates try to become
the leader simultaneously in Raft?
Answer:If multiple candidates attempt to become the leader
simultaneously, a split vote can occur. In such cases, none of
the candidates may receive a majority of votes, causing each
of them to time out eventually. This timeout leads to starting
new elections, ideally spaced to reduce the likelihood of
another split vote occurrence and increase the chances of
electing a leader.
Chapter 11 | 10 Replication| Q&A
[Link]
Why is data replication essential in distributed systems?
Answer:Data replication increases availability and
facilitates scalability. If data is stored on only one
node and that node fails, the data is lost. However,
with replication, clients can access other copies
seamlessly. Additionally, replicating data enables
multiple clients to access it concurrently, improving
performance.

[Link]
What role does the leader play in Raft’s replication
algorithm?
Answer:In Raft’s replication algorithm, the leader is the
primary process responsible for making changes to the
replicated state. It sequentially logs operations that alter its
state and communicates these to followers, ensuring a
consistent state across all replicas.

[Link]
How does Raft ensure consistency among replicas?
Answer:Raft ensures consistency by requiring a majority of
followers to acknowledge the replication of each log entry
before the leader considers it committed. This quorum-based
approach prevents stale data from being accepted and
guarantees that all committed entries will eventually be
executed by all followers.

[Link]
What happens if a follower is temporarily unavailable in
Raft?
Answer:If a follower is temporarily unavailable, the leader
will continue trying to send AppendEntries requests until
they are successfully appended to the follower's log. The Raft
protocol ensures that followers can recover without
introducing inconsistencies by maintaining the integrity of
the log.

[Link]
Can followers serve read requests in Raft? If so, what is
the trade-off?
Answer:Yes, followers can serve read requests, which helps
increase read throughput. However, this can lead to different
clients observing different states of the system, as followers
may lag behind the leader. This introduces a consistency
trade-off, as clients may see stale data.

[Link]
What are the definitions of strong consistency, sequential
consistency, and eventual consistency?
Answer:Strong consistency ensures clients see a single
coherent view of the data as if it were served from a single
copy. Sequential consistency guarantees that operations
appear in the same order for all observers but lacks real-time
guarantees. Eventual consistency means that while clients
can see stale data, all replicas will converge to the same state
eventually.

[Link]
What is the CAP theorem, and how does it relate to
consistency and availability in distributed systems?
Answer:The CAP theorem states that a distributed system
can achieve only two out of the three guarantees of
consistency, availability, and partition tolerance. It
emphasizes the trade-offs that must be made during network
partitions, usually between consistency (ensuring all copies
of data reflect the same state) and availability (ensuring the
system remains responsive).

[Link]
How does the PACELC theorem expand upon the CAP
theorem?
Answer:The PACELC theorem states that, in the presence of
a network partition (P), a choice must be made between
availability (A) and consistency (C). However, even when
there is no partition (E), a trade-off between latency (L) and
consistency (C) must still be made. Thus, the theorem
highlights the continuous balancing act between latency and
consistency.

[Link]
What practical considerations should developers keep in
mind when choosing a distributed data store?
Answer:Developers must understand the trade-offs between
performance and consistency guarantees offered by
distributed data stores. Some stores provide adjustable
consistency levels, allowing developers to tailor performance
to application needs while being aware of the implications of
their choices.

[Link]
Why might not all applications require strong
consistency?
Answer:Some applications, like tracking website visits, may
tolerate eventual consistency without issues since slightly
outdated data does not impact functionality. In contrast,
applications that handle sensitive transactions or data
integrity, like payment processing, necessitate strong
consistency to avoid errors.
Chapter 12 | 11 Transactions| Q&A
[Link]
What is the primary purpose of transactions in
distributed systems?
Answer:Transactions provide the illusion of
exclusive access to data while ensuring that a group
of operations either all complete successfully or none
do, thus maintaining data integrity.
[Link]
What are the main properties of ACID in transactions?
Answer:The main properties of ACID are Atomicity (either
all operations succeed or none do), Consistency
(application-level invariants must always hold), Isolation
(concurrent transactions do not interfere), and Durability
(completed transactions remain persistent despite failures).

[Link]
Why is isolation critical in transaction management, and
what types of race conditions can it guard against?
Answer:Isolation is critical to prevent race conditions such as
dirty writes (overwriting uncommitted data), dirty reads
(reading uncommitted changes), fuzzy reads (seeing different
values on multiple reads), and phantom reads (transactions
affecting the read results). It protects the integrity of
concurrent operations.

[Link]
What are the differences between pessimistic and
optimistic concurrency control?
Answer:Pessimistic concurrency control uses locks to
prevent access to data by other transactions, ensuring safety
at the cost of performance. Optimistic concurrency control
allows transactions to proceed without locks and checks for
conflicts only at commit time, making it preferable for
read-heavy workloads.

[Link]
How does the Two-Phase Commit (2PC) protocol
facilitate atomic transaction commits across multiple
systems?
Answer:2PC orchestrates a commit across multiple
participants in two phases: 'prepare' where each participant
indicates readiness, and 'commit' where a final decision is
made based on all responses. This ensures that all or none of
the changes are applied.

[Link]
What challenges does the 2PC protocol present in
real-world scenarios?
Answer:2PC can be slow due to multiple round trips and can
lead to blocking if a participant fails, as all transactions stop
until the failure is resolved, risking the overall system's
responsiveness.

[Link]
What is the Saga pattern, and how does it ensure
atomicity in distributed transactions?
Answer:The Saga pattern is a series of local transactions
where each has a compensating transaction to undo its work
in case of failure. It coordinates between local services to
ensure that all transactions either succeed or the previous
ones are undone.

[Link]
How can you implement asynchronous transactions
without sacrificing atomicity?
Answer:By using a message log that appends transaction
messages atomically, different services can process messages
asynchronously and maintain eventual consistency without
blocking the system if one component is temporarily
unavailable.

[Link]
Why is idempotence important in message processing
within distributed systems?
Answer:Idempotence ensures that processing the same
message multiple times does not change the outcome,
allowing systems to recover from failures without
introducing inconsistencies.

[Link]
What strategies exist for managing isolation in
asynchronous transactions?
Answer:One strategy is using semantic locks with dirty flags
that indicate which data records are being modified by a
transaction, allowing other transactions to either roll back or
wait, while also requiring deadlock mitigation strategies.
Chapter 13 | 12 Functional decomposition| Q&A
[Link]
What are the primary advantages of adopting a
microservices architecture over a traditional monolithic
architecture?
Answer:1. **Increased Development Speed**:
Smaller teams can manage their own services,
resulting in faster development cycles due to
reduced communication overhead. 2.
**Scalability**: Each service can be scaled
independently based on its load and requirements. 3.
**Technological Diversity**: Teams can choose the
best technology stacks for their specific service
needs, facilitating experimentation and innovation.
4. **Simplified Code Management**: Smaller
codebases are easier to understand and manage,
allowing for quicker onboarding of new team
members. 5. **Enhanced Fault Isolation**:
Problems in one service are less likely to affect
others, increasing overall system reliability.

[Link]
What are the potential drawbacks of using a
microservices architecture?
Answer:1. **Increased Complexity**: More services mean
more moving parts, making the system harder to understand
and manage. 2. **Communication Overhead**: Remote calls
between services can lead to latency and require complex
handling of failures and retries. 3. **Operational
Overhead**: Each microservice may require its own
monitoring and maintenance, increasing resource needs. 4.
**Testing Difficulties**: Integrating tests across multiple
services is significantly more challenging than testing a
monolithic application. 5. **Eventual Consistency**:
Maintaining data consistency across services becomes
complicated and often requires accepting eventual
consistency.

[Link]
How does an API gateway benefit a microservices
architecture?
Answer:The API gateway acts as a mediator that simplifies
client interactions with multiple microservices. It centralizes
routing, facilitates composition of responses from multiple
services, and provides a single point for implementing
cross-cutting concerns such as authentication, logging, and
rate limiting. This reduces the complexity for clients, as they
do not need to keep track of multiple service endpoints and
can work with a unified API.

[Link]
What are some considerations when using an API
gateway?
Answer:1. **Potential Bottleneck**: It can become a single
point of failure, leading to performance issues if it cannot
handle the request load. 2. **Maintenance Overhead**:
Requires ongoing development and insight to ensure that it
aligns with the services behind it. 3. **Version
Management**: Anytime a service API changes, the gateway
must be updated. This can lead to version management issues
if not properly handled.
[Link]
What is the Command Query Responsibility Segregation
(CQRS) pattern, and why would you implement it?
Answer:CQRS separates the read and write operations into
different models, each optimized for its purpose.
Implementing CQRS can improve performance and
scalability when dealing with high read loads or complex
read queries that differ significantly from write operations,
allowing the use of specialized storage solutions for each
path. It provides a clear separation of concerns and allows for
independent scaling.

[Link]
What role does messaging play in microservice
communication?
Answer:Messaging provides an asynchronous
communication style, allowing services to exchange
information without requiring immediate responses. This
decouples the services and enhances reliability, as messages
can be queued and processed as resources allow, avoiding
issues from service unavailability or load spikes. It enables
flexible scaling and load balancing across services.

[Link]
How does one ensure message processing is reliable in a
messaging system?
Answer:To ensure reliability, implement at-least-once
delivery guarantees, utilizing visibility timeouts to reprocess
messages that were not successfully handled. Use dead-letter
queues for messages that repeatedly fail to process, enabling
debugging without disrupting the main flow. Idempotency in
message processing is crucial to avoid adverse effects from
duplicate messages.
Chapter 14 | 13 Partitioning| Q&A
[Link]
What is partitioning and why is it important in
distributed systems?
Answer:Partitioning, or sharding, is the process of
dividing a large dataset into smaller, manageable
segments that can be distributed across multiple
nodes. It is crucial for scaling applications
effectively, particularly when a dataset exceeds the
capacity of a single node, or when you want to
balance data loads to improve performance and
reliability. Without partitioning, systems may face
bottlenecks that lead to decreased efficiency and
increased latency.

[Link]
What are the two main strategies for sharding data in
distributed systems?
Answer:The two primary sharding strategies are range
partitioning and hash partitioning. Range partitioning divides
data based on key ranges in a lexicographical sequence,
allowing for efficient range scans. However, it can lead to
unbalanced partitions and hotspots. Hash partitioning, on the
other hand, uses a hash function to evenly distribute keys
across partitions, mitigating hotspots but potentially
complicating data sorting and management when partitions
change.
[Link]
What is stable hashing and how does it address the
downsides of traditional hash partitioning?
Answer:Stable hashing minimizes the data reshuffling
required when partition configurations change, such as when
a new partition is added. It does this by using a consistent
hashing technique where keys and partitions are randomly
distributed on a circle. When a new partition is introduced,
only a small fraction of keys need to be reassigned, thus
reducing network costs and maintaining performance.

[Link]
Why is rebalancing necessary in a partitioned data store,
and what are its challenges?
Answer:Rebalancing is essential when the load on a data
store grows, requiring more nodes, or when it shrinks,
allowing for fewer nodes. The challenges include ensuring
minimal disruption during the transition while avoiding
excessive data transfer. Poorly managed rebalancing can lead
to system bottlenecks, performance hits, or imbalanced
partitions that impede scalability.

[Link]
What are the differences between static and dynamic
partitioning?
Answer:Static partitioning involves defining a set number of
partitions from the beginning, which can lead to
inefficiencies if not configured correctly. Dynamic
partitioning, however, allows for the creation of partitions
on-demand, initiated either by growth of a partition or
reduction in size, offering greater flexibility and adaptability
to changing workloads.

[Link]
What practical considerations should be kept in mind
while implementing partitioning in a distributed system?
Answer:Implementing partitioning adds complexity to
system architecture. One must consider potential partition
imbalances, which can lead to performance bottlenecks.
Additionally, transactions that require updates across
multiple partitions must be handled atomically to maintain
consistency. The need for careful management of partition
sizes and distributions is critical to ensure optimal
performance.

[Link]
What further reading is suggested for those interested in
understanding more about partitioning in distributed
systems?
Answer:For readers looking to delve deeper into the nuances
of partitioning and distributed systems, 'Designing
Data-Intensive Applications' by Martin Kleppmann is highly
recommended. This book covers architecture patterns, data
models, and challenges in data management that are pertinent
to scaling distributed applications.
Chapter 15 | 14 Duplication| Q&A
[Link]
What is the primary purpose of load balancing in
distributed systems?
Answer:The primary purpose of load balancing is to
distribute incoming requests evenly across multiple
service instances to optimize resource use, minimize
response times, and ensure high availability by
preventing any single instance from becoming a
bottleneck or point of failure.

[Link]
How can duplication help achieve horizontal scalability in
applications?
Answer:Duplication helps achieve horizontal scalability by
allowing you to create multiple instances of a stateless
service. When more instances are deployed, the capacity of
the service effectively doubles or increases proportionally,
enabling it to handle a larger number of requests without
degrading performance.

[Link]
What issues might arise when adding more service
instances that depend on a shared data store?
Answer:When increasing the number of service instances
that rely on a shared data store, it can lead to a bottleneck as
the data store becomes overwhelmed with requests. This can
ultimately hinder performance, as the shared resource may
slow down or fail under the increased load from multiple
service instances.

[Link]
What is the effect of using delayed load metrics in load
balancing?
Answer:Using delayed load metrics to route requests can
create a herding effect, where a load balancer mistakenly
sends too many requests to a newly joined server that reports
low load. This can cause the server to become overwhelmed
and alternate between being busy and idle, leading to
inefficient request distribution.

[Link]
How does randomness help in load balancing when
dealing with delayed metrics?
Answer:By randomly selecting two servers from a pool and
routing the request to the one with the lesser load,
randomness helps prevent the herding effect caused by
delayed metrics. This method efficiently combines delayed
load information with randomness, leading to more balanced
request distribution and better server utilization.

[Link]
What are the key differences between active and passive
health checks in load balancing?
Answer:Active health checks involve the load balancer
sending requests to the servers to determine their health
status, typically through a dedicated health endpoint. In
contrast, passive health checks are based on the load
balancer's observation of the outcome of server requests. If a
server fails to respond or returns an error during normal
operations, it is marked as unhealthy.

[Link]
What are the trade-offs between asynchronous and
synchronous replication in data replication?
Answer:Asynchronous replication offers faster performance
since it allows the leader to respond to clients before
ensuring that all followers have received the update, but it
risks potential data loss. In contrast, synchronous replication
guarantees data consistency by waiting for updates to be
confirmed by all followers before responding, but it can
introduce significant latency and affect the system's
availability if a follower is slow.

[Link]
What are the advantages and disadvantages of using a
multi-leader replication strategy?
Answer:The advantages of multi-leader replication include
increased write throughput and availability across
geographically dispersed data centers. However, the
complexity increases due to the risk of conflicting writes
among leaders, requiring a robust conflict resolution strategy,
which can complicate the system's design and operations.

[Link]
How can caching improve distributed system
performance?
Answer:Caching improves performance by providing a
high-speed storage layer that stores frequently accessed data,
reducing the load on downstream dependencies and
minimizing latency when retrieving stored data, as future
requests can typically serve from the cache.

[Link]
What is the significance of choosing the right eviction
policy in caching mechanisms?
Answer:Choosing the right eviction policy is significant
because it determines which cached entries will be removed
when new entries need to be added. An effective policy, like
least recently used (LRU), helps maintain high cache hit rates
by retaining the most relevant and frequently accessed data,
thereby improving overall system performance.

[Link]
What role does a sidecar pattern play in load balancing
within microservices?
Answer:The sidecar pattern enhances load balancing in
microservices by colocating a proxy alongside each service
instance. This allows for decentralizing load balancing
functionality to the individual clients, enabling features like
rate limiting, authentication, and monitoring while mitigating
the need for a centralized load balancing service.
Chapter 16 | 15 Common failure causes| Q&A
[Link]
What are some common causes of failure in distributed
systems?
Answer:The common causes of failure in distributed
systems include single points of failure, unreliable
networks, slow processes, unexpected load, and
cascading failures.

[Link]
How can single points of failure be mitigated in a
distributed system?
Answer:Single points of failure can be mitigated by
introducing redundancy and carefully architecting the system
to eliminate components that could singularly fail, thus
ensuring that the system remains functional even if one part
fails.

[Link]
What challenges do unreliable networks present in
distributed systems?
Answer:Unreliable networks can cause clients to experience
long waits for responses, which can lead to timeouts and
errors. This unreliability can create difficult debugging
scenarios and degrade system performance.

[Link]
What is the impact of slow processes on system
performance?
Answer:Slow processes can halt productivity, similar to
processes that are not running at all, due to issues like
resource leaks that accumulate over time, ultimately leading
to system resource exhaustion.

[Link]
How can unexpected load affect a distributed system's
performance?
Answer:Unexpected load can strain a distributed system by
overwhelming it with requests beyond its capacity, which can
occur due to sudden spikes, leading to potential failure or
service degradation.

[Link]
What are cascading failures and how do they propagate
through systems?
Answer:Cascading failures occur when a small failed
component increases the load on another component, leading
it to fail as well. This can create a feedback loop that causes
widespread operational failures within the system.

[Link]
How can risk management be applied to prevent and
handle failures in distributed systems?
Answer:Risk management involves assessing the likelihood
and impact of potential failures, allowing engineers to
prioritize which issues to address actively. By calculating
risk scores, they can focus on failures that are both probable
and impactful.

[Link]
What role does redundancy play in designing robust
distributed systems?
Answer:Redundancy plays a critical role by ensuring that if
one component fails, other components can take over the
load, helping to maintain system availability and prevent
total failure.
[Link]
Can you explain the concept of a 'blast radius' when
discussing failures in distributed systems?
Answer:The blast radius refers to the extent of impact a
single component's failure can have on the entire system.
Reducing the blast radius means minimizing the harm caused
by any single point of failure.

[Link]
Why is it crucial to consider third-party library behavior
in distributed systems?
Answer:Third-party libraries may consume shared resources
without transparency. Understanding their behavior is
essential to prevent unexpected behaviors that could lead to
resource leaks and instability within your system.
Chapter 17 | 16 Downstream resiliency| Q&A
[Link]
Why is it crucial to set timeouts for network requests?
Answer:Setting timeouts is essential because it limits
and isolates failures, preventing them from
cascading through the system. Without timeouts, a
network request can hang indefinitely, leading to
resource exhaustion, such as socket pools being
exhausted, making it impossible to establish new
connections.

[Link]
What should a client do upon a failed request or a
timeout?
Answer:The client has two main options: fail fast, which
means aborting the request immediately, or implement a retry
mechanism, especially if the failure is likely transient.

[Link]
Explain the concept of exponential backoff and its
importance in retries.
Answer:Exponential backoff is a strategy to increase the wait
time between retries exponentially, capping at a defined
maximum. This approach reduces pressure on the
downstream service during high-load scenarios and spreads
out retry attempts to avoid 'retry storms' where multiple
clients overwhelm the service.
[Link]
What is 'retry amplification' and why is it problematic?
Answer:Retry amplification occurs when multiple services in
a dependency chain retry their requests simultaneously after
failures, creating a compounding load on the downstream
service. This can lead to a cascading failure, where the
increased number of requests overwhelms the service,
potentially bringing down the whole system.

[Link]
How does a circuit breaker pattern work?
Answer:The circuit breaker acts as a state machine with three
states: open, closed, and half-open. In the closed state, it
monitors failures. If failures exceed a threshold, it opens the
circuit, blocking all calls to the failing service. After some
time, it switches to half-open to check the service's status
with a limited call; success returns to the closed state, while
failure returns to open.

[Link]
When should retries be avoided?
Answer:Retries should not be attempted if the error is likely
non-transient, such as authorization issues, or if the operation
is non-idempotent, because retrying could lead to incorrect
outcomes, like double processing a payment.

[Link]
What is the significance of monitoring in the context of
network calls?
Answer:Monitoring is vital to measure the lifecycle of
network calls, including their duration and statuses like
timeouts. Without adequate monitoring, diagnosing
production issues becomes difficult, hindering a team's
ability to respond to and resolve failures effectively.

[Link]
Provide an example of how a circuit breaker pattern
leads to graceful degradation.
Answer:A practical example is an online retail website; if the
recommendation engine fails, the site could render without
recommendations instead of displaying an error or crashing
completely, ensuring a functional and user-friendly
experience for customers.

[Link]
What is the main takeaway regarding the implementation
of retries and circuit breakers?
Answer:The core principle is to utilize retries for handling
transient failures but employ circuit breakers for
non-transient failures, thereby ensuring that systems can
gracefully handle issues without overwhelming dependencies
and maintain overall system health.
Chapter 18 | 17 Upstream resiliency| Q&A
[Link]
What is load shedding and how does it work?
Answer:Load shedding is a mechanism used by
servers to protect themselves from becoming
overloaded by rejecting excess incoming requests
when their capacity is strained. Instead of
continuing to accept new requests that would
degrade performance, the server begins to return a
503 (Service Unavailable) status code for new
requests once it detects that it is operating at or
beyond its load limit, prioritizing the processing of
existing requests.

[Link]
How can load leveling help prevent a server from
becoming overwhelmed?
Answer:Load leveling introduces a messaging channel
between clients and services, allowing the service to pull
requests at its own pace rather than being pushed requests by
clients. This decouples the incoming load from the service's
processing capacity, helping to smooth out short-lived spikes
and reducing the risk of overwhelming the service.

[Link]
What role does rate-limiting play in managing incoming
requests to a service?
Answer:Rate-limiting, or throttling, prevents a service from
being overwhelmed by rejecting requests when a certain
quota is exceeded. This can be based on various criteria such
as the number of requests per user or API key over a specific
time period, and helps ensure that one client does not
monopolize resources.

[Link]
Explain the bulkhead pattern and its significance in
managing service reliability.
Answer:The bulkhead pattern isolates faults within a service
by partitioning shared resources. This means that if one part
of a service faces issues, it won't impact the entire system.
Like a ship’s bulkheads protecting against water intrusion,
this pattern ensures that heavy load or degrading requests
from one client do not affect others, improving overall
service reliability.

[Link]
Describe the importance of implementing health checks
within a distributed system.
Answer:Health checks are vital for maintaining high
availability in distributed systems. By allowing load
balancers to query service instances for health status, they
can dynamically route traffic away from unhealthy instances,
thus ensuring service continuity and reducing the risk of
downtime or degraded performance when issues arise.

[Link]
What is the concept of a watchdog in distributed systems
and how does it contribute to system resilience?
Answer:A watchdog is a monitoring mechanism within a
process that checks its health at regular intervals. If certain
predefined thresholds are crossed, indicating that the process
may be in a degraded state, the watchdog can trigger a restart
of the process. This self-healing property helps maintain
service performance and reliability, allowing operators to
address root-cause issues without enduring prolonged
degradation.

[Link]
How does the CAP theorem relate to managing service
availability during adverse conditions?
Answer:The CAP theorem posits that during network faults,
a system must choose between maintaining consistency or
availability. In practice, services often prioritize availability,
by continuing to serve requests based on the last known good
state rather than strictly ensuring consistency, to avoid total
service outages during issues like database unavailability.

[Link]
Why is combining rate-limiting with other patterns
beneficial for mitigating denial-of-service attacks?
Answer:Although rate-limiting alone cannot fully protect
against denial-of-service attacks, it can significantly reduce
their impact by controlling traffic and preventing any single
client from overwhelming the service. When combined with
other patterns like load shedding and bulkheading, it creates
a robust defense by combining multiple layers of protection
against excessive loads.

[Link]
In what way does load shedding differ from rate-limiting?
Answer:Load shedding is based on the local state of a server,
rejecting requests when it is overloaded. Rate-limiting,
however, governs traffic based on a broader control over
request quotas across all instances and specific users. While
both methods aim to maintain performance, they have
different triggers and implications for resource management.

[Link]
How can virtual partitions enhance the effectiveness of
the bulkhead pattern?
Answer:Virtual partitions create random subsets of service
instances, reducing the likelihood that a problematic user will
be allocated the same partition as another, thus enhancing
fault isolation. This means that degradation is confined to a
smaller subset of instances, allowing better resource
availability for unaffected users.
Chapter 19 | 18 Testing| Q&A
[Link]
Why is catching bugs early in development important?
Answer:Catching bugs early is crucial because it
reduces the cost and complexity of fixing them later.
Early detection helps ensure that developers can
refactor, add features, and maintain existing
functionality with confidence, ultimately leading to a
more stable and robust application.

[Link]
What are the different types of tests in distributed
systems?
Answer:Tests can be categorized into unit tests, integration
tests, and end-to-end tests. Unit tests focus on individual
components, integration tests check interactions between
components or services, and end-to-end tests validate
complete user scenarios across all services.

[Link]
How can testing improve system design?
Answer:Testing encourages developers to view the code
from the user's perspective, leading to improvements in
system design by ensuring functionality aligns with user
requirements and fostering better overall architecture.

[Link]
What tradeoffs should be considered when writing tests
for a system?
Answer:When writing tests, developers must balance the
scope and size of tests, aiming for smaller tests that are
faster, more reliable, and easier to maintain, while still
ensuring sufficient coverage of expected behaviors and
failure scenarios.

[Link]
Why are end-to-end tests considered 'necessary evils'?
Answer:End-to-end tests are necessary because they validate
user-facing scenarios across multiple services, but they can
be painful and costly to maintain due to their complexity and
tendency to fail intermittently.

[Link]
How can the reliability of tests be improved?
Answer:To improve test reliability, developers should prefer
smaller scoped tests, utilize test doubles wisely, and ensure
good fault isolation mechanisms are in place to prevent tests
from impacting other parts of the system.

[Link]
What strategies can be employed to minimize end-to-end
tests?
Answer:One strategy is to frame end-to-end tests as user
journey tests, which simulate multi-step user interactions in
one test rather than having many separate tests, thereby
reducing the total number of tests needed.

[Link]
Why might a developer choose to use a fake or mock in
testing?
Answer:Fakes and mocks allow developers to simulate
components and their interactions without incurring the
overhead or potential errors associated with real
implementations, allowing for faster tests with reduced
complexity.

[Link]
How does the concept of contract testing improve
integration testing?
Answer:Contract testing formalizes the expected interactions
between a service and its external dependencies by ensuring
both sides adhere to predefined contracts, which can help
catch integration issues early and improve overall reliability.

[Link]
What role does documentation play in testing?
Answer:Tests serve as up-to-date documentation by
providing clear examples of how components should behave,
which helps new developers understand the system and
ensures that functionality is preserved during changes.

[Link]
What is a good rule of thumb for balancing different
types of tests?
Answer:A good rule of thumb is to maintain a large number
of unit tests, a smaller fraction of integration tests, and an
even smaller number of end-to-end tests to ensure reliability
without sacrificing coverage.

[Link]
Why might a test using a real implementation be
preferable to one using a test double?
Answer:Using a real implementation often provides more
accurate and reliable results, as it behaves in a predictable
manner under normal conditions, while test doubles may fail
to accurately simulate all aspects of real interactions.

[Link]
What specific risks are associated with user data
management functions like GDPR compliance?
Answer:Failing to comply with regulations such as GDPR
can lead to significant legal and financial penalties, making it
critical to have robust testing for functionalities that manage
sensitive user data.

[Link]
How can developers ensure that their tests do not
interfere with each other?
Answer:Developers should implement good fault isolation
mechanisms, such as rate-limiting, to prevent any single test
from affecting others, ensuring that all tests can run
independently without unintended side effects.
Chapter 20 | 19 Continuous delivery and
deployment| Q&A
[Link]
Why is automating the deployment process essential for
efficiency in distributed systems?
Answer:Automating the deployment process ensures
that changes can be released frequently and reliably,
reducing the chances of batching changes which
complicates debugging. It allows developers to focus
on their tasks rather than on deploying releases,
thereby optimizing engineering resources.

[Link]
What role does a 'pull request' play in the continuous
delivery and deployment pipeline?
Answer:A pull request (PR) initiates the review process for
code changes. It must be validated and approved before being
merged into the repository, ensuring that only safe and
correct changes are released through the continuous
deployment pipeline.

[Link]
What strategies can be employed in the pre-production
stage to ensure the reliability of new releases?
Answer:During pre-production, multiple environments can
be utilized to run extensive tests. These environments should
mimic production conditions closely, using the same metrics
and health signals to ensure that potential issues are caught
before the final production rollout.

[Link]
How does the production rollout process mitigate risks
associated with deploying new artifacts?
Answer:The production rollout starts with deploying to a
small subset of instances to uncover any undiscovered issues
early on. It then progressively increases rollout to the rest of
the fleet while continuously monitoring health metrics,
allowing for quick responses to any detected problems.

[Link]
What is the importance of monitoring both upstream and
downstream services during a deployment?
Answer:Monitoring both upstream and downstream services
is crucial to identify any indirect negative impacts caused by
the deployment, ensuring that the overall system health is
maintained and preventing issues that may not be evident
when focusing only on the service being deployed.

[Link]
Why should changes be designed for backward
compatibility during a release?
Answer:Designing changes for backward compatibility
minimizes risks in the deployment pipeline as it allows for
safer rollbacks if an issue arises. It prevents breaking existing
integrations and ensures smoother transitions when changing
interfaces or formats.

[Link]
Can you explain how the 'bake time' is used within the
CD pipeline?
Answer:Bake time is the waiting period between deployment
stages, ensuring that sufficient time has passed to accurately
assess the health of the newly deployed artifact. This helps to
identify issues that may not surface immediately and allows
for proper monitoring before proceeding with the rollout.

[Link]
What is 'canary testing' and how does it contribute to the
deployment process?
Answer:Canary testing involves releasing new software to a
small, controlled subset of users or systems before a full
rollout. This method helps to identify potential problems
early on with minimal impact, allowing for adjustments
before broader deployment.

[Link]
How can teams prepare for backward-incompatible
changes in a deployment?
Answer:To safely introduce backward-incompatible changes,
teams should break them down into smaller, manageable
updates that can be rolled back independently if necessary.
Each stage of the change should ensure that systems can still
operate effectively with both old and new formats.

[Link]
What are some key indicators of a healthy deployment in
a CD pipeline?
Answer:Key indicators include successful results from
end-to-end tests, acceptable latency and error rates as
captured by health metrics, and the absence of alerts during
and after the deployment process.
Chapter 21 | 20 Monitoring| Q&A
[Link]
What is the primary purpose of monitoring in distributed
systems?
Answer:The primary purpose of monitoring in
distributed systems is to detect failures that impact
users in production and trigger notifications (alerts)
to human operators responsible for mitigation, while
also providing a high-level overview of system health
through dashboards.

[Link]
How has the approach to monitoring evolved over the
years?
Answer:Monitoring has evolved from a black-box approach,
which only reported whether a service was up or down, to a
white-box approach where developers instrument their code
to emit application-level measurements, allowing for deeper
insights into system performance.

[Link]
What are the differences between black-box and
white-box monitoring?
Answer:Black-box monitoring focuses on external
perceptions of system health and performance, often tracking
user experiences via scripts that test external services. In
contrast, white-box monitoring involves internal metrics that
help identify root causes of failures and system behavior,
making it easier to detect issues before they impact users.

[Link]
What are metrics in the context of monitoring, and how
are they categorized?
Answer:Metrics are numeric representations of information
measured over a time interval, typically organized as
time-series data. They can be categorized by dimensions
through tagging with key-value pairs, allowing detailed
analysis and granular data slicing for monitoring services.

[Link]
What is a Service Level Indicator (SLI) and how is it used
in monitoring?
Answer:A Service Level Indicator (SLI) is a metric that
measures a specific aspect of service quality, such as
response time or error rate. SLIs are used to assess service
performance, support alerting mechanisms, and inform
stakeholders about the level of service being provided.

[Link]
How should one determine which metrics to monitor for
alerting purposes?
Answer:Metrics chosen for alerting should focus on those
that accurately reflect user experience and system
performance. It is essential to select metrics that provide
actionable insights rather than overwhelming operators with
noise or irrelevant data.

[Link]
What role do Service Level Objectives (SLOs) play in
monitoring and service reliability?
Answer:Service Level Objectives (SLOs) define acceptable
performance ranges for SLIs, helping set user expectations
and guide teams on priority actions. They also establish error
budgets, which quantify acceptable levels of service
degradation and inform decision-making regarding feature
prioritization and incident management.

[Link]
What is the significance of alerting in a monitoring
system?
Answer:Alerting is crucial because it prompts immediate
responses to significant changes or issues within a system,
allowing teams to take action before problems escalate into
user-impacting incidents. Effective alerts must be actionable
to minimize downtime and enhance service resilience.

[Link]
How can the effectiveness of alerts be optimized in
monitoring systems?
Answer:The effectiveness of alerts can be optimized by
balancing precision and recall. Alerts should be based on
relevant SLIs and tuned to avoid false positives while
ensuring critical issues are recognized in a timely manner.
Implementing burn-rate alerts can help trigger proactive
responses to potential SLO violations.

[Link]
Why is it essential to have a structured on-call strategy in
monitoring?
Answer:A structured on-call strategy ensures that developers
and operators are engaged effectively in maintaining service
reliability. It enables prompt incident management, reduces
operational tolls, and emphasizes accountability, fostering a
culture of reliability and ownership over the systems being
monitored.
Chapter 22 | 21 Observability| Q&A
[Link]
Why is observability crucial for distributed systems?
Answer:Observability is essential for distributed
systems because it provides insights into the myriad
of emergent behaviors that can arise due to the
system's complexity. It aids operators in debugging
issues and validating hypotheses about system health
beyond mere monitoring. This understanding is
necessary as distributed systems can never be 100%
healthy at any given time; thus, observability tools
are vital for diagnosing and mitigating failures.

[Link]
How do metrics, logs, and traces differ in their roles
within a distributed system?
Answer:Metrics primarily focus on monitoring the system's
health by tracking performance over time, whereas logs
capture detailed, timestamped events for debugging
purposes. Traces provide a holistic view of a request's
journey through the system, showing causality and timing
across multiple services. Together, they create a
comprehensive observability strategy that allows for both
high-level monitoring and in-depth analysis.

[Link]
What are some best practices for logging in a distributed
system?
Answer:Best practices for logging include structuring logs as
key-value pairs for easier parsing, consolidating all relevant
data about a work unit into a single event to minimize the
need for data joins, and employing varying logging levels to
control verbosity. Additionally, sanitizing logs to remove
sensitive data, using asynchronous logging to avoid
performance hits, and implementing rate limiting on log
collectors are also recommended.

[Link]
How can organizations manage the costs associated with
logging?
Answer:Organizations can manage logging costs by
implementing dynamic logging levels, sampling logs to
capture only a subset of events, prioritizing important logs,
and rate-limiting log ingestion to prevent overwhelming log
collectors. Additionally, using metrics for high-level insights
instead of raw logs can reduce data volume while still
providing necessary operational oversight.

[Link]
What challenges exist when introducing tracing into an
existing distributed system?
Answer:Introducing tracing in an existing system is
challenging as it requires modification of every component in
the request path to propagate trace context. This includes
ensuring that not only custom components but also
third-party libraries and services support tracing. Retrofitting
tracing necessitates thorough planning to attain a seamless
implementation across all system parts.

[Link]
How do tracing and logging complement each other in
debugging?
Answer:Tracing and logging complement each other as
traces provide a high-level overview of the request flow and
its duration, while logs offer detailed accounts of events
within those flows, enabling engineers to pinpoint where an
error occurred. Together, they enhance the ability to debug
problems, giving both broader context and specific insights.

[Link]
What is the significance of having unique trace IDs in
distributed tracing?
Answer:Unique trace IDs are crucial in distributed tracing as
they allow the system to link related spans across various
services and components. This connection facilitates a clear
understanding of the entire request's lifecycle and helps trace
issues back to their origin, improving fault diagnosis and
performance monitoring in the distributed architecture.

[Link]
Why might traditional monitoring systems fall short
compared to observability solutions?
Answer:Traditional monitoring systems often focus only on
tracking symptoms of failures, such as downtime or
performance degradation, rather than enabling deeper
insights into the root causes of those issues. Observability
solutions go further by providing the necessary tools to
analyze and understand the behaviors and interactions within
complex distributed systems, making them more effective for
debugging and incident response.

[Link]
What is the ideal outcome of implementing a
comprehensive observability strategy?
Answer:The ideal outcome of a comprehensive observability
strategy is to significantly reduce the time it takes to identify
and resolve incidents, enhance the ability to diagnose issues
quickly, and improve system performance and reliability.
This enables organizations to maintain higher service levels
and better meet user expectations in a dynamic, distributed
environment.
Chapter 23 | 22 Final words| Q&A
[Link]
What are the key takeaways from the book
'Understanding Distributed Systems'?
Answer:The key takeaways include the
understanding of fundamental concepts in
distributed systems, the importance of consistency
models, and how large-scale systems are composed.
The emphasis is on learning from existing literature
and understanding practical applications in
real-world systems.

[Link]
How can one effectively continue learning about
distributed systems after finishing the book?
Answer:One can continue learning by reading industry
papers, starting with critical works such as 'Windows Azure
Storage: A Highly Available Cloud Storage Service with
Strong Consistency' which dives into Azure's storage system,
followed by 'Azure Data Explorer' for insights into big data
analytics. Engaging with resources like Alex Xu's 'System
Design Interview' can also prepare you for practical
applications in system design.

[Link]
Why is strong consistency significant in distributed
systems?
Answer:Strong consistency simplifies the developer's job by
ensuring that once a write is completed, all subsequent reads
reflect that write immediately. This leads to predictable
behavior across distributed components, making it easier to
reason about system performance and reliability.

[Link]
What is the recommended starting point for exploring
cloud storage systems?
Answer:The recommended starting point is the paper on
'Windows Azure Storage', which explains Azure's storage
architecture and highlights the importance of strong
consistency as a design goal that benefits developers.

[Link]
What insight does the Azure Data Explorer paper
provide?
Answer:The Azure Data Explorer paper discusses a
cloud-native event store built on Azure's storage, illustrating
how disparate large-scale systems can integrate seamlessly,
which is crucial for designing effective cloud-based
solutions.

[Link]
What is the significance of standing on the shoulders of
giants in learning system design?
Answer:This metaphor highlights the value of building on
existing knowledge and innovations. By studying the work of
experts and established systems, one gains a deeper
understanding and can apply learned concepts to new
challenges in system design.

[Link]
How does the author suggest one prepares for system
design interviews?
Answer:The author suggests studying Alex Xu's 'System
Design Interview' book, which equips individuals with
frameworks and case studies to effectively tackle design
problems in interviews.
Understanding Distributed Systems Quiz
and Test
Check the Correct Answer on Bookey Website

Chapter 1 | Understanding Distributed Systems|

Quiz and Test
[Link] demand for expertise in distributed systems is
highlighted by the Stack Overflow’s 2020
developer survey, indicating that top-paying
engineering roles require such skills.
[Link] author of the book aims to provide an exhaustive
resource that covers every minute detail of distributed
systems without any gaps.
[Link] of data consistency models is important for
reducing debugging time and architectural redesigns in
distributed systems.
Chapter 2 | 1 Introduction| Quiz and Test
1.A distributed system can consist of both physical
machines and software processes that collaborate
to achieve tasks.
[Link] availability in a distributed system means that each
node must operate independently without any shared data
to prevent data loss.
[Link] in a distributed system can only be achieved by
upgrading existing hardware without adding new machines.
Chapter 3 | 2 Reliable links| Quiz and Test
[Link] (Transmission Control Protocol) ensures
ordered delivery of byte streams without gaps,
duplication, or corruption.
[Link] TCP connection lifecycle includes a five-way
handshake process to establish a connection.
[Link] (User Datagram Protocol) is a reliable protocol that
guarantees data delivery by using acknowledgments.
Chapter 4 | 3 Secure links| Quiz and Test
[Link] Layer Security (TLS) runs on top of
UDP to secure transmissions.
[Link] ensures message integrity by using a HMAC
(Hash-based Message Authentication Code).
[Link] client's verification of the server's certificate during the
TLS handshake is crucial for ensuring the authenticity of
the server.
Chapter 5 | 4 Discovery| Quiz and Test
[Link] DNS resolution process includes a step where
the browser checks its local cache for the hostname
before contacting the DNS resolver.
[Link] a DNS name server fails, it does not affect client access
to services.
[Link] time to live (TTL) for DNS records determines how
long a hostname-to-IP mapping is considered valid and is
crucial for optimizing DNS performance.
Chapter 6 | 5 APIs| Quiz and Test
[Link] act as adapters translating messages from
 IPC mechanisms to service interface calls.
[Link] 2 allows for sequential handling of requests, thus
improving efficiency compared to HTTP 1.1.
[Link] idempotency in request methods is
important for safely retrying requests.
Chapter 7 | 6 System models| Quiz and Test
[Link] Fair-Loss Link Model guarantees that
messages are delivered exactly once without loss or
duplication.
[Link] Crash-Recovery Model assumes that nodes do not
deviate from their logic but can crash and restart, losing
in-memory state.
[Link] Synchronous Model is a practical assumption used in
distributed systems as it accurately reflects real-world
message sending and operation execution timing.
Chapter 8 | 7 Failure detection| Quiz and Test
[Link] failure detection is attainable in distributed
systems.
[Link] can configure a timeout to handle server response
uncertainties effectively.
[Link] can be used to indicate that a process is
operational in distributed systems.
Chapter 9 | 8 Time| Quiz and Test
[Link] clocks in distributed systems are always
 accurate due to synchronization methods.
[Link] clocks allow for a partial ordering of operations in
distributed systems.
[Link] clocks, such as Lamport clocks, guarantee the
correct ordering of all operations across multiple processes.
Chapter 10 | 9 Leader election| Quiz and Test
[Link] distributed systems, leader election is only
necessary if multiple processes require
coordination for resource management.
[Link] Raft, a candidate wins the election only if it receives
votes from all processes in the system, not just the
majority.
[Link] is possible to have multiple leaders in a distributed
system without proper implementations of leader election
protocols like Raft.
Chapter 11 | 10 Replication| Quiz and Test
[Link] of data in distributed systems
primarily enhances availability, scalability, and
performance.
[Link] Raft algorithm allows all nodes, not just the leader, to
make changes to the data in a distributed system.
[Link] CAP theorem discusses the trade-off between
consistency and availability in the presence of network
partitions.
Chapter 12 | 11 Transactions| Quiz and Test
[Link] can be implemented easily across
multiple services with different datastores.
[Link] stands for Atomicity, Consistency, Isolation, and
Durability in the context of transactions.
[Link] allow for atomicity in distributed transactions
without blocking, unlike 2-phase commit.
Chapter 13 | 12 Functional decomposition| Quiz and
Test
[Link] from a monolith to a microservice
architecture can alleviate productivity issues
associated with codebase complexity.
2.A key benefit of microservices is that it allows for greater
operational overhead due to varied technology stacks.
[Link] gateways are only used for enhancing security in
microservice architectures and have no other significant
functions.
Chapter 14 | 13 Partitioning| Quiz and Test
[Link] in distributed systems is also known
as sharding.
[Link] partitioning allows for easy alteration of the number
of partitions after initialization.
[Link] partitioning starts with multiple partitions and
does not split until none are needed.
Chapter 15 | 14 Duplication| Quiz and Test
[Link] balancing improves the capacity and
 availability of a service by distributing requests
across multiple instances.
[Link] checks are only active checks and do not include
passive checks for monitoring server statuses.
[Link] replication allows any replica to handle writes,
thereby simplifying the overall complexity of data
management in distributed systems.
Chapter 16 | 15 Common failure causes| Quiz and
Test
1.A single point of failure in a distributed system can
cripple it if a critical component fails.
[Link] processes in a distributed system can only be
attributed to high memory usage and never to network
issues.
[Link] failures can occur when one component's failure
leads to additional components failing, threatening the
entire system.
Chapter 17 | 16 Downstream resiliency| Quiz and
Test
[Link] a timeout for network calls is crucial to
avoid resource leaks when calls do not return.
[Link] a backoff strategy for retries is unnecessary
and does not help in managing retries effectively.
3.A circuit breaker pattern operates solely in one state and
does not monitor the health of downstream dependencies.
Chapter 18 | 17 Upstream resiliency| Quiz and Test
[Link] shedding helps servers maintain
 performance by accepting all incoming requests,
even during overload conditions.
[Link] leveling allows clients to send requests without
expecting an immediate response, helping to manage traffic
spikes.
[Link] watchdog mechanism can monitor a process’s health
and automatically restart it if it encounters issues.
Chapter 19 | 18 Testing| Quiz and Test
[Link] tests are designed to focus on large
subsystems of the codebase.
[Link]-to-end tests validate user-facing scenarios and are
usually fast and easy to maintain.
[Link] test doubles is a way to reduce test size without
compromising the scope of testing.
Chapter 20 | 19 Continuous delivery and
deployment| Quiz and Test
[Link] Delivery (CD) automates the process
of releasing changes to a system, making it safer
and more efficient.
[Link] release process consists of three stages: review, build,
and production rollout.
[Link] the production stage, the artifact is initially deployed to
the entire production environment without incremental
rollout.
Chapter 21 | 20 Monitoring| Quiz and Test
[Link] in distributed systems has transitioned
 only from a white-box approach to a black-box
approach.
[Link]-level Objectives (SLOs) help prioritize the
implementation of new features over repairs by using an
error budget.
[Link] alerting requires a balance between precision and
recall.
Chapter 22 | 21 Observability| Quiz and Test
[Link] tools are unnecessary for
understanding emergent behaviors in distributed
systems.
[Link] can vary in format and are crucial for debugging in
distributed systems.
[Link] tracing in distributed systems does not
require modifying components within the request path.
Chapter 23 | 22 Final words| Quiz and Test
[Link] book 'Understanding Distributed Systems'
recommends reading about Windows Azure
Storage for insights on strong consistency.
[Link] to the chapter summary, Azure Data Explorer is
a traditional on-premise storage solution.
[Link] book suggests reading 'System Design Interview by
Alex Xu' for a structured framework for system design
preparation.