Distributed Systems Notes
Distributed Systems
Proper Notes with Diagrams
Topic-wise study notes covering concepts, communication models, ordering, and clocks.
Figure 1. Distributed architecture, client–server, and peer-to-peer.
Topic-wise notes with diagrams
Distributed Systems Notes
1. Distributed Systems: Meaning, Goals, Challenges, Scalability, and
Security
De nition
A distributed system is a collection of independent computers that communicate through a network and coordinate
their actions so that, from the user's perspective, the entire arrangement behaves like a single coherent system. The
machines may be physically separate, may run different operating systems, and may fail independently, yet
together they provide common services, data, and computation.
Examples include cloud storage services, online banking systems, e-commerce platforms, search engines, and
university ERP systems. A user sees one service, but internally many servers, databases, and communication links
cooperate.
Goals of a distributed system
• Resource sharing: multiple users and applications can access les, devices, storage, software services, and
databases.
• Scalability: the system should keep functioning effectively even when users, data volume, or number of
machines increase.
• Reliability and fault tolerance: the failure of one component should not immediately stop the complete
service.
• Performance: work can be divided among multiple machines to increase throughput and shorten response
time.
• Transparency: the complexity of many machines should be hidden as far as possible from the user.
• Openness: components built with different technologies should still interoperate using standard interfaces.
Challenges in distributed systems
A distributed system has no single shared memory and no single perfectly accurate global clock. Components
execute concurrently, messages may be delayed or lost, and one part may fail while the rest continue working.
Because of these realities, coordination is much harder than in a centralized system.
Topic-wise notes with diagrams
fi
fi
Distributed Systems Notes
Challenge Meaning Why it matters
Interoperability requires common
Different hardware, operating systems,
Heterogeneity interfaces, middleware, and data
languages, and protocols coexist.
formats.
Without synchronization, race
Many users and processes access
Concurrency conditions and inconsistent data may
shared resources at the same time.
appear.
The system must detect problems and
Partial failure Some parts fail while others continue.
recover or degrade gracefully.
The system must handle growth in A design that works on a small scale
Scalability
users, data, and machines. may fail under heavy load.
Authentication, authorization,
Communication happens over a
Security encryption, and auditing become
network with many entry points.
necessary.
Scalability
Scalability means that the system can grow without serious loss of performance or manageability. A scalable
distributed system avoids depending on one central bottleneck for every operation. Instead, it uses techniques
such as replication, caching, partitioning, load balancing, and decentralization.
• Size scalability: handling more users, more data, and more requests.
• Geographical scalability: working across long distances where communication delay is larger.
• Administrative scalability: functioning across multiple departments or organizations with different policies.
A common example is an online shopping site that begins on one server and later evolves into separate distributed
services for login, catalog, search, payments, and order tracking.
Security challenges
Security is more dif cult in distributed systems because communication happens over networks and the system is
spread over many nodes. An attacker may target user accounts, network traf c, APIs, databases, or weakly
secured services.
• Authentication veri es identity.
• Authorization controls what an authenticated user or service may access.
• Con dentiality protects data from unauthorized reading, usually with encryption.
• Integrity ensures that data is not altered improperly in transit or storage.
• Availability aims to keep the service usable even during faults or attacks.
Topic-wise notes with diagrams
fi
fi
fi
fi
Distributed Systems Notes
2. Resource Sharing and Heterogeneity
Resource sharing and its importance
Resource sharing is one of the main reasons distributed systems are built. Instead of duplicating every le server,
printer, database, or software package for every user, a distributed system allows controlled shared access. This
reduces cost, improves utilization, and supports collaboration.
For example, a university may keep attendance, course registration, fee records, and results in a central shared
database while students, faculty, and administrators use the same system through different devices and roles.
Heterogeneity
Heterogeneity means diversity in components. A distributed system may contain mobiles, laptops, servers, and
cloud machines. These may run Windows, Linux, Android, or macOS. Applications may be written in Java,
Python, C++, or JavaScript. Data formats and protocols may also differ.
Heterogeneity is useful because each component can use suitable technology, but it creates compatibility
challenges. Standard protocols, APIs, middleware, and portable data formats such as JSON and XML help solve
these problems.
3. Failures and Architectural Models
Types of failure
Because the system is spread across machines and networks, several failure types are possible.
Failure type Meaning Example
A process or node stops and does
Crash failure A server crashes because of power loss.
nothing further.
A message or response is not sent or
Omission failure A packet is dropped by the network.
not received.
Timing failure A result arrives too early or too late. A control message misses its deadline.
The component responds but with an A service returns stale data instead of
Response failure
incorrect value or wrong sequence. the latest value.
A component behaves unpredictably or A compromised node sends con icting
Byzantine / arbitrary failure
maliciously. information to peers.
The network path between healthy Two working servers cannot reach each
Link failure
nodes breaks. other because a switch fails.
Partial failure and why it is common
A partial failure means one or more parts fail while other parts continue to work. This is very common in
distributed systems because the components are independent. One machine may crash, one network link may fail,
one message may be delayed, or one service may become overloaded while other services continue normally.
Topic-wise notes with diagrams
fl
fi
Distributed Systems Notes
This is harder to handle than total failure because a missing response does not immediately tell us what happened.
The sender cannot easily know whether the receiver crashed, the network is slow, or the message is still in transit.
Architectural models
Architecture Main idea Strengths Limitations
Multiple cooperating nodes Needs coordination,
Flexible, scalable, and can
Distributed architecture collectively provide the monitoring, and careful fault
mix many internal styles.
service. handling.
Server may become
Clients request services from Easy to manage and update
Client–server bottleneck or single point of
centralized servers. centrally.
failure.
Each node can both request Decentralized and often Harder to manage, secure,
Peer-to-peer
and provide service. scalable. and discover peers.
4. IPC, Message Passing, Shared Memory, RPC, RMI, and Data
Conversion
Inter-Process Communication (IPC)
IPC is the exchange of data and coordination signals between processes. In distributed systems, IPC usually
happens over a network, so explicit communication becomes central.
Message passing
In message passing, one process explicitly sends a message and another receives it. This model is natural for
distributed systems because processes do not share physical memory.
Shared memory
In shared memory, multiple processes access the same memory region. It is very fast on the same machine, but
synchronization using locks or semaphores is necessary. Across machines, shared memory is far less natural
because memory is not physically common.
Aspect Message passing Shared memory
Read/write on a common memory
Model Explicit send/receive of messages.
region.
Distributed systems and networked Processes on the same machine or
Best suited for
processes. tightly coupled systems.
Often built around communication Requires explicit control such as locks
Synchronization
events. or semaphores.
Clear boundaries and good network
Advantage Very fast local sharing.
suitability.
Topic-wise notes with diagrams
Distributed Systems Notes
Aspect Message passing Shared memory
Dif cult across distributed nodes and
Limitation Affected by latency and bandwidth.
prone to races.
RPC and RMI
Remote Procedure Call (RPC) allows a process to call a procedure on another machine as though it were local.
Remote Method Invocation (RMI) extends the idea in an object-oriented setting.
RMI differs from RPC in three important ways. First, RMI is object-oriented, so methods are invoked on objects
rather than simple procedures. Second, RMI can deal with remote object references, allowing a client to hold and
use a reference to a remote object. Third, RMI commonly uses object serialization and deserialization to transfer
object state.
Feature RPC RMI
Programming style Procedure/function oriented Object-oriented method invocation
Unit invoked Procedure or function Method on a remote object
Usually simple values or structured
References Can use remote object references
parameters
More strongly associated with object
Serialization Basic marshalling of parameters
serialization
Typical use Traditional distributed services Object-oriented distributed applications
Marshalling and unmarshalling
Marshalling converts application data into a network-transferable format such as a byte stream. Unmarshalling
reconstructs the original data at the receiver. These steps are needed because different machines may represent
data differently.
Figure 2. Basic application → network → application ow.
Topic-wise notes with diagrams
fi
fl
Distributed Systems Notes
Steps in message-based IPC
In a typical message-based interaction, the sender prepares data, the send operation hands it to a buffer or
communication subsystem, the network transfers it, and the receiver uses a receive operation to obtain and
process it.
Figure 3. Typical path followed by a message in IPC.
• Send/receive: the sender transmits and the receiver obtains a message.
• Buffering: the operating system or middleware may hold messages temporarily in a queue.
• Blocking send or receive: the process waits until the action completes or a message becomes available.
• Non-blocking send or receive: the process continues immediately and can do other work in parallel.
Latency and bandwidth
Latency is the time taken for a message to travel from sender to receiver. Bandwidth is the amount of data that can
be transferred per unit time. High latency hurts short request–response interactions, while low bandwidth hurts
large transfers. Good distributed system design tries to reduce unnecessary communication, combine messages,
cache results, and place related services near each other.
5. MPI, Indirect Communication, and Decoupling
Point-to-point and collective communication in MPI
MPI (Message Passing Interface) is a standard used widely in parallel and distributed computing. In point-to-point
communication, one process sends a message to one chosen receiver, for example through MPI_Send and
MPI_Recv. In collective communication, a whole group participates together in an operation such as broadcast,
reduce, scatter, or gather.
Topic-wise notes with diagrams
Distributed Systems Notes
Figure 4. Two important MPI communication patterns.
Indirect communication and its advantages
In indirect communication, sender and receiver communicate through an intermediary such as a mailbox, channel,
broker, or message queue.
• The sender does not need to know the exact identity of the receiver.
• The receiver may collect the message later, so both sides need not be active at the same time.
• Systems become more modular and scalable because producers and consumers are loosely coupled.
Three types of decoupling
• Time decoupling: sender and receiver do not need to be active at the same time. Example: queued messages.
• Space decoupling: sender and receiver do not need to know each other directly. Example: publish–subscribe
topics.
• Synchronisation decoupling: sender and receiver do not have to wait for each other. Example: non-blocking
send.
6. Event Ordering Models
Because there is no perfect global clock in a distributed system, message ordering must often be enforced by
protocols.
Topic-wise notes with diagrams
Distributed Systems Notes
Figure 6. FIFO, causal, and total ordering.
FIFO ordering
FIFO (First In First Out) ordering preserves the order in which a single sender transmits messages to the same
receiver. If P sends m1 and then m2, the receiver must not deliver m2 before m1.
Causal ordering
Causal ordering preserves cause-and-effect relationships. If one event could have in uenced another, all processes
must observe them in that order. For example, a reply should not appear before the original message that caused
it.
Total ordering
Total ordering requires every process to deliver all messages in exactly the same order, even when those messages
are independent. This is valuable in replicated databases and state machine replication.
Ordering model Guarantee Where used Main limitation
Does not capture causal
Simple messaging and
FIFO Preserves per-sender order. relations across different
protocol channels.
senders.
Preserves cause-and-effect Collaborative systems, chats, Needs extra metadata and
Causal
order. shared replicated spaces. protocol complexity.
Topic-wise notes with diagrams
fl
Distributed Systems Notes
Ordering model Guarantee Where used Main limitation
Replicated databases, atomic
All processes see the same
Total broadcast, strong Highest coordination cost.
order.
consistency systems.
7. Lamport Clocks and Vector Clocks
Lamport logical clocks
Lamport clocks assign one logical timestamp to each event. They do not represent physical time, but they help
create a consistent order of events.
• Increment rule: before every local event, increment the local clock.
• Send rule: attach the current timestamp when sending a message.
• Receive rule: on receiving a message with timestamp t, set local clock to max(local clock, t) + 1.
Lamport clocks can produce a total order when timestamps are combined with a tie-breaker such as process ID.
However, they cannot detect concurrency accurately. If timestamp(A) < timestamp(B), A did not necessarily cause
B; the events may still be independent.
Vector clocks
Vector clocks keep a vector of counters rather than one number. In a system with n processes, each process
maintains a vector of length n.
• Structure example: [2, 5, 1] means the process knows it has seen 2 events from P1, 5 from P2, and 1 from P3.
• Local update rule: increment your own entry before a local event.
• Send rule: increment your own entry and attach the full vector to the message.
• Receive rule: take the element-wise maximum of the received vector and local vector, then increment your
own entry.
Vector clocks capture causal ordering because one vector being smaller than another in all components means the
rst event happened before the second. They detect concurrency when neither vector is smaller than the other.
Figure 7. Lamport clocks vs vector clocks.
Topic-wise notes with diagrams
fi
Distributed Systems Notes
Aspect Lamport clocks Vector clocks
State maintained One integer per process One vector per process
Causality capture Partial Much more accurate
Concurrency detection No Yes
Higher because vectors grow with
Overhead Low
number of processes
Simple logical ordering and total-order Reasoning about happened-before and
Typical use
support concurrent events
8. Physical Clocks, Drift, Skew, and Synchronization
Physical clocks
Physical clocks represent real-world time maintained by hardware. Distributed systems use them for timestamps,
logs, leases, deadlines, and time-based coordination.
Clock drift and clock skew
Clock drift means a clock runs slightly faster or slower than ideal time. Clock skew is the difference between the
times shown by two clocks at the same moment.
Figure 8. Drift and skew in physical clocks.
Drift and skew affect event ordering because logs from different machines may not match real causal order. A
message may appear to be received before it was sent. Timeout calculations and distributed debugging also
become dif cult.
Synchronization goals and limitations
Clock synchronization aims to keep machine clocks suf ciently close for the system’s needs. Goals include
reducing skew, making timestamps more reliable, and supporting coordinated behavior.
Topic-wise notes with diagrams
fi
fi
Distributed Systems Notes
Perfect synchronization is impossible in practice because network delays vary, messages take uncertain time to
arrive, and hardware clocks continuously drift. Therefore, systems aim for acceptable bounds rather than exact
equality.
9. Quick Recall Table
Topic One-line recall
Distributed system Independent computers cooperate and appear as one system.
Multiple users access common resources such as les,
Resource sharing
services, and storage.
Heterogeneity Different technologies must still interoperate.
Partial failure Some components fail while others continue.
IPC Processes exchange data and coordination signals.
RPC is procedure-oriented; RMI is object-oriented with
RPC vs RMI
remote references and serialization.
FIFO / causal / total Three increasingly stronger message-ordering guarantees.
Lamport gives simple logical order; vector clocks capture
Lamport vs vector
causality and concurrency.
Drift / skew Drift is difference in rate; skew is difference in shown time.
Topic-wise notes with diagrams
fi