MODULE 5: INTERPROCESS COMMUNICATIONS IN

DISTRIBUTED SYSTEMS
INSTRUCTIONAL HOURS: 4, 5 or 6

MODULE OVERVIEW
welcome to the fifth module of distributed systems. In this module you will learn how inter-
process communication (IPC) enable processes across different nodes to exchange data and
coordinate activities.
Learning Outcomes
By the end of this module, you will be able to:
1. Explain the placement of the various inter-process communication methods within the
computer network communication layers

2. Demonstrate inter-process between a client and a server using gRPC, RabbitMQ

messaging and Sockets

3. Develop a simple client server hello world application in python using gRPC, RabbitMQ
messaging and Sockets

4. Implement a simple client server hello world application in python using gRPC,
RabbitMQ messaging and Sockets

LEARNING ACTIVITIES/TASK LIST

1. Read Module Notes

2. Watch lecture related videos
3. Complete practical exercises

Activity 1 : READING MATERIAL

Inter-process communication is at the heart of all distributed systems since they involve linking
of processes between remote nodes

Since the distributed systems are implemented over computer networks, the communication
between the various components is implemented over the layered communication architecture
of computer networks.

In distributed systems, a layered implementation for communication involves structuring the

system into distinct layers, each responsible for a specific aspect of communication. This
approach simplifies the design, enhances modularity, and facilitates communication across
various components and network nodes

P a g e 83 | 152
PROF SIMON MAINA KARUME
Communication in distributed systems utilizes the layered architecture of computer networks.
There are two network models that demonstrate how layered architecture is organized in
computer networks:
 OSI Model
 TCP/IP Model
The layers in the two models are demonstrated in figure 5.1

Figure 5.1: communication layers in OSI and TCP/IP Models

Benefits of Layered communication Architecture
 Modularity
 Interoperability
 Flexibility
 Reusability
 Scalability
 Security
Challenges in Layered Architecture
 Performance Overhead
 Complexity in Implementation
 Resource Utilization
 Debugging and Troubleshooting
 Protocol Overhead
In distributed systems the Middleware and distributed applications are implemented on top of
a network protocol as demonstrated in figure 5.2

Figure 5.2: Implementation of layered communication architecture in Distributed systems.

P a g e 84 | 152
PROF SIMON MAINA KARUME
The role of the middleware is to facilitate interoperability between different network protocols,
diverse hardware systems and dissimilar software technologies. This is the reason why
middleware needs to be abstracted from the physical network by being implemented at a
higher abstract layer close to the application layer.
 Examples:
Examples of middleware include message queues, transaction processing monitors, and APIs.
 Benefits:
Middleware enables the creation of distributed systems where different components can work
together seamlessly, regardless of their underlying technology.

middleware simplifies the development of distributed applications by abstracting away the

complexities of networking and inter-process communication.

Summary of communication Layers in distributed system and their roles

LAYER 5 APPLICATIONS used by the Examples: Mobile apps for accessing bank services,
users to interact with the USSD Codes for accessing services of a distributed
distributed sys system, SMS, WHATSAPP CHATS, short codes ETC
e.g. *247#, *522#
LAYER 4 Remote procedure call Marshaling / serialization is done by RMI or RPC
(RPC) and Remote Method
LARE 3 AND 4 FORM THE MIDDLEWARE

Invocation (RMI) Marshaling is where data is packaged in a

LAYER 3 Request and Reply technology neutral format for transmission (IT IS A
primitives SOLUTION FOR HETEROGEINITY)
 Message passing
 Distributed Shared Serialization is a type of marshaling in java where
Memory (DSN) objects are converted to bytes (8 bits)
Common technologies used to implement
marshaling / serialization at this layer include XML,
JSON, protocol buffer
 SOAP (xml)
 RESTful (json)
There are more technologies used to provide web
data serialization as presented in the handout
uploaded on the E-learning Platform
LAYER 2 OS and Network Protocols Stream oriented communication happens at this
(TCP and UDP) layer
Stream communication is implemented using
Socket program
socket = ip address + port number
1. Datagram socket (use UDP)
2. Stream socket (TCP)
LAYER 1 H/W + N/W (physical) Physical connections WAN connections

From the tabulated summary of the layered interprocess communication in distributed

systems, we can isolate three types of interprocess communication as follows:
P a g e 85 | 152
PROF SIMON MAINA KARUME
 1. Remote procedure calls (RPC) and Remote method invocation (RMI)
2. Message passing communication
3. Distributed shared memory
4. Stream oriented communication

Remote procedure calls (RPC) and Remote method invocation (RMI)

Remote Procedure Call (RPC) and Remote Method Invocation (RMI) are considered middleware
technologies.
 The main purpose of RPC is to allow a local computer (client) to invoke procedures on a
remote computer (server).
 The main purpose of RMI is to allow a local computer (client) to invoke objects on a
remote computer (server).
 Both RPC and RMI contains piece of code called a stub which package the parameters in
the message being passed by the client to the server. the packaging is done in a
technology neutral manner to ensure that the message can be interpreted by the server
regardless the technology it is using. the process of packaging and unpackaging the
message parameters or data in a technology neutral format is called Marshaling.
 Marshaling ensures data can be converted to a common format for either
o network transmission or
o for storage
 Serialization is a form of marshaling that concerts data objects into byte streams
Common Marshaling Formats:
 JSON (JavaScript Object Notation): A lightweight data-interchange format widely used
in web applications for its ease of use and readability.
 XML (Extensible Markup Language): A more structured format than JSON, often used
for data exchange between systems that require more complex data structures.
 Protobuf (Protocol Buffers): A binary format designed for high-performance data
exchange, often used in applications where speed and efficiency are critical.
 YAML (YAML Ain't Markup Language): YAML is a human-readable data serialization
format. It is particularly suited for configuration files and data that's being directly
edited by humans. YAML uses a non-strict whitespace syntax with key-value pairs. It can
represent scalars (strings, numbers), lists, and associative arrays.
o YAML is often used in configuration files and for data that requires a high degree
of human readability.
Example:
o name: John Doe
o age: 30
o email: johndoe@[Link]
RPC /RMI STEPS
1. The client procedure calls a client stub passing parameters in the normal way.
2. The client stub marshals the parameters, builds the message, and calls the local OS.
3. The client's OS sends the message (using the transport layer) to the remote OS.
4. The server remote OS gives transport layer message to a server stub.
5. The server stub demarshals the parameters and calls the desired server routine.
6. The server routine does work and returns result to the server stub via normal procedures.
P a g e 86 | 152
PROF SIMON MAINA KARUME
 7. The server stub marshals the return values into the message and calls local OS.
8. The server OS (using the transport layer) sends the message to the client's OS.
9. The client's OS gives the message to the client stub
10. The client stub demarshals the result, and execution returns to the client.

RPC and RMI are built upon message passing as their fundamental communication
mechanism. They utilize message passing to transport requests and replies between client and
server. hence the reason RPC and RMI are build above the message passing layer.

Activity 2 : VIDEO ON GRPC FRAMEWORK

gRPC is a high-performance, open-source Remote Procedure Call (RPC) framework created by

Google to facilitate communication between services. It's designed for building connected
systems, particularly in microservices environments, and can run across various platforms and
programming languages. gRPC uses HTTP/2 as its transport protocol and Protocol Buffers for
serialization, offering faster and more efficient communication compared to older protocols like
HTTP/1.1.

click here to get started on gRPC ([Link] ) and implement the

simple hello world code given

Define the service ([Link]).

syntax = "proto3";
package hello;
service Greeter {
rpc SayHello (HelloRequest) returns (HelloReply);
}
message HelloRequest {
string name = 1;
}
message HelloReply {
string message = 1;
}

Generate gRPC code.

python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. [Link]

//NB this code is generate into a google protocol buffer code (this is marsalling)

P a g e 87 | 152
PROF SIMON MAINA KARUME
Implement the server ([Link]).
import grpc
from concurrent import futures
import hello_pb2
import hello_pb2_grpc

class Greeter(hello_pb2_grpc.GreeterServicer):
def SayHello(self, request, context):
return hello_pb2.HelloReply(message=f"Hello, {[Link]}!")

def serve():
server = [Link]([Link](max_workers=10))
hello_pb2_grpc.add_GreeterServicer_to_server(Greeter(), server)
server.add_insecure_port("[::]:50051")
[Link]()
server.wait_for_termination()

if __name__ == "__main__":
serve()

Implement the client ([Link]).

import grpc
import hello_pb2
import hello_pb2_grpc

def run():
with grpc.insecure_channel("localhost:50051") as channel:
stub = hello_pb2_grpc.GreeterStub(channel)
response = [Link](hello_pb2.HelloRequest(name="World"))
print(f"Greeter client received: {[Link]}")

if __name__ == "__main__":
run()

P a g e 88 | 152
PROF SIMON MAINA KARUME
 Start the server
python [Link]

Start the client on another terminal

python [Link]

Activity 3 : MESSAGE PASSING COMMUNICATION

Message Passing IPC

 The message passing communication layer is build on top of the stream communication
layer which is the TCP/UDP network protocol layer. This is because message passing
communication often uses socket programming as a fundamental mechanism, especially
in distributed systems. Sockets provide a way to establish communication channels
between different processes, whether on the same machine or across a network,
making them suitable for implementing message passing. stream communication is
implemented using socket programs
 Messaging is a technology that enables high-speed, asynchronous communication.
 A messaging system is responsible for transferring data from one application to another
so the applications can focus on data without getting bogged down on data
transmission and sharing.
 Distributed messaging is based on the concept of reliable message queuing. Messages
are queued asynchronously between client applications and messaging system. There
are two types of messaging patterns.
 point-to-point
 “Publish–subscribe” (pub-sub) messaging system.
 Most of the messaging systems follow the pub-sub pattern.

Message Passing constructs

 There are 2 basic message passing primitives, send and receive
 send primitive: sends a message (data) on a specified channel from one process to
another,
 The send primitive has different semantics depending on whether the message
passing is synchronous or asynchronous.
 receive primitive: receives a message on a specified channel from other processes.
 Message passing can be viewed as extending semaphores to convey data as well as
synchronization.
 Messages can be of FIXED or VARIABLE length
 Fixed size is easy to implement at system level but it difficult to program

P a g e 89 | 152
PROF SIMON MAINA KARUME
  Variable size is easy to program but implementation at system level is
challenging
 If process P and Q want to communicate, they must send message to and receive
message from each other
 For P and Q to Communicate a COMMUNICATION link or message channel must exist as
demonstrated in Figure 5.3

Figure 5.3: setup of a messaging communication

The communication link as well as the send and the receive operations can be implemented in
a variety of ways
The various methods of logically implementing the link and the send() and receive() operations
include:
 Direct or indirect communication
 Synchronous or asynchronous communication
 Automatic or explicit buffering
 Issues associated with each of the logical implementation above include:
 Namig
 Synchronization
 Buffering

 Process that want to communicate can do so either directly or indirectly and they must
have a way to refer to each other

Direct communication
 in direct communication each process that wants to communicate must explicitly name
the recipient or sender of the communication e.g
 Send (P, Message) – send a message to process P
 Receive (Q, Message) – receive a message from process Q

 A link is established automatically between every pair of processes that wants to

communicate. The processes need to know only each others identity to communicate
 A link is associated with exactly two processes
 Between each pair of commutating processes there is exactly one link

P a g e 90 | 152
PROF SIMON MAINA KARUME
  This scheme of message passing must exhibit symmetry in addressing i.e. both the
sender process and the receiver process must name each other in order to
communicate

Another variant of Direct communication

 The sender names recipient but recipient is not required to name sender
 Send (P, Message) – send a message to process P
 Receive (id, Message) – receive a message from any process with a variable id
where the variable is set to the name of the sending process
 This scheme employs asymmetry in addressing

Limitation of Direct communication

 The disadvantage of both symmetric and asymmetric direct message passing schemes is
the limited modularity of the resulting process definitions, changing the identifier of a
process may necessitate examining all other process definitions

Indirect communication
 With indirect communications the messages are sent to and received from mailboxes or
ports e.g. in email communication
 A mailbox can be viewed abstractly as an object into which messages can be placed by
processes and from which messages can be removed
 Each mailbox has a unique identification
 Two processes can communicate only if they have a shared mailbox
 The send () and receive primitives will be as follows
 Send (A, Message) : Send a message to mailbox A
 Receive (A, Message): Receive a message from mailbox A
 A link is established between a pair of processes only if both members have a shared
mailbox
 A link may be associated more that one processes
 Between each pair of commutating processes there may be more than one link i.e. a
number of different links may exist between a pair of communicating processes with
each link corresponding to one mailbox

Case Scenario

 Suppose processes p1, p2 and p3 share the same mail box A

 Process p1 sends a message to A while p2 and p3 execute a receive() from A

 Which process will receive the message sent by p2?

 The answer to this question depends on which of the following methods is chosen
1. Allow a link to be associated with TWO processes at most
2. Allow at most one process at a time to execute a receive () operation

P a g e 91 | 152
PROF SIMON MAINA KARUME
 3. Allow the system to select arbitrarily which process will receive the message (that
is either p2 or p3 but not both will receive the message) the system also may
define an algorithm for selecting which process will receive the message e.g.
round robin where processes take turns in receiving the message, the system may
identify the receiver to the sender

Asynchronous vs Synchronous Messaging

• The two basic kinds of messages are asynchronous and synchronous. A sender of
an asynchronous message continues to execute after sending the message, whereas a
sender of a synchronous message waits until it receives a reply from the receiver that it
has completed its processing of the message before continuing execution.
• The send () and receive () primitives implement asynchronous and synchronous
messaging differently using either blocking or non-blocking

o Synchronous messaging is implemented using blocking send and blocking

receive
 For blocking send the sending process is blocked until the message is
received by the receiving process or by the mailbox
 For blocking receive: receiver blocks until a message is available

o Asynchronous messaging is implemented using non-blocking send and non-

blocking receive
 For non-blocking send: the sending process resumes operation after
sending the message (no waiting hence it is fast)
 For non-blocking receive: the receiver constantly checks for a
message without blocking and thus retrieves either a valid message
or a null

Buffering in message passing

• A buffer is a temporary holding location where messaged are held awaiting transfer to
recipients
• Whether communication is direct or indirect, messages exchanged by communicating
processes reside in a temporary queue.
• Basically, such queues can be implemented in three ways

1. Zero Capacity Buffer or Queue

• The queue has a maximum length of zero, thus the link cannot have any messages
waiting in it. In this case the sender MUST block until the recipient receives the message
• This is applied in synchronous messaging

2. Bounded Capacity Buffer or Queue

• The queue has a finite length n; thus at most n messages can reside in it. If the queue is
not full when a new message is sent, the message is placed in the queue and the sender
can continue executing without waiting. However if the link capacity is full the sender
MUST block until there is space available in the Queue
P a g e 92 | 152
PROF SIMON MAINA KARUME
 • This is applied in synchronous messaging

3. unbounded Capacity Buffer or Queue

• The queue length is potentially infinite thus any number of messages can wait in the
queue hence the sender never blocks

Publish-subscribe and group messaging models (Examples: RabbitMQ and

Apache Kafka)
• The messaging communication primitives discussed so far involve direct coupling
between sender and receiver where the sender names the receiver and receiver names
the sender
o E.g. Send (request) to server
• This kind of communication is very rigid e.g. in microservices architecture using API to
communicate the communication was seen to be direct and rigid resulting to complexity
and tight coupling which is not desirable in distributed systems.
• The alternative to direct messaging is indirect messaging which is asynchronous and
more flexible. In this type of messaging the messages are sent to an intermediary which
is commonly referred to as MESSAGE BROKER. There is no direct coupling between
sender and receiver resulting to a loosely coupled system which is flexible and scalable.
There are two dimensions of decoupling in this type of communication
o Space uncoupling: here the sender does not need to know the receiver
o Time uncoupling: sender and receiver do not have to operate at the same time
• Two examples of this kind of messaging are:
o Group communication and
o Publish subscribe
• The assumption in client/server and RPC/RMI communication is that only two processes
are involved i.e. the client and the server. How sometimes there are more than two
processes in communication. If we were to use RMI/RPC in this then we need no send
the message to each receiver separately and this is not desirable.
• The alternative is to use group or publish subscribe communication models. Click on the
pdf icon to read about this model on page 33 to 49 – of the handout

IPC Communication in
Distributed Systems [Link]

Distributed shared Memory

 Distributed shared memory is a virtual memory where nodes in a distributed system can
share communication through read and write rather than through send and receive
primitives
 Although the nodes using the shared virtual memory have their own main memory, data
moves between the main memories through the shared virtual memory creating an
illusion that the nodes are operating on a physical shared memory thus making DSM
transparent to all the nodes sharing it.

P a g e 93 | 152
PROF SIMON MAINA KARUME
The challenge of time in distributed shared Memory

Time plays a crucial role in distributed shared memory (DSM) systems for ensuring data
consistency and maintaining a coherent view of the shared memory across multiple nodes. This
is because each node in a DSM system has its own internal clock, which can drift out of
synchronization with others, potentially leading to conflicts and inconsistencies when accessing
shared data

INTER-PROCESS COMMUNICATION USING SCOKETS

Activity 4 :
(IMPLELEMTING SOCKET COMMUNICATION IN PYTHON)

Inter-process communication (IPC) using sockets takes place at the transport layer which is
layer 4 of the OSI model and layer 3 of the TCP/IP model. In the five-layers of IPC it comes at
layer 2. In the 10 steps of the RPC/RMI communication Socket communication takes place at
step 8.

There are two types of scokets

 Stream sockets or TCP sockets
 Datagram sockets or UDP sockets

The choice between TCP and UDP depends on the application's requirements. TCP provides
reliable, ordered delivery of data, while UDP is faster but does not guarantee delivery or order.

Socket communication establishes a connection between two nodes in a distributed

environment

We will use a python example to demonstrate creation of a socket.

To establish communication, both the server and client follow a sequence of steps:

P a g e 94 | 152
PROF SIMON MAINA KARUME
Server-side:
 Create a socket:
The server creates a socket object using [Link](), specifying the address family
(e.g., socket.AF_INET for IPv4) and socket type (e.g., socket.SOCK_STREAM for TCP).
 Bind the socket:
The server binds the socket to a specific IP address and port using [Link]().
 Listen for connections:
The server starts listening for incoming connection requests using [Link]().
 Accept connections:
When a client requests a connection, the server accepts it using [Link](), which returns
a new socket for communication with that client.
 Receive and send data:
The server can then receive data from the client using [Link]() and send data back
using [Link]().
 Close the connection:
After communication is complete, the server closes the connection using [Link]().
Client-side:
 Create a socket:
The client creates a socket object similar to the server.
 Connect to the server:
The client connects to the server's IP address and port using [Link]().
 Send and receive data:
The client can send data to the server using [Link]() and receive data back
using [Link]().
 Close the connection:
After communication is complete, the client closes the connection using [Link]().

Click here to Follow the video on the implementation of a socket in python

Socket Programming in Python(Simplified) - in 7 minutes!
([Link] )

P a g e 95 | 152
PROF SIMON MAINA KARUME
Server ([Link]):
import socket

HOST = '[Link]' # Standard loopback interface address (localhost)

PORT = 65432 # Port to listen on (non-privileged ports are > 1023)

with [Link](socket.AF_INET, socket.SOCK_STREAM) as s:

[Link]((HOST, PORT))
[Link]()
conn, addr = [Link]()
with conn:
print(f"Connected by {addr}")
while True:
data = [Link](1024)
if not data:
break
[Link](data)

Client ([Link]):
import socket
HOST = '[Link]' # The server's hostname or IP address
PORT = 65432 # The port used by the server
with [Link](socket.AF_INET, socket.SOCK_STREAM) as s:
[Link]((HOST, PORT))
[Link](b'Hello, world')
data = [Link](1024)
print(f"Received {data!r}")

P a g e 96 | 152
PROF SIMON MAINA KARUME
REFERENCE LIST AND FURTHER READING

TEXT BOOKS:
1. Distributed Systems Concepts and Design, G Coulouris, J Dollimore and T Kindberg, Fourth
Edition, Pearson Education. 2009.

2. Distributed Systems, Principles and paradigms, Andrew [Link], Maarten Van Steen,
Second Edition,PHI.

3. Distributed Systems, An Algorithm Approach, Sikumar Ghosh, Chapman & Hall/CRC, Taylor
& Fransis Group,2007.

WEBSITES
4. “What Is Distributed Shared Memory and Its Advantages.” GeeksforGeeks, 4 Sept. 2021,
[Link]/what-is-distributed-shared-memory-and-its-advantages/.

P a g e 97 | 152
PROF SIMON MAINA KARUME