innovateITzone subscribe on youtube for more
Parallel & Distributed Computing (BSCS)
Asynchronous vs Synchronous
Computation & Communication
Synchronous Execution
In synchronous execution, each task waits for its turn to complete before the next one
starts.
Example: Imagine you're talking to your friends one by one. You first talk to friend A,
and once that conversation ends, you move on to friend B, and so on.
Key points:
o Tasks occur in a sequential order, one after the other.
o Predictable but slower, as tasks have to wait for each other to finish.
Asynchronous Execution
In asynchronous execution, each task runs independently without waiting for the
others to complete.
Example: Sending WhatsApp messages to multiple people at once. You can keep
working and check replies when they come, without stopping your tasks.
Key points:
o Tasks operate independently and do not wait for each other.
o Faster but more complex, as coordination between tasks may be harder.
Communication Models
There are two primary communication models used in computing:
1. Message-Passing Communication
In message-passing, separate processes communicate with each other by sending
messages.
Example: Think of two people in separate rooms passing notes to each other. They
communicate by sending and receiving messages.
Key points:
o Each process has its own memory space.
o Processes communicate by sending messages through a communication
system (like MPI).
2. Shared-Memory Communication
In shared-memory, multiple processes use the same memory space to communicate.
innovateITzone subscribe on youtube for more
Example: Imagine a whiteboard where everyone can write. If coordination isn’t done
well, things can get messy. But if synchronized, it works efficiently.
Key points:
o All processes share the same memory space.
o Fast and efficient if synchronization is handled properly.
o Used in multicore systems and CPUs (e.g., OpenMP, pthreads).
Blocking vs Non-blocking Communication
Blocking Communication: The process waits for an operation (like sending or
receiving data) to finish before continuing.
o Example: Talking on the phone. You wait for the person to pick up before
you can speak. You can't do anything else until the call is connected.
o Key points:
The process is "blocked" until the operation completes.
Examples in MPI: MPI_Send() and MPI_Recv() are blocking
functions.
Non-blocking Communication: The process does not wait for the operation to
complete and can continue doing other tasks while checking the operation's status
later.
o Example: Sending a courier package. While waiting for the delivery, you can
continue with your tasks. You check the delivery status later.
o Key points:
The process keeps running, not waiting for the operation to complete.
Examples in MPI: MPI_Isend() and MPI_Irecv() are non-blocking
functions.
MPI Hello World Program Example (Real Use)
simple MPI program to demonstrate message-passing:
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
MPI_Init(&argc, &argv); // Start the communication system
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Get the process rank
MPI_Comm_size(MPI_COMM_WORLD, &size); // Get the total number of
processes
printf("Hello from process %d of %d\n", rank, size);
MPI_Finalize(); // Close the communication system
return 0;
}
innovateITzone subscribe on youtube for more
Output example (if 4 processes are running):
Hello from process 0 of 4
Hello from process 1 of 4
Hello from process 2 of 4
Hello from process 3 of 4
Introduction to MPI (Message
Passing Interface)
Definition:
MPI is a standard API (Application Programming Interface) used for parallel
programming on distributed memory systems.
It allows multiple processes (running on the same or different computers) to
communicate by sending and receiving messages.
MPI Functions
1. MPI_Init, MPI_Comm_rank, MPI_Comm_size
MPI_Init(&argc, &argv)
o Called at the beginning of every MPI program.
o Initializes the MPI environment (sets up communication channels).
MPI_Comm_rank(MPI_COMM_WORLD, &rank)
o Returns the rank (ID) of the process in the communicator.
o Example: If there are 4 processes, the ranks would be 0, 1, 2, 3.
o Each process has a unique rank, helping identify which process is doing the
work.
MPI_Comm_size(MPI_COMM_WORLD, &size)
o Returns the total number of processes that are running in the communicator.
Real-life Example for MPI_Comm_rank & MPI_Comm_size:
Imagine you’re leading a team of 4 people. Each person has a unique ID (rank), and
you know the total size of the team (size).
o You (Rank 0) are the leader, and there are 3 other team members in total
(size = 4).
2. MPI_Send and MPI_Recv (Point-to-Point Communication)
MPI_Send()
o Sends a message from one process to another.
o Syntax:
innovateITzone subscribe on youtube for more
MPI_Send(buffer, count, datatype, destination_rank, tag,
communicator);
MPI_Recv()
o Receives a message sent by another process.
o Syntax:
MPI_Recv(buffer, count, datatype, source_rank, tag, communicator,
&status);
Real-life Example for MPI_Send & MPI_Recv:
Imagine you are sending a letter (message) to your friend (destination rank). Your
friend receives and reads it (MPI_Recv).
o You are using MPI_Send, and your friend is using MPI_Recv.
3. MPI_Bcast (Broadcast Communication)
MPI_Bcast()
o One process (usually rank 0) sends a message to all other processes.
o Syntax:
MPI_Bcast(buffer, count, datatype, root_rank, communicator);
Real-life Example for MPI_Bcast:
Imagine you're a teacher (rank 0) announcing a notice to the entire class (all other
processes).
o You broadcast the same message to everyone at once.
4. MPI_Scatter & MPI_Gather
MPI_Scatter()
o The root process distributes data chunks to all processes.
o Syntax:
o MPI_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount,
recvtype, root, comm);
MPI_Gather()
o All processes send their results to the root process.
o Syntax:
MPI_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount,
recvtype, root, comm);
Real-life Example for MPI_Scatter & MPI_Gather:
Scatter: You are a teacher distributing parts of a test paper to your students.
Gather: After completing the test, the students send their answers back to you.
innovateITzone subscribe on youtube for more
Point-to-Point vs Collective Communication
Communication Description Examples
Type
Point-to-Point 1-to-1 communication MPI_Send, MPI_Recv
Collective 1-to-many or many-to- MPI_Bcast, MPI_Scatter,
1 MPI_Gather
Real-life Analogy:
Point-to-Point: Sending a message to one friend.
Collective: Sending a message to a whole group of friends on WhatsApp.
Practical Examples in MPI
1. Hello World Program Using MPI
A simple "Hello World" program using MPI to display the process rank and total
processes.
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
MPI_Init(&argc, &argv); // Initialize MPI environment
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank); // Process ID
MPI_Comm_size(MPI_COMM_WORLD, &size); // Total number of processes
printf("Hello from process %d out of %d processes\n", rank, size);
MPI_Finalize(); // Clean up MPI environment
return 0;
}
Output (for 4 processes):
Hello from process 0 out of 4 processes
Hello from process 1 out of 4 processes
Hello from process 2 out of 4 processes
Hello from process 3 out of 4 processes
2. Practical MPI_Send / MPI_Recv Example
A program where process 0 sends data to process 1, and process 1 receives it.
innovateITzone subscribe on youtube for more
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
MPI_Init(&argc, &argv);
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0) {
int data = 100;
MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
printf("Process 0 sent data: %d\n", data);
} else if (rank == 1) {
int data;
MPI_Recv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
printf("Process 1 received data: %d\n", data);
}
MPI_Finalize();
return 0;
}
Explanation:
Process 0 sends data to Process 1 using MPI_Send().
Process 1 receives the data using MPI_Recv().
Expected Output:
Process 0 sent data: 100
Process 1 received data: 100