Parallel Programming module 3
DISTRIBUTED MEMORY PROGRAMMING WITH
MPI
Distributed memory programming refers to a parallel computing model where each
processor has its own private memory, and processors communicate with each other by
passing messages. Unlike shared memory systems, where all processors access the same
memory space, distributed memory systems consist of multiple independent computers
(nodes), each with its own RAM, CPU, and storage. To solve a large computational
problem, these nodes must work together by exchanging data. The most widely used
standard for distributed memory programming is MPI (Message Passing Interface).
MPI provides a portable and efficient framework for managing communication between
nodes in a cluster. It allows developers to break a large problem into smaller tasks,
distribute the tasks across multiple processors, and combine the results through
communication primitives such as point-to-point messages and collective operations. MPI
supports parallel execution on high-performance clusters, supercomputers, cloud
platforms, and even small multi-core systems. The main purpose of MPI is to help
applications scale efficiently; as the number of processors increases, MPI ensures that
communication overhead remains minimal.
MPI Architecture and Working Principle
MPI follows a process-based parallel model, where each process runs its own copy of the
program. These processes may run on different nodes, but all execute the same program
using the SPMD (Single Program, Multiple Data) approach. Each MPI process is
assigned a unique identifier called a rank, which determines its role. MPI programs
typically begin with initialization using MPI_Init() and end with MPI_Finalize().
Between these calls, processes exchange data using various message-passing routines.
The most basic communication primitives are MPI_Send() and MPI_Recv(), used for
point-to-point communication. MPI also supports collective communication methods
such as MPI_Bcast(), MPI_Scatter(), and MPI_Gather() which help in distributing and
collecting data across all processes.
MPI uses a variety of communication modes—synchronous, asynchronous, buffered, and
ready-mode. These modes help manage how data is transferred between processes. For
example, asynchronous communication allows computation and communication to
overlap, improving performance. MPI also supports communicators, which define groups
of processes that can communicate with each other. This makes MPI flexible for large
applications that need to divide processes into functional subgroups.
Dept Of. CSE (DS), AIET 12
Parallel Programming module 3
Distributed memory programming with MPI is essential for large-scale scientific
computations because it enables applications to scale across hundreds or thousands of
nodes. Since each node maintains its own memory, data movement must be explicitly
handled using MPI calls, giving programmers fine control over communication patterns.
This control is crucial in high-performance computing (HPC) systems, where minimizing
communication time is as important as maximizing computation speed.
Real-Time Examples of MPI Usage
MPI is used in several real-time and industry-level applications that require enormous
computational power. One major real-time example is weather prediction and climate
modeling. Weather forecasting involves solving complex mathematical equations related
to atmospheric pressure, wind speed, humidity, and temperature. These computations are
distributed across thousands of nodes in a supercomputer, where each node handles a
small part of the global simulation grid. MPI is used to exchange boundary values
between regions, enabling accurate and fast prediction results.
Another real-time example is molecular dynamics simulation used in drug discovery.
Simulating the behavior of atoms in a protein structure requires immense computational
power. MPI helps distribute atoms across different processors, and forces between them
are calculated in parallel. Each processor sends and receives particle data from other
processors, enabling extremely fast simulations.
MPI is also used in large-scale financial modeling, such as Monte Carlo simulations for
stock price prediction, risk analysis, and portfolio optimization. Financial institutions use
MPI clusters to carry out millions of random simulations in parallel, reducing execution
time from hours to minutes.
In aerodynamics and automotive engineering, MPI is used to run fluid dynamics
simulations (CFD) that predict airflow behavior over aircraft wings and car bodies. These
simulations require solving Navier–Stokes equations repeatedly. MPI divides the flow
domain into smaller blocks and assigns them to different processors, allowing faster
computation.
Even in real-time astronomy and astrophysics, MPI clusters process huge amounts of
telescope data, simulate galaxy formation, and model black hole interactions. MPI
enables scientists to parallelize the data pipeline and complete tasks that would be
impossible on a single machine.
Dept Of. CSE (DS), AIET 13
Parallel Programming module 3
PROGRAM 1
Parallel Sum Calculation Using MPI
#include <mpi.h>
#include <stdio.h>
int main(int argc, char* argv[]) {
int rank, size;
int number, total_sum;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
number = rank + 1; // Each process stores a different value
MPI_Reduce(&number, &total_sum, 1, MPI_INT, MPI_SUM, 0,
MPI_COMM_WORLD);
if (rank == 0) {
printf("Total Sum = %d\n", total_sum);
}
MPI_Finalize();
return 0;
}
OUTPUT
Process 0: number = 1
Process 1: number = 2
Process 2: number = 3
Process 3: number = 4
Total Sum = 10
Dept Of. CSE (DS), AIET 14
Parallel Programming module 3
Applications of MPI
Scientific Simulations – Climate models, astrophysics simulations, molecular
dynamics.
Engineering Analysis – CFD (airflow simulation), structural analysis, crash
simulation.
AI and Machine Learning – Training huge models across compute clusters.
Financial Modeling – Risk analysis, Monte Carlo simulations, fraud detection.
Big Data Processing – Distributed graph analytics, bioinformatics, genomic
analysis.
Advantages of MPI
Highly scalable – Can run on hundreds or thousands of nodes efficiently.
Portable standard – Works on supercomputers, clusters, and cloud systems.
Fine control over communication – Developers can optimize data transfer.
Low overhead – Communication is very fast compared to other frameworks.
Supports heterogeneous systems – Can integrate CPUs, GPUs, and accelerators.
Limitations of MPI
Complex to program – Requires manual handling of communication patterns.
Difficult debugging – Errors in distributed systems are hard to detect.
Network dependency – Performance suffers if network latency is high.
Memory is not shared – Data must be explicitly passed between processes.
Not ideal for small systems – Overkill for simple or low-parallel tasks.
Dept Of. CSE (DS), AIET 15
Parallel Programming module 3
CONCLUSION
memory programming with MPI is one of the most powerful techniques used in modern
high-performance computing. It enables parallel execution on massive clusters and
supercomputers by allowing tasks to communicate through message passing. MPI
provides explicit control over communication, making it suitable for scientific
simulations, engineering models, AI training, finance, and many other domains requiring
large-scale computation. Although MPI programs are harder to write and debug, their
ability to scale across thousands of nodes makes them essential for real-time and
compute-intensive workloads. As data sizes grow and the need for high-speed
computation increases, MPI continues to be the backbone of distributed computing and
plays a vital role in modern research, industry, and technology.
Dept Of. CSE (DS), AIET 16