Distributed Computing
Lab3
2025
Agenda
• Introduction to Parallel Programming
• Shared memory
• Distributed memory
• HPC Interfaces
• MPI (Distributed memory)
Introduction to Parallel Programming
Shared Memory
• Multiple processor can operate independently but share the same
memory resources.
• Changes in a memory location effected by one processor are visible to
all other processors. CPU CPU CPU
Cache Cache Cache
memory memory memory
Bus
Primary I/O
Memory Devices
Shared Memory
• OpenMP is an Application Program Interface (API) for shared memory
programming
• You can expect OpenMP to be supported by all compilers on all HPC platforms
Distributed Memory
• Processors have their own local memory.
• Memory addresses in one processor do not map to another
processor.
• Each processor can operate independently.
• Access to the data in another processor is done via communication
Multi-Processors Systems
• MPI is a portable library used for writing parallel programs using the
message passing model
• You can expect MPI to be available on any HPC platform you use
• In message-passing all the
What Explicit Parallelism
• It is up to the developer to design the parallel decomposition and
implement it
• How will you divide up the problem?
• When will you need to communicate between processes?
• MPI parallelism is explicit
• OPENMP Parallelism is less explicit than (Distributed memory) MPI
• You specify which parts of the program you want to parallelize, and the
compiler produces a parallel executable
Hybird
• Hybrid Distributed-Shared Memory
Our Scope
• MPI
Message-Passing Programming
• Abstraction of Multi-Computers system.
• MPI (Message Passing Interface) is the standard programming
interface, that defines a set of functions that allow a programmer to
instruct their code to execute tasks in parallel.
Message-Passing Programming
• Similar to task/channel model where.
• Process = Task; Interconnection = implicit channel between every two
processes.
• The number of active processes remains constant.
• Each process executes the same program.
• Each process is assigned a unique ID (rank).
• Each process may follow different execution path.
• Send/receive to communicate and synchronize.
Basics
Initialization
• All MPI programs must begin with a call to
MPI_Init(argc,argv ) //Initialize the MPI execution environment
• And close with a call to
MPI_Finalize( ) // Terminates MPI execution environment
Basics
Size and rank
• Get how many processes are running in a given communicator,
int size;
MPI_Comm_size(MPI_COMM_WORLD, &size);
• and the rank of the calling process within that communicator.
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
Hello World!
Thank You ☺