Distributed memory
programming with MPI
Module 3
• In this chapter, we’re going to start looking at how to program
distributed memory systems using message-passing.
• Recall that in message-passing programs, a program running on one
core-memory pair is usually called a process, and two processes can
communicate by calling functions: one process calls a send function
and the other calls a receive function.
• MPI is not a new programming language. It defines a library of
functions that can be called from C.
• We’ll learn about some of MPI’s different send and receive functions.
We’ll also learn about some “global” communication functions that
can involve more than two processes.
• we’ll also learn about some of the fundamental issues involved in
writing message-passing programs—like data partitioning and I/O in
distributed-memory systems.
MPI functions
• Instead of having each process simply print a message, we’ll
designate one process to do the output, and the other processes will
send it messages, which it will print.
• In parallel programming, it’s common for the processes to be
identified by nonnegative integer ranks. So if there are p processes,
the processes will have ranks 0,1,2,...,p−1.
• For our parallel “hello, world,” let’s make process 0 the designated
process, and the other processes will send it messages.
Compilation and execution
• The details of compiling and running the program depend on your
system, we’ll assume that we’re using a text editor to write the
program source, and the command line to compile and run.
$ mpicc −g −wall −o mpi_hello mpi_hello.c
• mpicc is a script that’s a wrapper for the C compiler, and it tells where
to find the necessary header files, and which libraries to link with the
object file. (includes debugging info (-g), enables all warnings (-Wall))
• Many systems also support program startup with mpiexec:
$ mpiexec −n <number of processes>./mpi_hello
• So to run the program with one process, we’d type
$ mpiexec −n 1 ./mpi_hello
• and to run the program with four processes, we’d type
$ mpiexec −n 4 ./mpi_hello
• With one process, the program’s output would be
Greetings from process 0 of 1!
• and with four processes, the program’s output would be
Greetings from process 0 of 4!
Greetings from process 1 of 4!
Greetings from process 2 of 4!
Greetings from process 3 of 4!
• How do we get from invoking mpiexec to one or more lines of
greetings? The mpiexec command tells the system to start instances
of our program.
• MPI programs
• The first thing to observe is that this is a C program. it includes the
standard C header files stdio.h and string.h. It also has a main
function, just like any other C program.
• mpi.h header file, This contains prototypes of MPI functions, macro
definitions, type definitions, and so on; it contains all the definitions
and declarations needed for compiling an MPI program.
• The second thing to observe is that all of the identifiers defined by
MPI start with the string MPI_.
• The first letter following the underscore is capitalized for function
names and MPI-defined types. All of the letters in MPI-defined
macros and constants are capitalized.
• MPI_Init and MPI_Finalize:
• The call to MPI_Init tells the MPI system to do all of the necessary
setup. For example, it might allocate storage for message buffers, and
it might decide which process gets which rank. No other MPI
functions should be called before the program calls MPI_Init. Its
syntax is-
• The arguments, argc_p and argv_p, are pointers to the arguments to
main, argc and argv. However, when our program doesn’t use these
arguments, we can just pass NULL for both.
• Like most MPI functions, MPI_Init returns an int error code.
MPI_SUCCESS (0), MPI_ERR_BUFFER, MPI_ERR_TAG, & MPI_ERR_COMM
• The call to MPI_Finalize tells the MPI system that we’re done using
MPI, and that any resources allocated for MPI can be freed. The
syntax is quite simple:
int MPI_Finalize(void);
• In general, no MPI functions should be called after the call to
MPI_Finalize. Thus a typical MPI program has the following basic
outline:
Communicators, MPI_Comm_size, and MPI_Comm_rank
• In MPI a communicator is a collection of processes that can send
messages to each other. One of the purposes of MPI_Init is to define
a communicator that consists of all of the processes started by the
user when he started the program.
• This communicator is called MPI_COMM_WORLD. The function calls
in Lines 13 and 14 are getting information about
MPI_COMM_WORLD. Their syntax is
• For both functions, the first argument is a communicator and has the
special type defined by MPI for communicators, MPI_Comm.
• MPI_Comm_size returns in its second argument the number of
processes in the communicator, MPI_Comm_rank returns in its
second argument the calling process’s rank in the communicator.
• SPMD programs:- a single program is written so that different
processes carry out different actions, and this can be achieved by
simply having the processes branch on the basis of their process rank.
• This approach to parallel programming is called single program,
multiple data or SPMD. The if−else statement in Lines 16–29 makes
our program SPMD.
• Communication:- In Lines 17–18, each process, other than process
0, creates a message it will send to process 0. (The function sprintf is
very similar to printf, except that instead of writing to stdout, it writes
to a string.)
• Lines 19–20 actually send the message to process 0. Process 0, on the
other hand, simply prints its message using printf, and then uses a for
loop to receive and print the messages sent by processes
1,2,...,comm_sz − 1.
• MPI_Send :- Each of the sends is carried out by a call to MPI_Send,
whose syntax is
• The first argument, msg_buf_p, is a pointer to the block of memory
containing the contents of the message. (greeting)
• The second argument, msg_size, determine the amount of data to be
sent. In our program, the msg_size argument is the number of
characters in the message, plus one character for the ’\0’ character
that terminates C strings.
• The third argument, MPI_CHAR defines the type of the message.
(char)
• The size of the message sent should be less than or equal to the
amount of storage in the buffer— in our case the string greeting. (31)
• The fourth argument, dest, specifies the rank of the process that
should receive the message. The fifth argument, tag, is a nonnegative
int. It can be used to distinguish messages that are otherwise
identical.
• For example, suppose process 1 is sending floats to process 0. Some
of the floats should be printed, while others should be used in a
computation.
• So process 1 can use (say) a tag of 0 for the messages that should be
printed and a tag of 1 for the messages that should be used in a
computation.
• The final argument to MPI_Send is a communicator. One of the most
important purposes of communicators is to specify communication
universes.
• A message sent by a process using one communicator cannot be
received by a process that’s using a different communicator.
• Message matching
• The parameters specified by the first three pairs of arguments,
send_buf_p/recv_buf_p, send_buf_sz/recv_buf_sz, and
send_type/recv_type, must specify compatible buffers.
• If recv_type = send_type and recv_buf_sz ≥ send_buf_sz, then the
message sent by q can be successfully received by r.
• It can happen that one process is receiving messages from multiple
processes, and the receiving process doesn’t know the order in which
the other processes will send the messages.
• For example, suppose process 0 is doing out work to processes
1,2,...,comm_sz − 1, and processes 1,2,...,comm_sz − 1, send their
results back to process 0 when they finish the work at irregular time
interval.
• If we follow the order to send the result, process like comm_siz-1 if
completes his work first then need to wait till the completion of the
other process.
• To avoid this problem MPI provides a special constant
MPI_ANY_SOURCE that can be passed to MPI_Recv.
• Then, if process 0 executes the following code, it can receive the
results in the order in which the processes finish:
• Similarly, it’s possible that one process can be receiving multiple
messages with different tags from another process, and the receiving
process doesn’t know the order in which the messages will be sent.
• For this circumstance, MPI provides the special constant
MPI_ANY_TAG that can be passed to the tag argument of MPI_Recv.
• Only a receiver can use a wildcard argument. Senders must specify a
process rank and a nonnegative tag. There is no wildcard for
communicator arguments; both senders and receivers must always
specify communicators.
The status_p argument:-
• MPI_Status is a struct with at least the three members MPI_SOURCE,
MPI_TAG, and MPI_ERROR.
• Suppose our program contains the definition MPI_Status status; Then
after a call to MPI_Recv, in which &status is passed as the last
argument, we can determine the sender and tags by examining the
two members:
• status. MPI_SOURCE
• status. MPI_TAG
• The amount of data that’s been received isn’t stored in a field that’s
directly accessible to the application program. However, it can be
retrieved with a call to MPI_Get_count.
• For example, suppose that in our call to MPI_Recv, the type of the
receive buffer is recv_type and, once again, we passed in &status.
• Then the call MPI_Get_count(&status , recv_type , &count) will
return the number of elements received in the count argument.