Distributed Computing
Lab4
2024
Agenda
• MPI (Distributed memory)
• Point-to-Point Communication
• MPI Send/Receive vs MPI I_send/I_Receive
• Point-to-Point Communication Blocking Deadlock Cases
• Hands-on
Point to Point communication
• Is the most basic form of communication in MPI, allowing a program
to send a message from one process to another over a given
communicator.
• There are two kinds of communication for sending and receiving
message via MPI which are Blocking and Non-Blocking.
Send & Receive process
Point-to-Point Communication
• MPI_Send • MPI_Recv
(const void *buf, (const void *buf,
int count, int count,
MPI_Datatype datatype, MPI_Datatype datatype,
int dest, int source,
int tag, int tag,
MPI_Comm comm) MPI_Comm comm,
MPI_STATUS stat)
MPI_Send(&data, 1, MPI_Status stat;
MPI_INT,Receiver, 0, MPI_Recv(&data, 1, MPI_INT,
MPI_COMM_WORLD); sender, 0, MPI_COMM_WORLD,
&stat);
Send & Receive process Blocking
Send & Receive process Non-blocking
Deadlocks
• In bidirectional communication you have to be careful about deadlocks.
• When a deadlock occurs, processes involved in the deadlock will not proceed
any further. Deadlocks can take place either due to:
– the incorrect order of send and receive.
– or the limited size of the system buffer.
Deadlocks
• There are essentially three cases depending on the order of send and receive
subroutines called by both processes.
1. Case 1 Both processes call the send subroutine first, and then receive.
2. Case 2 Both processes call the receive subroutine first, and then send.
3. Case 3 One process calls send and receive subroutines in this order, and the other
calls in the opposite order.
• For each case, there are further options based on your use of blocking or non-
blocking subroutines
Case 1
IF (myrank==0) THEN
CALL MPI_SEND(sendbuf, ...)
CALL MPI_RECV(recvbuf, ...)
ELSEIF (myrank==1) THEN
CALL MPI_SEND(sendbuf, ...)
CALL MPI_RECV(recvbuf, ...)
ENDIF
• As long as the system buffer is larger than sendbuf, the program ends normally
Case 1- Free of deadlock
IF (myrank==0) THEN
CALL MPI_ISEND(sendbuf, ..., ireq, ...)
CALL MPI_RECV(recvbuf, ...)
CALL MPI_WAIT(ireq, ...)
ELSEIF (myrank==1) THEN
CALL MPI_ISEND(sendbuf, ..., ireq, ...)
CALL MPI_RECV(recvbuf, ...)
CALL MPI_WAIT(ireq, ...)
ENDIF
Case 2
IF (myrank==0) THEN
CALL MPI_RECV(recvbuf, ...)
CALL MPI_SEND(sendbuf, ...)
ELSEIF (myrank==1) THEN
CALL MPI_RECV(recvbuf, ...)
CALL MPI_SEND(sendbuf, ...)
ENDIF
• code leads to a deadlock regardless of how much system buffer you have.
Case 2 – Free of deadlock
IF (myrank==0) THEN
CALL MPI_IRECV(recvbuf, ..., ireq, ...)
CALL MPI_SEND(sendbuf, ...)
CALL MPI_WAIT(ireq, ...)
ELSEIF (myrank==1) THEN
CALL MPI_IRECV(recvbuf, ..., ireq, ...)
CALL MPI_SEND(sendbuf, ...)
CALL MPI_WAIT(ireq, ...)
ENDIF
Case 3
IF (myrank==0) THEN
CALL MPI_SEND(sendbuf, ...)
CALL MPI_RECV(recvbuf, ...)
ELSEIF (myrank==1) THEN
CALL MPI_RECV(recvbuf, ...)
CALL MPI_SEND(sendbuf, ...)
ENDIF
• It is always safe to order the calls of MPI_(I)SEND and MPI_(I)RECV so that a send subroutine
call at one process and a corresponding receive subroutine call at the other process appear in
matching order.
• In this case, you can use either blocking or non-blocking subroutines.
Array Sum
Distribute 15 Element Array to 3 processes, each calculate a partial sum of 5 elements and
then the master calculate the final sum
Solution : Array Sum
Thank You ☺