|| Jai Sri Gurudev||
Sri Adichunchanagiri Shikshana Trust(R)
SJB INSTITUTE OF TECHNOLOGY
BGS Health & Education City, Dr. Vishnuvardhan Road, Kengeri, Bengaluru–560060
An Autonomous Institute under Visvesvaraya Technological University, Belagavi
Affiliated to Visvesvaraya Technological University, Belagavi & Approved by AICTE, New Delhi, Certified by ISO 9001-2015
Accredited by NBA & NAAC, New Delhi with ‘A+’ Grade, Recognized by UGC, New Delhi with 2(f) and 12(B)
Notes On
Course Name: Parallel Computing
Course Code: B C S 7 0 2
Module – 3
By
Faculty Name: Dr Ranjith J , Prof. Kiran Kumar V , Prof. Veeresh K M
Semester: VIIth Semester
Department of Information Science & Engineering
Aca. Year ODD SEM /2025-26
MODULE3
DISTRIBUTED MEMORY PROGRAMMING WITH MPI
1. GettingStartedusingMPI
DistributedMemorySystems-MPIprogramming
• Inmessage-passingprograms,aprogramrunningononecore-memorypairisusuallycalleda
process.
• Twoprocessescancommunicatebycallingfunctions:
– oneprocesscallsasendfunction
– andtheothercallsareceive function.
• Theimplementationofmessage-passingcanbedoneusingMessage-PassingInterface(MPI).
• MPIdefinesalibraryoffunctionsthatcanbecalledfromC,C++,andFortranprograms.
• InParallelProgrammingcommonpracticetoidentifyprocessesbynonnegativeintegerranks.
• pprocessesarenumbered0,1,2,..p-1
• Oneprocesstodotheoutput,andtheotherprocesseswillsenditmessages,which itwillprint
Example: Hello World in MPI
Compile:
mpicc-g-Wall-ompi_hellompi_hello.c
Execution:
mpiexec-n<numberofprocesses><executable>
mpiexec-n4./mpi_hello
• Writtenin C.
– Has main.
– Usesstdio.h,string.h,etc.
• [Link].
• IdentifiersdefinedbyMPIstartwith“MPI_”.
• Firstletterfollowingunderscoreisuppercase.
– ForfunctionnamesandMPI-definedtypes.
– Helpstoavoidconfusion.
2. MPIComponents(MPI_Init)
MPI_Init
– TellsMPItodoallthenecessary setup.
– Itmightallocatestorageformessage buffers,
– Itmightdecidewhichprocessgetswhichrank.
– nootherMPIfunctionsshouldbecalledbeforetheprogramcallsMPI Init.
MPI_Finalize
– TellsMPIthatanyresourcesallocatedforMPIcanbefreed.
– noMPIfunctionsshouldbecalledafterthecalltoMPIFinalize.
– It’salsonotnecessarythatthecallstoMPIInitandMPIFinalizebeinmainfunction.
CommunicatorsMPI_Comm_size&MPI_Comm_rank
• InMPIacommunicatorisacollectionofprocessesthatcansendmessagestoeachother.
• MPI_Initdefinesacommunicatorthatconsistsofalltheprocessescreatedwhentheprogramis
started.
• Itiscalled MPI_COMM_WORLD.
Communication-MPI_Send()
• msg_buf_p-isapointertotheblockofmemorycontainingthecontentsofthemessage.
– Intheprogram,thisisjustthestringcontainingthemessage,greeting.
• Thesecondandthirdarguments,msg_sizeandmsg_type,determinetheamountofdatatobe sent.
• msg_size-isthenumberofcharactersinthemessageplusonecharacterforthe‘\0’character that
terminates C strings.
• msg_type-[Link]
contains strlen(greeting)+1 chars.
• dest-specifiestherankoftheprocessthatshouldreceivethemessage.
• tag-isanonnegativeint.
• [Link],suppose
[Link],whileothers should
be used in a computation.
• ThenthefirstfourargumentstoMPISendprovidenoinformationregardingwhichfloatsshould be
printed and which should be used in a computation.
• Soprocess1canuse,say,atagof0forthemessagesthatshouldbeprintedandatagof1for the
messages that should be used in a computation.
MPIDatatypes:
• [Link]
communication have a communicator argument.
• Oneofthemostimportantpurposesofcommunicatorsistospecifycommunicationuniverses.
• acommunicatorisacollectionofprocessesthatcansendmessagestoeachother.
• Conversely,amessagesentbyaprocessusingonecommunicatorcannotbereceivedbya
process that’s using a different communicator.
• Since MPI provides functions for creating new communicators, this feature can be used in
complexprogramstoinsurethatmessagesaren’t“accidentallyreceived”inthewrongplace.
Communication–MPI_Recv()
• Firstthreeargumentsspecifythememoryavailableforreceivingthemessage:
– msg_buf_ppointstotheblockofmemory,
– buf_sizedeterminesthenumberofobjectsthatcanbestoredinthe block,
– buf_typeindicatesthetypeoftheobjects.
• Thenextthreeargumentsidentifythemessage.
– source-specifiestheprocessfromwhichthemessageshouldbe received.
– tag-shouldmatchthetagargumentofthemessagebeingsent,
– communicator-mustmatchthecommunicatorusedbythesendingprocess.
SemanticsofMPI_SendandMPI_Recv
whenwesendamessagefromoneprocessto another
• [Link],itwilladdthe“envelope”
information to the actual data being transmitted.
• Thedestinationprocessrank,thesendingprocessrank,thetag,thecommunicator,andsome
information on the size of the message.
• Oncethemessagehasbeenassembled,thereareessentiallytwopossibilities:thesending
process can buffer the message or it can block.
• Ifitbuffersthemessage,theMPIsystemwillplacethemessage(dataandenvelope)intoits own
internal storage, and the call to MPI Send will return.
• Ifthesystemblocks,itwillwaituntilitcanbegintransmittingthemessage,andthecallto
MPI_Send may not return immediately.
• IfweuseMPI_Send,whenthefunctionreturns,wedon’tactuallyknowwhetherthemessage has
been transmitted. We only know that the storage we used for the message.
• Ifthesizeofamessageisless thanthecutoff,[Link] greater
than the cutoff, MPI Send will block.
• Unlike MPI Send, MPI_Recv always blocks until a matching message has been received. Thus,
whenacalltoMPI_Recvreturns,weknowthatthereisamessagestoredinthereceivebuffer (unless
there’s been an error).
• Thereisanalternatemethodforreceivingamessage,inwhichthesystemcheckswhethera
matching message is available and returns, regardless of whether there is one.
• MPIrequiresthatmessagesbenonovertaking.
• Thismeansthatifprocessqsendstwomessagestoprocessr,thenthefirstmessagesentbyq must be
available to r before the second message.
• However,thereisnorestrictiononthearrivalofmessagessentfromdifferentprocesses.
• That is, ifq and t bothsendmessages tor, theneven ifq sends itsmessagebeforet sendsits
message,thereisnorequirementthatq’smessagebecomeavailabletorbeforet’smessage.
ParallelizingtheTrapezoidalRule
• Twotypesoftask
– onetypeisfindingtheareaofasingle trapezoid.
– theotheriscomputingthesumoftheseareas.
• Thenthecommunicationchannelswilljoineachofthetasksofthefirsttypetothesingletaskof the
second type.
• splittheinterval[a,b]upintocommsz subintervals.
• Ifcommszevenlydividesn,thenumberoftrapezoids,wecansimplyapplythetrapezoidalrule with
n=comm sz trapezoids to each of the comm sz subintervals.
• Tofinish,wecanhaveoneoftheprocesses,sayprocess0,addtheestimates.
3. Collectivecommunication
• Eachprocesswithrankgreaterthan0is,ineffect,saying“addthisnumberintothe
total(trapezoidal rule).”
• Process0 isdoingnearlyalltheworkincomputingtheglobalsum,whiletheotherprocessesare doing
almost nothing
communicationfunctionsthatinvolvealltheprocessesinacommunicatorarecalledcollective
communications
Atree-structuredCommunicationglobalsum
Tree-structuredcommunication
1. Inthefirst phase:
(a) Process1sendsto0,3sendsto2,5sendsto4,and7sendsto6.
(b) Processes0,2,4,and6addinthereceivedvalues.
(c) Processes2and6sendtheirnewvaluestoprocesses0and4,respectively.
(d) Processes0and4addthereceivedvaluesintotheirnewvalues.
2. (a)Process4sendsitsnewestvaluetoprocess 0.
(b)Process0addsthereceivedvaluetoitsnewestvalue.
MPI_Reduce
Example
Predefinedreductionoperatorsin MPI
MPI_Allreduce
• Usefulinasituationinwhichalloftheprocessesneedtheresultofaglobalsuminorderto complete
some larger computation.
ButterflyStructured
Broadcast
Get_inputthatuses MPI_Bcast
Scatter
• MPI_Scattercanbeusedinafunctionthatreadsinanentirevectoronprocess0butonlysends the
needed components to each of the other processes.
Gather
• Collectallofthecomponentsofthevectorontoprocess0,andthenprocess0canprocessallof the
components.
Allgather
• Concatenatesthecontentsofeachprocess’send_buf_pandstoresthisineachprocess’
recv_buf_p.
• Asusual,recv_countistheamountofdatabeingreceivedfromeach process.
4. [Link]-to-PointCommunications
Point-to-pointcommunications Collectivecommunications
• Point-to-pointcommunicationsare • Collectivecommunicationsdon’tusetags.
matched on the basis of tags and
• They’rematchedsolelyonthebasisofthe
communicators.
communicator and the order in which
they’recalled.
Eachprocesscommunicatesusingsendand receive • Alltheprocessesinthecommunicator
function mustcallthesamecollectivefunction.
For example, a program that attempts to match a
call to MPI_Reduce on one process with a call to
MPI_Recvonanotherprocessiserroneous,and,in
alllikelihood,theprogramwillhangorcrash
Thecorrespondingdestinationaloneisused • Theoutput_data_pargumentisonlyused
on dest_process.
• However,alloftheprocessesstillneedto
passinanactualargumentcorresponding to
output_data_p, even if it’s just NULL.
TheargumentspassedbyeachprocesstoanMPI • Theargumentspassedbyeachprocessto
communication may vary with each process anMPIcollectivecommunicationmustbe
“compatible.”
• For example, if one process passes in 0 as
the dest_process and another passes in 1,
then the outcome of a call to MPI_Reduce
iserroneous,and,onceagain,theprogram is
likely to hang or crash.
5. LibraryFunctions
Libraryfunctionslistedbelowcanbeexplainedwithanexample.
1. MPI_Init()
2. MPI_Comm_size()
3. MPI_Comm_rank()
4. MPI_Finalize()
5. MPI_Send()
6. MPI_Receive()
7. MPI_Reduce()
8. MPI_Allreduce()
9. MPI_Bcast()
10. MPI_Scatter()
11. MPI_Gather()
12. MPI_Allgather()
13. MPI_Address()
#include<mpi.h>
intMPI_Address(void*location,MPI_Aint*address)
Description
[Link](byte)address of
location.
14. MPI_Barrier()
#include<mpi.h>
intMPI_Barrier(MPI_Commcomm) MPI_Get_version
Description
AnMPIbarriercompletesafterallgroupmembershaveenteredthebarrier. #include
<mpi.h>
intMPI_Get_version(int*version,int*subversion)
Description
Since OpenMPIisMPI3.1compliant,thisfunctionwillreturnaversionvalueof3andasubversionvalue of 1 for
this release.
15. MPI_Wait
#include<mpi.h>
intMPI_Wait(MPI_Request*request,MPI_Status*status)
InputParameter
Request-Request(handle).
OutputParameters
Status-Statusobject(status).
Description
AcalltoMPI_Waitreturnswhentheoperationidentifiedbyrequestiscomplete.
PARTB–16 Marks
1. MPISendandReceive
2.1to2.7
2. PointtoPointCommunication
to2.7
3. CollectiveCommunication
to3.8
4. LibraryFunctions
Chapter5
PART A
1. Differencebetweenpointtopointandcollectivecommunication.
2,Syntax
[Link]
[Link]