Module -2
GPU programming, Programming hybrid systems
Course Outcome:
At the end of the course, the student will be able to demonstrate
parallelism in MIMD system
Topic
●
GPU Programming
●
I/O in MIMD Systems
●
Performance
– Speedup and efficiency in MIMD Systems
– Amdahl’s Law
– Scalability in MIMD Systems
GPU Programming
●
GPUs are not “standalone” Processors, i.e., they don’t work
independently
●
GPUs don’t run OS or system services
●
GPUs itself may have multiple processors, each capable of running
hundreds of threads
●
GPUs usually have their own memory, shared among the processors
●
The GPU processors also has a small block of memory that can only
be accessed by threads running on that processor
GPU Programming
●
Programming GPU also involves writing code for CPU
●
The code that runs on CPU typically allocates and initializes
storage on both CPU and GPU
●
CPU delegates the program execution to GPU
●
CPU is responsible for outputting the result of GPU
●
Thus, GPU programming is “Heterogenous” programming, i.e.,
it involves two different types of processors
I/O in MIMD Systems
●
Multiple processes/threads can access STDOUT, STDERR and
STDIN
●
This leads to input or output sequence that is nondetermininstic
●
Output may appear in different order each time the program is run
OR
●
Output of one process/thread may be broken by output of another
process/thread
●
Input read by each process/thread may be different on each run
I/O in MIMD Systems
●
To address these issues, following assumptions are made:
Performance
●
The purpose of writing parallel programs is to get increased
performance
●
How to measure performance?
●
Two parameters used to measure performance are:
– Speedup
– Efficiency
Performance
●
Speedup:
– Rate of increase in speed of execution
– ‘Linear speedup’ is achieved when parallel program does not introduce
any additional work other than execution
– For linear speedup, S = p, where ‘p’ is the number of processes/threads,
since
– In practice, linear speedup cannot be achieved because multiple
process/threads introduce overhead
Performance
●
Efficiency:
– Fraction of the parallel run-time that is spent by
each core on solving the problem
– Since parallel programs always introduce
overhead
–
Performance
Speedups and efficiencies of parallel program on
different problem sizes.
Performance
Speedups of parallel program on different problem sizes.
Performance
Efficiencies of parallel program on different problem sizes.
Performance
●
Amdahl’s Law:
– It specifies the relation between speedup and ‘serial’ part of a parallel program
– It says that unless virtually all of a serial program is parallelized, the possible
speedup is going to be very limited
– More generally, if a fraction ‘r’ of our serial program remains unparallelized, we
can’t get a speedup better than 1/r
– For eg., if unparallelized part, r = 0.1 (i.e, 10%), then we couldn’t get a speedup
better than 10
Performance
●
Scalability in MIMD Systems:
– A program is said to be scalable, if after increasing the number of
processes/threads, we can get same efficiency ‘E’
– Strongly scalable: if we can keep the same efficiency ‘E’ after increasing
the number of processes/threads without increasing the problem size
– Weakly scalable: if we can keep the same efficiency ‘E’ after increasing
the problem size at the same rate as the increase in the number of
processes/threads