VII Semester

Course: Parallel Computing

Course Code: BCS702
Credits: 03& 2022 Scheme

Module 2: GPU Programming

.IN
Course Instructors
K PRASANTH
Assistant Professor, Dept. of CSE,
C
EPCET
N
[Link]@[Link]
SY

Department of Computer Science & Engineering, EPCET

U
VT

Studied smart, not hard — thanks to [Link]

 Module – 02
GPU programming

GPU programming

GPU programming is fundamentally heterogeneous programming, involving the coordination of two distinct
types of processors: the CPU “host” system and the GPU itself.

This is because GPUs are typically not standalone processors; they don’t run their own operating systems or
directly access secondary storage.

.IN
Here’s a breakdown of the key aspects of GPU programming:

1. CPU Host’s Role: C

N
SY

The CPU host is responsible for the overall control and coordination.

It allocates and initializes memory on both the CPU (host memory) and the GPU (device memory), which are
U

usually separate.
VT

It initiates the execution of programs on the GPU.

It handles the output and retrieval of results from the GPU program.

2. GPU Architecture:

A GPU contains one or more processors.

Each of these processors is designed to run hundreds or thousands of threads concurrently.

Processors typically share a large block of global memory.

Critically, each individual processor also has a small block of much faster memory, often referred to as a
Studied smart, not hard — thanks to [Link]
programmer-managed cache. This faster memory is only accessible by threads running on that specific
processor.

3. Thread Organization and Execution (SIMD Groups):

Threads running on a GPU processor are typically organized into groups, often called SIMD (Single Instruction,
Multiple Data) groups.

Within a SIMD group: Threads generally operate under the SIMD model. While they may not execute in strict

.IN
“lockstep” (i.e., not all execute the exact same instruction at the precise same time), no thread in the group will
move to the next instruction until all threads in the group have completed the current instruction.

C
Across different SIMD groups: Threads in different groups can run largely independently.
N
SY

4. Branching and Idling Threads:

A significant challenge in SIMD execution is branching (e.g., if-else statements) where threads within the same
group take different execution paths.
U

Example:
VT

C
// Thread private variables
int rank_in_gp, my_x;

my_x += 1;

else
my_x -= 1;

If there are 32 threads in a SIMD group, threads with rank_in_gp < 16 will execute the my_x += 1 branch, while
threads with rank_in_gp >= 16 will be idled. Once the first set of threads finishes, the roles are reversed: the
rank_in_gp < 16 threads are idled, and the rank_in_gp >= 16 threads execute my_x -= 1.

Studied smart, not hard — thanks to [Link]

Efficiency Impact: Idling half of the threads for portions of execution is inefficient. Therefore, GPU
programmers must strive to minimize branching within SIMD groups.

5. Hardware Scheduling and Maximizing Resource Use:

- GPUs utilize a hardware scheduler (unlike CPUs, which typically use software schedulers). This hardware
scheduler operates with very low overhead.
- The scheduler's goal is to execute an instruction when all threads in a SIMD group are ready.
- To maximize hardware utilization, it's common practice to create a large number of SIMD groups.
- This provides the scheduler with more options: if some SIMD groups are not ready (e.g., waiting for data
from memory or a previous instruction), they can be idled, and the scheduler can pick another ready group to
execute, thereby keeping the GPU busy.

.IN
Programming hybrid systems

C
When programming systems composed of clusters of multicore processors, a common approach for achieving
N
high performance is to combine different programming models:
SY

- Shared-memory API: Used for communication and data sharing within a single node (i.e., among the cores
of a single multicore processor).
U

- Distributed-memory API: Used for communication and data exchange between different nodes in the cluster.
VT

This combination is often referred to as a "hybrid" API programming model.

Why use a hybrid API?

The primary motivation for employing a hybrid API is to achieve the highest possible levels of performance. By
utilizing the shared memory model within a node, programs can exploit faster communication paths (e.g.
directly accessing shared memory) compared to the overheads of inter-node communication.

Challenges of Hybrid API Programming:

Despite the performance benefits, hybrid API programming introduces significant complexity, making program
development much more difficult. This increased complexity arises from:

- Managing two distinct programming models: Developers must understand and effectively integrate both
Studied smart, not hard — thanks to [Link]
shared-memory and distributed-memory paradigms.
- Data placement and movement: Carefully orchestrating data movement between shared memory within a node
and distributed memory across nodes becomes crucial for performance.
- Synchronization: Ensuring correct synchronization both within and across nodes adds layers of complexity.

Alternative Approach:

- Given the challenges, such systems are often programmed using a single, unified distributed-memory API for
both inter-node and intra-node communication.
- While this might not always yield the absolute peak performance achievable with a finely-tuned hybrid
approach, it significantly reduces program development complexity by providing a consistent communication

.IN
model across the entire cluster.
- In essence, the choice between a hybrid API and a purely distributed-memory API for clusters of multicore

C
processors is a trade-off between maximizing performance and managing development complexity.
- Hybrid APIs are typically reserved for applications where squeezing out every last bit of performance is
N
paramount.
SY

MIMD systems
MIMD Systems: Navigating Input/Output Challenges
U

When discussing MIMD (Multiple Instruction, Multiple Data) systems, the topic of input/output (I/O) often
presents unique challenges compared to serial programming.
VT

While parallel I/O is a complex field in itself, the vast majority of parallel programs for these systems perform
relatively little I/O, often relying on standard C functions like printf, fprintf, scanf, and fscanf.

However, even this limited use can lead to significant issues due to the inherent nature of parallel execution.

I/O Challenges in MIMD Systems:

1. Standard C's Ambiguity for Processes: Standard C is a serial language, and its specifications do not define the
behavior of I/O functions when called by multiple independent processes. This means their behavior in a
multiprocess environment can vary significantly across different systems.
2. Nondeterministic Outcomes for Threads: While threads forked by a single process do share stdin, stdout, and
stderr, simultaneous access by multiple threads to these shared resources leads to nondeterministic outcomes.
It's impossible to predict the exact order interleaving of operations.
3. Output to Console (printf, fprintf):
Studied smart, not hard — thanks to [Link]
 - Developer Expectation: Developers usually want all output to appear on the console of the single system
where the program is initiated.
- System Behavior: Most systems indeed coalesce output to a single console. However, there's no guarantee.
Some systems might restrict stdout or stderr access to only one process, or even none.
- Nondeterminism: When multiple processes/threads can access stdout, the sequence of output is usually
nondeterministic. Data from different processes/threads might interleave unexpectedly, making it hard to read or
debug.
4. Input from Console (scanf, fscanf):
- Ambiguity: It's less obvious how scanf should behave with multiple processes/threads. Should input be
divided? Should only one be allowed to read?

.IN
- System Behaviour: The majority of systems allow at least one process (commonly process 0) to call scanf.
Some, and most allow multiple threads to call scanf. However, some systems might not allow any processes to
call scanf.
C
- Non-determinism: When multiple processes/threads read from stdin, the data read by such an I/O on different
N
runs, even with identical input, due to race conditions.
SY

Assumptions and Rules for I/O in Parallel Programs:

- To manage these potential issues and ensure predictable (or at least understandable) I/O behavior in parallel
U

programs, the following conventions are often adopted:

VT

Standard Input (stdin) Access:

- Distributed-Memory Programs: Only process 0 will access stdin.

- Shared-Memory Programs: Only the master thread or thread 0 will access stdin.
- Rationale: This constrains input, avoiding nondeterminism and ensuring a single, clear source for input data.

Standard Output (stdout) and Standard Error (stderr) Access:

- Both Distributed and Shared-Memory Programs: All processes/threads can access stdout and stderr.
- However, for most cases: Only a single process/thread (e.g. process 0 or the master thread) will be used for all
intended output to stdout.
- Exception: Debugging prints to stdout or stderr can be done by all processes/threads. This is acceptable
because such output is for debugging/observation means, not for interpreted output.
- Rationale: This completely avoids race conditions and interleaving when dealing with file I/O.
Studied smart, not hard — thanks to [Link]
File I/O (other than stdin/stdout/stderr):
- Only a single process/thread will attempt to access any specific file.
- This means each process/thread opens its own private file for reading or for writing.
- Rationale: This completely avoids race conditions and interleaving when dealing with file I/O.
Debug Output Best Practice:
- Debug output should always include the rank or ID of the process/thread generating the output. This is crucial
for distinguishing messages and understanding the flow of execution from different parallel entities, especially
when output is interleaved.

GPUs

.IN
GPUs and I/O: Host-Centric Approach with Debugging Exceptions

C
In GPU programming, the CPU host code is generally responsible for all input/output (I/O) operations. This
N
simplifies I/O management significantly for a few key reasons:
SY

- Typically, only one process or thread runs on the CPU host to manage the GPU computations. This means that
the standard C I/O functions (like printf, scanf, fprintf, fscanf) behave exactly as they would in a regular serial C
U

program, without the complexities of nondeterministic behavior found in multi-process/multi-thread CPU

environments
VT

GPU Limitations:

- GPUs are specialized for parallel computation and generally do not have direct access to standard I/O streams
or secondary storage (like hard drives). This offloads I/O responsibilities entirely to the host.

Exception for Debugging:

- stdout for GPU Threads: In the systems commonly used for GPU programming, individual GPU threads can
write to stdout.
- Nondeterministic Output: Similar to MIMD programs running on CPUs, when multiple GPU threads write to
stdout concurrently, the order of the output is nondeterministic. This means output can vary from one run to the
next.
- No Access to Other Streams/Storage: Crucially, GPU threads typically do not have access to stderr, media, or
Studied smart, not hard — thanks to [Link]
secondary storage. This underscores the host's role for all but very specific debugging output.
Performance - Speedup and efficiency in MIMD systems

The primary goal of writing parallel programs is to achieve increased performance. When evaluating the
performance of homogeneous MIMD systems (where all cores have the same architecture, unlike GPUs), we
use metrics like speedup and efficiency.

.IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]

 .IN
C
N
SY
U
VT

Studied smart, not hard — thanks to [Link]