Parallel Programming module 2
GPU PROGRAMMING
GPU programming is an advanced computing paradigm that utilizes the massively
parallel architecture of Graphics Processing Units (GPUs) to accelerate computational
tasks that involve large-scale mathematical operations, vector processing, and repetitive
data operations. Originally designed for real-time rendering of graphics in games and
visualization systems, GPUs have evolved into powerful parallel processors capable of
executing thousands of threads simultaneously. Unlike CPUs, which are optimized for
low-latency single-threaded tasks, GPUs are optimized for high-throughput parallel
workloads. This shift has enabled significant performance improvements in fields such as
deep learning, scientific simulations, big data analytics, medical imaging, and real-time
rendering. GPU programming focuses on writing specialized programs that can leverage
this architecture using frameworks like CUDA, OpenCL, Vulkan compute shaders, and
recently high-level libraries in Python such as CuPy, RAPIDS, PyTorch, and TensorFlow.
The core reason GPUs are efficient is due to their architecture. A GPU consists of
thousands of simple, low-frequency compute cores grouped into Streaming
Multiprocessors (SMs). These cores execute the same instruction on multiple data
elements simultaneously, making GPUs a practical example of SIMD (Single Instruction
Multiple Data) and SIMT (Single Instruction Multiple Threads) architecture. The
performance of a GPU is often described by the throughput equation
Performance=Number of Cores×Clock Speed×Operations per Cycle
This highlights why GPUs outperform CPUs on data-parallel tasks: they have far more
cores that, even at lower clock frequencies, achieve higher total computational capability.
GPUs also rely on high-bandwidth memory such as GDDR6 or HBM2 to feed data
rapidly to these cores. Therefore, GPU programming involves efficiently mapping
computational tasks to thousands of threads while ensuring proper memory access
patterns, avoiding thread divergence, and maximizing occupancy.
A key concept in GPU programming is the grid-block-thread hierarchy introduced by
CUDA. A GPU kernel is a function that runs on the GPU and is launched with many
threads. Threads are grouped into blocks, and blocks together form a grid. The GPU
schedules these threads across available SMs. This model allows developers to divide a
problem into small parallel jobs that run independently. In mathematical terms, if a task
has NNN operations, and the GPU spawns TTT parallel threads, the execution time
ideally becomes
Tparallel=N/T
Dept Of. CSE (DS), AIET 7
Parallel Programming module 2
However, real-world performance also depends on memory latency, coalescing, thread
divergence, and instruction complexity. Therefore, GPU programming requires
algorithmic and architectural understanding to achieve optimal results.
Today, GPUs are widely used in artificial intelligence due to their ability to accelerate
matrix multiplication, convolution operations, and vector transformations. In deep
learning, a single forward pass of a neural network involves millions of multiplications.
For example, for matrix multiplication:
Ci,j=k=1∑nAi,k⋅Bk,j
GPUs compute these operations in parallel by assigning multiple threads to compute
different rows, columns, or elements of the matrix. Frameworks like PyTorch and
TensorFlow automatically utilize GPU programming internally, but developers can write
custom GPU kernels when needed.
PROGRAM 1
VECTOR ADDITION USING NUMBA CUDA
from numba import cuda
import numpy as np
@[Link]
def vector_add(a, b, c):
i = [Link](1)
if i < [Link]:
c[i] = a[i] + b[i]
# Input data
n = 10
a = [Link](n).astype(np.int32)
b = [Link](n, 0, -1).astype(np.int32)
c = [Link](n, dtype=np.int32)
threads_per_block = 32
blocks_per_grid = ([Link] + (threads_per_block - 1)) // threads_per_block
vector_add[blocks_per_grid, threads_per_block](a, b, c)
print("Array A:", a)
print("Array B:", b)
print("Result:", c)
OUTPUT
Array A: [0 1 2 3 4 5 6 7 8 9]
Array B: [10 9 8 7 6 5 4 3 2 1]
Dept Of. CSE (DS), AIET 8
Parallel Programming module 2
Result : [10 10 10 10 10 10 10 10 10 10]
APPLICATIONS
GPUs are widely used in deep learning for training neural networks, especially
convolutional and transformer models.
Scientific simulations involving physics, chemistry, and astronomy rely on GPUs
for solving large numerical equations.
Medical imaging applications such as CT/MRI reconstruction use GPUs for real-
time [Link] rendering, animation, VFX, and gaming heavily depend on
GPU acceleration.
Financial modeling, big data analytics, and high-frequency trading systems utilize
GPU-based parallel computation.
ADVANTAGES
Provides extremely high parallel computing power with thousands of cores
working simultaneously.
Greatly reduces computation time for large matrix, vector, and numerical
operations.
Offers high memory bandwidth, enabling fast data movement for real-time tasks.
Ideal for scalable machine learning, especially for training large models.
Reduces CPU load, enabling better overall system performance and multitasking.
LIMITATIONS
Difficult to program due to the need to understand thread scheduling, memory
hierarchy, and parallel algorithms.
Not all tasks can be parallelized; sequential tasks still depend on CPU
performance.
Dept Of. CSE (DS), AIET 9
Parallel Programming module 2
GPU memory is limited compared to CPU RAM, restricting extremely large
datasets.
Improper thread or memory usage can lead to performance loss instead of
improvement.
High-performance GPUs are expensive and require adequate cooling and power.
Real-Time Examples Of Gpu Programming
Medical Imaging
o GPUs accelerate CT and MRI reconstruction.
o Millions of pixels/voxels are processed in parallel.
o Results appear within seconds instead of minutes.
Autonomous Vehicles
o GPUs process camera images, LIDAR, and radar data in real time.
o Used for pedestrian detection, obstacle recognition, and lane tracking.
o Enables fast decision-making for safe driving.
Weather Forecasting
o GPUs run large climate and atmospheric simulations.
o Reduces model execution from days to hours.
o Handles huge datasets like wind, pressure, and humidity maps.
Finance and Stock Market Analysis
o GPUs run Monte Carlo simulations and risk analysis.
o Used for fraud detection and price prediction.
o Allows real-time computations needed for trading.
Mobile Phone Image Processing
o GPUs handle filters, night mode, and photo enhancement.
o Performs convolution on millions of pixels instantly.
o Improves camera quality and speed.
Gaming and Animation
o GPUs render millions of polygons, shadows, and lighting effects.
o Essential for real-time 3D graphics.
o Used in movie studios for fast rendering.
Deep Learning and AI
o GPUs accelerate neural network training using tensor/matrix operations.
o Frameworks like TensorFlow rely on GPU computation.
o Reduces training time from weeks to hours.
Robotics and Drones
Dept Of. CSE (DS), AIET 10
Parallel Programming module 2
o GPUs process real-time video, sensor data, and object tracking.
o Used for navigation, path planning, and obstacle avoidance.
o Helps robots respond quickly and accurately.
CONCLUSION
GPU programming represents the future of high-performance computing due to its ability
to process huge amounts of data in parallel. With the rise of AI, robotics, simulations, and
big data, the need for GPU-accelerated computation has grown massively. By
understanding GPU architecture, memory hierarchy, parallel thread execution, and
frameworks like CUDA or OpenCL, developers can build highly efficient applications
that outperform CPU-based systems. The ability to run thousands of threads in parallel
makes GPUs indispensable in modern computing.
Dept Of. CSE (DS), AIET 11