Introduction to
CUDA
Programming
Presented by: [Vikas Kumar TIwari]
What is CUDA?
CUDA (Compute Unified Device
Architecture) is a parallel computing
platform by NVIDIA.
Allows developers to use GPU power
for general-purpose computing
(GPGPU).
Why Use CUDA?
GPUs contain thousands of lightweight
cores designed for parallel execution, making
them ideal for compute-intensive tasks such as:
Image and video processing
Machine learning and deep learning
Scientific simulations
Signal processing
Compared to CPUs (which have a few powerful
cores), GPUs excel at executing the same
operation on large datasets
simultaneously.
CUDA Architecture
Overview
CUDA Cores: Small processing units in
the GPU.
Streaming Multiprocessors (SMs):
Groups of CUDA cores executing
instructions in parallel.
Global, shared, and local memory.
Grid, blocks, and threads hierarchy:
Grid
└── Block
└── Thread
CUDA Programming Model
Host (CPU) and Device (GPU) model.
Functions executed on the device are
called kernels.
Kernels are launched with a grid of
thread blocks.
CUDA Compilation Flow
1. Write CUDA Code (CUDA C/C++)
2. Compile using NVCC (NVIDIA CUDA
Compiler)
3. Generate PTX (Parallel Thread
Execution) Code
4. PTX converted to SASS (Streaming
Assembler) architecture-specific
machine code for execution
5. Kernel execution on GPU
CUDA Compilation Flow
CUDA Compilation Flow
When you compile a CUDA program using nvcc, the following
steps occur:
[Link] Source Code (.cu file)
Your CUDA C++ program contains host and device code.
[Link] to PTX (.ptx file)
The nvcc compiler compiles CUDA kernels into PTX code
(an intermediate representation).
[Link] to SASS Conversion
The PTX code is then compiled into SASS (machine
code) using Just-In-Time (JIT) compilation by the GPU
driver at runtime.
This step ensures that the code is optimized for the
specific GPU architecture where it runs.
[Link] on GPU
The GPU executes the SASS (binary instructions) to
perform computations.
CUDA Execution Flow
Host (CPU) initializes data and
transfers it to Device (GPU).
CUDA Kernel executes on the GPU.
Results are transferred back to the
CPU.
Synchronization ensures correct
execution.
Device Memory
Allocation
Global Memory: Large but slow,
accessible by all threads.
Shared Memory: Faster, shared among
threads in a block.
Local Memory: Private to each thread.
Texture/Constant Memory: Optimized for
specific use cases.
CUDA Memory
Management
cudaMalloc() – Allocates memory on
GPU.
cudaMemcpy() – Transfers data
between CPU and GPU.
cudaFree() – Frees GPU memory..
Memory Allocation
Example
Example Code:
float *d_A;
cudaMalloc((void**)&d_A, size);
cudaFree(d_A);
cudaMalloc() allocates memory on the
GPU.
cudaFree() deallocates memory.
Host to Device Data
Transfer
cudaMemcpy(d_A, h_A, size, cudaMemcpyHostToDevice);
Transfers data from CPU (Host) to GPU
(Device).
Kernel Launch
A kernel is a function executed on the GPU.
Syntax:
kernelFunction<<<numBlocks, threadsPerBlock>>>(args);
Kernel Launch Configuration:
blockIdx, threadIdx: Identifies execution location.
<<<num_blocks, threads_per_block>>> defines
execution hierarchy.
Example:
vectorAdd<<<16, 256>>>(d_A, d_B, d_C, N);
Kernel Specific System
Variables
Device to Host Memory
Transfer
cudaMemcpy(h_C, d_C, size, cudaMemcpyDeviceToHost);
Transfers computed results back to the CPU.
Types of CUDA Function
Specifiers
Global Functions (__global__)
Device Functions (__device__)
Host Functions (__host__)
Unified Memory Functions
(cudaMallocManaged)
CUDA Function
Declaration Keywords
__global__ → Kernel function (executes
on GPU, called from CPU).
__device__ → Function executed on the
GPU (called from another GPU function).
__host__ → Function executed only on
the CPU.
CUDA Synchronization
cudaDeviceSynchronize() ensures kernel
execution completes before proceeding.
Prevents race conditions and incorrect results.
CUDA Streams and
Concurrency
CUDA supports concurrent execution
using streams.
Allows overlapping memory transfers
and computations.
Common CUDA Errors &
Debugging
cudaMemcpy() failure → Check memory
sizes and pointers.
Kernel launch failure → Verify grid/block
size.
Out-of-bounds memory access → Use
proper indexing.
Writing a Simple CUDA
Program
Example: Vector Addition
CUDA Kernel for Vector
Addition
__global__ void vectorAdd(float *A, float *B, float *C,
int n)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) C[i] = A[i] + B[i];
}
Example Vector Addition
CPU only
Example Vector Addition
CPU-GPU
Device Memory Allocation
for Vector Addition
Example
Host to Device Data Transfer
for Vector Addition Example
Kernel Launch for Vector
Addition Example
Device to Host Memory Transfer
for Vector Addition Example
Compile and Run
Performance Optimization
Techniques
Minimize global memory accesses.
Minimize memory transfers between host and
device.
Optimize memory coalescing.
Use shared memory effectively.
Optimize thread and block configuration.
Increase occupancy with more active threads.
Avoid warp divergence (if-else within a warp).