PROGRAMMING MASSIVELY
PARALLEL PROCESSORS
01 – Preliminaries and Introduction
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 1
LEARNING GOALS
➢ Introduce course
➢ Clarify assessment methods
➢ Motivate course topic
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 2
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
Q&A LINK
▪ Collect and vote for questions under
[Link]
▪ We will take some time at the end of the
session to review and answer questions
▪ If your question references a particular slide,
please note the slide number (bottom right)
▪ Keep link open throughout the session
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 3
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
TEAM
▪ Lecturer
▪ Dr. Johannes S. Mueller-Roemer
[Link]-roemer@[Link]
▪ Teaching assistant
▪ Sebastian Besler
[Link]@[Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 4
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
ABOUT ME
▪ Started CUDA programming in 2009 as a
student research assistant at BTU Cottbus
▪ Research assistant at Fraunhofer IGD since
2011, working on various GPGPU topics
▪ PhD on GPGPU-accelerated modeling,
simulation, and visualization in 2019
▪ Deputy department head since 2023
▪ Selection of GPGPU applications
▪ Signal and image processing
▪ Physical simulation (CFD, CSM)
▪ (Volumetric) mesh processing
▪ Sparse matrix data structures and tuning
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 5
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
CONCEPT
▪ “Integrierte Lehrveranstaltung” (Integrated course)
▪ Flexible combination of lectures and practical exercises as needed
▪ Lectures
▪ Introduction to the topic of programming massively parallel processors, including
▪ architecture of massively parallel processors, especially graphics processing units (GPUs),
▪ overview of massively parallel programming models, with a focus on CUDA,
▪ massively parallel algorithms and data structures,
▪ and performance tuning with profiling tools
▪ Practical exercises
▪ Practical programming and profiling exercises using CUDA and Nsight
▪ Final group project (2–3 people per group)
▪ Topic(s) for the final project will be presented and co-supervised by application domain experts
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 6
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
PREREQUISITES
▪ (Recommended) prerequisites
▪ C or, preferably, C++ programming
experience
▪ (Parallel) algorithms and data structures
▪ Interest in parallel programming and GPU
computing
▪ Not an “Introduction to C++” course!
▪ Some tips and pointers to resources for self-
study following literature
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 7
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
MOODLE COURSE
▪ Slides and additional materials will be
provided on the Moodle course page
[Link]
[Link]/course/[Link]?id=44023
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 8
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
FINAL PROJECTS
▪ Choice between one or more topics
▪ Topics from various application domains
▪ Details and list of topics TBA
▪ Each project will have up to two supervisors
▪ An application domain expert
▪ A massively parallel programming expert
▪ Goal:
Learn massively parallel programming by applying knowledge to real-world problems, not toy
examples
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 9
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
ASSESSMENT
▪ Written or oral final exam
▪ Depending on number of students
▪ Scheduling of oral exams will take place during final Q&A session, if applicable
▪ Exercises and final project
▪ Up to one full bonus grade
▪ You must pass the exam for the bonus to apply!
▪ Up to ⅓ grade bonus for exercises
▪ Up to ⅔ grade bonus for final project
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 10
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
SCHEDULE
▪ Mondays, 13:30–15:10, S1|01 A4
▪ Wednesdays, 9:50–11:30, S1|01 A5
▪ Both weekly slots can be used for
▪ lectures,
▪ exercises,
▪ or neither, i.e., not all slots will be used
▪ e.g., during the final project
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 11
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
COURSE SCHEDULE
2025-10-13 Preliminaries and introduction 2025-12-03 No lecture / exercise
2025-10-15 No lecture / exercise 2025-12-08 Half-time project status presentations
2025-10-20 Programming model 2025-12-10 Half-time project status presentations
2025-10-22 Memory spaces 2025-12-15 Reserved for topic requests / unplanned scheduling changes
2025-10-27 Programming exercise 2025-12-17 Reserved for topic requests / unplanned scheduling changes
2025-10-29 Project topic presentation(s) 2026-01-12 Reserved for topic requests / unplanned scheduling changes
2025-11-03 Synchronization 2026-01-14 Reserved for topic requests / unplanned scheduling changes
2025-11-05 Scan & sorting 2026-01-19 Reserved for topic requests / unplanned scheduling changes
2025-11-10 Profiling exercise 2026-01-21 Reserved for topic requests / unplanned scheduling changes
2025-11-12 Reserved for topic requests / unplanned scheduling changes 2026-01-26 Reserved for topic requests / unplanned scheduling changes
2025-11-17 Reserved for topic requests / unplanned scheduling changes 2026-01-28 Reserved for topic requests / unplanned scheduling changes
2025-11-19 No lecture / exercise 2026-02-02 Reserved for topic requests / unplanned scheduling changes
2025-11-24 Reserved for topic requests / unplanned scheduling changes 2026-02-04 Exam Q&A (and scheduling, if oral)
2025-11-26 Reserved for topic requests / unplanned scheduling changes 2026-02-09 Final project presentations
2025-12-01 Reserved for topic requests / unplanned scheduling changes 2026-02-11 Final project presentations
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 12
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
ACCOUNTS
▪ Exercises and projects use CUDA
▪ All exercises and projects must run on
Lichtenberg cluster nodes
▪ Register in TUCaN ASAP for account creation
▪ Fill, sign, and hand in account forms
▪ Use of a private CUDA GPU is possible but
not recommended
▪ No support
▪ Code must run on cluster nodes for grading
2025-10-13 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 13
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
LITERATURE
▪ NVIDIA CUDA C++ Programming Guide
[Link]
programming-guide/
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 14
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
LITERATURE
▪ NVIDIA CUDA C++ Best Practices Guide
[Link]
practices-guide/
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 15
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
LITERATURE
▪ Various articles from NVIDIA’s Technical Blog
[Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 16
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
LITERATURE
▪ Hijma, P. et al.
Optimization Techniques for GPU
Programming
2023, ACM Computing Surveys 55 (11),
DOI* 10.1145/3570638
*: [Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 17
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
LITERATURE
▪ Hwu, W. / Kirk, D. / El Hajj, I.
Programming Massively Parallel
Processors
2022, 4th edition,
DOI 10.1016/C2020-0-02969-5
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 18
C++ RESOURCES
▪ Bjarne Stroustrup
Programming: Principles and Practice Using C++, 3rd ed.
2024, ISBN 0-13-830868-3
▪ Learn C++
[Link]
▪ One of few acceptable online C++ resources, most (free) tutorials and videos should be avoided!
▪ C++ Reference
[Link]
▪ Good, up-to-date online reference, avoid others such as [Link]
▪ An Introduction to Modern CMake
[Link]
▪ Introduction to the de-facto standard cross-platform build system
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 19
C++ TIPS
▪ Avoid paired functions
▪ new/delete, malloc/free, create/destroy, begin/end, lock/unlock, register/unregister …
▪ Make use of constructors and destructors and follow the RAII* principle!
▪ Many standard helpers, e.g., std::unique_ptr, std::make_unique, std::lock_guard
▪ Don't separate initialization from creation, i.e., init methods separate from the constructor
▪ All objects should be valid after construction (Initialization to a valid “empty” value is ok!)
▪ Avoid shared ownership, i.e., std::shared_ptr
▪ …and if you can’t, prefer const over mutable
▪ If you overwrite one special function (destructor, copy/move constructor/assignment), overwrite
them all (or explicitly delete them) → rule of zero / rule of five
▪ …and never manage more than one resource per class/struct (use composition)
*: Resource Acquisition Is Initialization – Use constructors / destructors and object lifetimes for resource management
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 20
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
THE NEED FOR SPEED
▪ Demand for processing power keeps
increasing
▪ More processing power allows for…
▪ …faster or more precise simulations
▪ …faster or more realistic graphics
▪ …larger machine learning models, etc. © Barcelona Supercomputing Center
© NVIDIA
© Epic Games
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 21
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
MOORE’S LAW
▪ Regarding transistor count, “Moore’s Law” *
remains valid
▪ i.e., transistor count doubles every two years
▪ Until ~2003, clock rates grew exponentially
▪ Power limits have stopped this trend
▪ How to compensate?
▪ Parallelism
▪ An end to transistor scaling is expected soon
Source: IEEE International Roadmap for Devices and Systems, “Executive
*: Technically an observation (and goal), not a law Summary 2022.” DOI: 10.60627/c13z-v363.
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 22
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
FREQUENCY SCALING
▪ Dynamic power consumption 𝑃 ~ 𝐶 ⋅ 𝑉 2 ⋅ 𝑓 Frequency Scaling
▪ 𝐶 gate capacitance 600
▪ 𝑉 voltage
500
▪ 𝑓 frequency
▪ However, voltage needs to be increased to 400
reach gate threshold more quickly!
TDP [W]
300
▪ 𝑃 rises super-linearly with 𝑓
200
100
0
1 2 3 4 5
Frequency [GHz]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 23
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
AMDAHL’S LAW
▪ Power scales linearly with core count Core Count Scaling
▪ …as does performance 600
▪ …if the code is perfectly parallel!
500
▪ If it isn’t, Amdahl’s law applies
1 400
𝑆= 𝑝
1−𝑝 +
TDP [W]
𝑁 300
▪ 𝑆 speedup
▪ 𝑝 percentage of work that is parallel 200
➢ Massively parallel processors require
massively parallel algorithms 100
0
1 5 9 13 17 21 25
Cores [#]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 24
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
GREEN500
▪ Many real-world problems are parallelizable,
and 𝑝 tends to increase with problem size
▪ In high-performance computing (HPC),
energy consumption is a big cost factor Green500 – Top 10
System Cores PFlop/s GFlops/W
▪ Green500 sorts 500 fastest (public) HPC JEDI – […] NVIDIA GH200 Superchip […]
ROMEO-2025 – […] NVIDIA GH200 Superchip […]
19,584
47,328
4.50
9.86
72.73
70.91
systems by flop/s per Watt Adastra 2 – […] AMD Instinct MI300A […]
Isambard-AI phase 1 – […] NVIDIA GH200 Superchip […]
16.128
34,272
2.53
7.42
69.10
68.83
▪ flop/s – floating-point operations per second Otus (GPU only) – […] NVIDIA H100 SXM5 80GB […]
Capella – […] Nvidia H100 SXM5 94GB […]
19,440
85,248
4.66
24.06
68.18
68.05
SSC-24 Energy Module – […] NVIDIA H100 SXM5 80GB […] 11,200 3.82 67.25
▪ All top 10 systems use GPU accelerators Helios GPU – […] NVIDIA GH200 Superchip […]
AMD Ouranos – […] AMD Instinct MI300A […]
89,760
16,632
19.14
2.99
66.95
66.46
Henri – […] NVIDIA H100 80GB PCIe […] 8,288 2.88 65.40
➢ Massively parallel processors are
essential for HPC and Green Computing
Source: Green500, June 2025. [Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 25
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
HPC TO EMBEDDED
▪ Programmable GPUs are not limited to HPC
▪ Used in consumer hardware
▪ Gaming PCs
▪ Consoles
▪ Cell phones and portable consoles
▪ …and even embedded systems!
▪ Right: Jetson Nano compute module
▪ Quad-core ARM Cortex-A57
▪ 4 GiB RAM
▪ Embedded Linux
© NVIDIA
➢ Massively parallel processors are relevant
in all domains of computing
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 26
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
WHY PMPP?
▪ So far, we have covered why massively
parallel processors are important
▪ …but what exactly are massively parallel
processors?
▪ …and what is different about programming
them?
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 29
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
WHAT DO YOU THINK?
▪ What do you think differentiates a massively
parallel processor from other processors?
▪ Collect your suggestions in the survey
section of [Link]
▪ Same link as for Q&A
▪ New section should appear, otherwise, reload
the page
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 30
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
GENERAL-PURPOSE
PROCESSORS
▪ Latency-oriented design
▪ Arithmetic logic units (ALUs) and floating-
point units (FPUs) designed for low latency
▪ Large last-level on-chip caches
▪ Sophisticated execution control
▪ Branch prediction
▪ Superscalar execution
▪ Out-of-order (OoO) execution
▪ Low latency and sophisticated control logic
cost chip area and power
▪ Legacy OS, application, and I/O support
2025-10-13 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 31
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
GENERAL-PURPOSE
PROCESSORS
▪ High base clock (~2.5 GHz) and boost clock
(up to 6 GHz) frequencies
▪ Modern central processing units (CPUs) have
many parallel features
▪ Multicore architecture (multiple cores per die)
▪ Single instruction, multiple data (SIMD) vector
extensions
▪ Simultaneous multithreading to hide latency
and increase efficiency
2025-10-13 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 32
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
MASSIVELY PARALLEL
PROCESSORS
▪ Throughput-oriented design
▪ Parallel-native designs with a focus on high-
throughput, parallel floating-point operations
▪ Wide, high-throughput memory buses
▪ Example: NVIDIA H100 NVL
▪ 2×12×512-bit = 2×6144-bit bus width
▪ 2×3.9 TB/s memory bandwidth
▪ Comparison: Intel Xeon 6780E
▪ 8×64-bit = 512-bit bus width
▪ 819 GB/s memory bandwidth
▪ Fewer cache levels and programmable on-
chip memory
▪ Very wide SIMD or SIMT units
2025-10-13 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 33
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
MASSIVELY PARALLEL
PROCESSORS
▪ Simpler control logic
▪ Limited or no branch prediction
▪ No OoO execution
▪ Usually a coprocessor, i.e., needs a CPU to
for OS and I/O
▪ Exceptions such as Intel Xeon Phi KNL
▪ Lower base clock (~1.2 GHz) and boost clock
(< 4 GHz) frequencies
▪ Also known as manycore processors
2025-10-13 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 34
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
MPP = GPU?
▪ Are all massively parallel processors GPUs?
▪ No, GPUs are just the most important
category due to their ubiquity
▪ Non-GPU massively parallel processors
▪ Intel Xeon Phi (discontinued) manycore x86
▪ PEZY-SCx used in Japanese HPC
▪ Sunway SW26010P used in Chinese HPC
▪ Cerebras WSE-3 wafer-scale AI processor
▪ Data-center versions of GPUs typically do not
have external graphics connectors and focus
on compute
© AMD
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 35
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
CORES VS. CUDA CORES
▪ High-end CPUs such as AMD’s Ryzen Threadripper PRO 7995WX or Intel’s Xeon 6780E have
on the order of 100 cores (96 and 144, respectively)
▪ High-end GPUs such as AMD’s Instinct MI250X or NVIDIA’s H100 SXM5 are quoted as having
more than 10,000 cores (14,080 “Stream processors” and 16,896 “CUDA cores”, respectively)
▪ Are these numbers comparable?
▪ Show of Hands: do you think they are comparable?
▪ No, “CUDA cores” are more closely related to the number of floating-point arithmetic units
The NVIDIA H100 GPU with SXM5 board form-factor includes the following units:
8 GPCs, 66 TPCs, 2 SMs/TPC, 132 SMs per GPU
128 FP32 CUDA Cores per SM, 16896 FP32 CUDA Cores per GPU
▪ We will go into more detail what these terms mean in future sessions; for now: SM ≈ CPU Core
Quote source: [Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 36
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
CONVERGENCE?
▪ With actual core counts on the same order of
magnitude and many common parallel
features, are we seeing a convergence of
CPUs and GPUs?
▪ Show of Hands: do you think the two
technologies are converging?
▪ Currently, no. Power limits apply to GPUs as
well, so specialization (latency vs. throughput)
is necessary. In fact, we are seeing more and
more specialized processors and SoCs with
multiple heterogeneous processors!
▪ Programming massively parallel
processors will remain relevant for the
foreseeable future
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 37
LEARNING GOALS
➢ Introduce course
➢ Clarify assessment methods
➢ Motivate course topic
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 38
PROGRAMMING MASSIVELY PARALLEL PROCESSORS / J. S. MUELLER -ROEMER
Q&A
Time to review questions asked via [Link]
Please fill out the session
feedback survey on
[Link]
2 0 2 5 -1 0 -1 3 Computer Science | Interactive Graphics Systems Group | Dr. Johannes S. Mueller -Roemer 39