Debugging & Performance
Analysis
Profiling with GDB | Finding bottlenecks
Rana Abdullah
Alishba
Muhammad Abubakar Usman
Muhammad Usama
Mehreen Ali Mughal
Dept. of Information Technology — Lahore Garrison University
Agenda
Debugging Multithreaded Profiling Bottlenecks &
Essentials Debugging gprof, perf, VTune Optimization
Bugs, GDB, Threads, deadlocks, Find hotspots, apply
breakpoints watchpoints Amdahl's Law,
optimize
Case Study & Best Practices
Real-world example, key takeaways
What Is Debugging?
Systematic process to find, isolate, and fix software errors.
Reproduce the bug
Isolate the code
Find the root cause
Fix and verify
History: “Bug” was popularized by Grace Hopper in 1947.
Types of Bugs in Parallel Programs
Race Deadlock Livelock
Condition
Threads block Threads keep
Shared data is each other reacting, but
accessed at the forever. nothing moves
same time, so forward.
results vary by
General Bugs
timing.
Syntax — compile-time errors
Runtime — crashes or exceptions
Starvation Memory
Logical — wrong output
Inconsistenc Memory — overflow or use-after-free
A thread never
y
gets the resource
it needs. Different nodes
see different data
because of
cache/coherence
issues.
Introduction to GDB
GNU Debugger for inspecting program state while running or after a crash.
Supports C, C++, Fortran, Ada, Go, and more on Linux, macOS, and Windows
(MinGW).
Capabilities
Breakpoints
Variables
Call stack
Step-through debugging
Threads and core dumps
Compile with Symbols
g++ -g -o myprogram [Link]
Core GDB Commands & Workflow
Execution Breakpoints
run, run args, continue, kill, quit break func | file:line | if cond; info,
delete, disable
Stepping Inspection
next, step, finish, until print, display, locals, backtrace,
frame N
Memory
x/10d &arr, x/s str
Breakpoints & Watchpoints
Breakpoints pause at lines, functions,
or conditions.
tbreak auto-deletes after hit. Example:
break [Link] if counter == 500.
Watchpoints stop on variable
changes: watch x, rwatch x, awatch x.
Debugging Multithreaded Programs
Concurrency bugs are timing-dependent Heisenbugs.
Use GDB to list threads and control scheduler behavior.
01 02 03
Thread Commands Scheduler Control Deadlock Debugging
info threads; thread N; thread apply all bt set scheduler-locking on|off|step Ctrl+C → info threads → thread apply all
bt → look for pthread_mutex_lock
Introduction to Profiling
Profiling shows where execution time
goes: CPU, memory, I/O, and threads.
Debugging finds bugs; profiling finds
bottlenecks.
Profile before optimizing.
More threads can hurt performance.
Profiling Tools & Next Steps
Actionable Workflow
1. Debug correctness first.
2. Profile hotspots with gprof, perf, or VTune.
3. Classify the bottleneck: CPU, memory, I/O, sync.
gprof 4. Optimize, then re-profile.
GCC profiler for flat profiles and call graphs.
Use gprof for quick insight; use Valgrind, perf, or VTune for deeper
Valgrind / Callgrind
analysis.
Deep function and cache analysis.
perf
Low-overhead Linux sampling with hardware counters.
Intel VTune / Nsight
Advanced CPU, GPU, and parallel profiling.
Reading gprof Output
Understanding where your program spends its time is the first step to optimization.
% time seconds calls name
58.3% 3.50 10000 matrixMultiply()
25.0% 1.50 1000 sortArray()
8.3% 0.50 1000000 computeHash()
5.0% 0.30 10 loadData()
How to Interpret Key Takeaways from this Example
% time: Percentage of total runtime spent in the function. matrixMultiply() consumes 58% of runtime. This is the primary
seconds: Total time spent in the function itself (exclusive). bottleneck to address.
calls: Number of times the function was invoked.
computeHash() is called 1 million times but is relatively fast.
Optimizing this would be a lower priority than matrixMultiply().
What is a Bottleneck?
A point in the system where performance is severely limited, Symptoms:
slowing down everything else.
Some cores at 100%, others idle
Analogy: 🚗 A 6-lane highway merging into 1 lane — no Adding more threads doesn't improve speed
matter how fast cars move elsewhere, this section controls Execution time doesn't improve with better hardware
total speed. Memory usage grows unexpectedly
Types of Bottlenecks
Recognizing specific types of bottlenecks is the first step toward effective diagnosis and optimization in parallel and distributed
computing environments.
Type Cause Fix
CPU Uneven workload across cores Better load balancing
Memory Cache misses, false sharing Improve data locality
I/O Slow disk or network Asynchronous I/O, caching
Synchronization Too many locks/barriers Reduce critical sections
Communication MPI message delays Batch messages
Load Imbalance Some threads finish early, others still Dynamic task assignment
working
Amdahl's Law
Amdahl's Law defines the maximum theoretical speedup of a program when only a portion of it is parallelized. It highlights that the speedup is ultimately
limited by the sequential (non-parallelizable) part of the task, regardless of how many processing units are added.
Where: S = Sequential fraction, P = Parallel fraction, N = Number of cores
Sequential % 2 Cores 4 Cores 8 Cores ∞ Cores
5% 1.9× 3.5× 5.9× 20×
25% 1.6× 2.3× 3.0× 4×
50% 1.3× 1.6× 1.8× 2×
Key Insight: If 50% of your code is sequential, you can never achieve more than a 2× speedup, no matter how many processing cores you
add. This underscores the critical importance of minimizing the sequential portion of any parallelizable task.
Real-World Case Study: Image Processing
An image processing program with 8 cores only achieved 2× speedup. Profiling revealed a bottleneck in applyFilter() (62% of runtime) due to all 8 threads
serializing on a single mutex lock.
Before (Broken) After (Fixed)
[Link](); // Each thread works on its own image data
for (int i = 0; i < pixels; i++) for (int i = 0; i < pixels; i++)
data[i] = transform(data[i]); data[i] = transform(data[i]);
[Link]();
By ensuring each thread processed its own independent data, the mutex was
The lock limited execution to only one thread at a time, negating parallelism. removed, enabling true parallelism.
Metric Before After
Speedup 2× 6.8×
Runtime 480s 70s
CPU Usage 28% 89%
Bottleneck Identification Techniques
Identifying where your program spends most of its time and resources is crucial for effective optimization. Here are common techniques and the
tools that support them.
Technique Tool What It Finds Red Flags in Code:
CPU Profiling gprof, perf Slowest functions [Link](); // lock
inside a tight loop → huge overhead
Memory Analysis Valgrind Cache misses, leaks
int* p = new int[N]; // memory
allocation inside loop → slow
Thread Tracing Intel VTune Idle threads, lock waits
for(i) for(j) for(k) // O(n³) nested
loops → CPU bottleneck
Scalability Test Manual Sync bottlenecks
Amdahl's Analysis Math Sequential ceiling
These code patterns often indicate performance traps.
Code Review Manual Nested loops, locks in Identify them early to avoid major bottlenecks.
loops
Best Practices & Summary
Applying these debugging and profiling techniques consistently will significantly improve your ability to identify and resolve performance issues in parallel programs.
Debugging Best Practices Profiling Best Practices
Always compile with -g for debug symbols. Always profile first, optimize second.
Use conditional breakpoints for efficient loop analysis. Focus optimization efforts on the top 1-2 functions identified.
Utilize thread apply all bt to diagnose deadlocks. Re-profile after every significant change to validate impact.
Deploy watchpoints to catch unexpected variable changes.
Key Takeaways: Debugging & Profiling
Topic Key Tool Primary Takeaway
Debugging GDB Pause & inspect program state.
Thread Bugs GDB thread commands Find deadlocks instantly.
Profiling gprof Know where time is actually spent.
Bottlenecks perf, Valgrind CPU / Memory / Sync / I/O issues.
Speedup Limit Amdahl's Law Sequential code is the true ceiling.
Your insights and queries
are welcome!
Thank You!