Shared Memory Parallel Programming.

Guide to OpenMP
and PThreads

1. Learning Objectives

By the end of this study material, you should be able to:

· Introduction: Articulate the fundamental concepts of shared memory parallelism,

distinguish between the compiler directive-based model of OpenMP and the library-based
threading model of POSIX Threads (PThreads), and explain the Fork-Join execution
paradigm.
· Familiarity with Functions: Identify and correctly utilize the core functions and directives
of both OpenMP (e.g., #pragma omp parallel, omp_get_thread_num(), reduction) and PThreads
(e.g., pthread_create(), pthread_join(), pthread_mutex_lock()).
· Coding, Compiling, Running, and Benchmarking: Write correct C programs that
leverage multi-core processors, compile them with the appropriate GCC flags (-fopenmp, -
pthread), execute them in a Linux environment, and measure performance speedup using
wall-clock timing functions.
· Advantages and Limitations: Critically evaluate the trade-offs between OpenMP and
PThreads regarding ease of implementation, control granularity, and debugging complexity.
· Relevant Visual Elements: Interpret and create simple visual representations of thread
interaction, such as the Fork-Join model diagram and Mutex locking diagrams, to reason
about concurrency.
· Other Relevant Information: Recognize common pitfalls in shared memory programming,
including data races, false sharing, and deadlock, and apply basic strategies to avoid them.

1
2. Theoretical Foundations: Shared Memory and Threading
Models

2.1 The Shared Memory Paradigm

In shared memory parallel programming, multiple execution threads reside within a single
operating system process. They share a common address space. This means a global variable
modified by Thread A is immediately visible to Thread B.

· Advantage: Communication is fast and implicit (through memory reads/writes).

· Disadvantage: Uncontrolled access leads to Race Conditions, where the final result
depends on the non-deterministic timing of thread execution.

2.2 OpenMP vs. PThreads: A Tale of Two Approaches

Feature OpenMP POSIX Threads (PThreads)

Abstraction Low-level. You manage thread

High-level. You annotate the code.
Level lifecycle.

Compiler Directives (#pragma) and

Implementation Pure C API Library (pthread.h).
runtime library functions.

Code compiles as serial code if - Requires explicit thread logic; serial

Portability fopenmp is omitted. Excellent for fallback requires #ifdef or separate
incremental parallelization. code paths.

Fine-grained. Ideal for complex

Control Coarse-grained. Ideal for loops and
client/server models and custom
Granularity sections.
synchronization.

Easier to trace specific function

Ease of Often harder to debug low-level
calls, but easier to introduce
Debugging interaction due to compiler magic.
memory leaks.

2
2.3 The Fork-Join Model

Both libraries operate on the Fork-Join principle. A single master thread (main) executes until a
parallel region is encountered. The runtime Forks a team of worker threads. They execute
concurrently, then synchronize and terminate (or sleep) at an implicit barrier—this is the Join.

3. Part I: OpenMP Programming

3.1 Core Model and Compilation

OpenMP relies on compiler directives. To compile an OpenMP program named prog.c:

gcc -fopenmp -o prog prog.c

3.2 Essential Functions and Directives

Directive / Function Purpose

Creates a team of threads. Code block executes on every

#pragma omp parallel
thread.

omp_get_thread_num() Returns the ID of the calling thread (0 to N-1).

3
 omp_get_num_threads() Returns the total number of threads in the current team.

Divides loop iterations among threads. Must be inside

#pragma omp for
a parallel region.

#pragma omp parallel for Shortcut for creating parallel region AND dividing the loop.

Creates private copies of sum for each thread, combines

reduction(+:sum)
them safely at the end.

#pragma omp critical Allows only one thread to execute the block at a time.

3.3 Code Example 1: "Hello World" and Thread Discovery

3.4 Code Example 2: Vector Addition with Reduction (Benchmarking)

This example demonstrates a data-parallel loop and proper timing.

#include <stdio.h>

4
#include <stdlib.h>

#include <omp.h>

#define N 100000000 // 100 Million elements

int main() {

double *a = (double*)malloc(N * sizeof(double));

double *b = (double*)malloc(N * sizeof(double));

double *c = (double*)malloc(N * sizeof(double));

// Initialize arrays

for(int i=0; i<N; i++) {

a[i] = i * 1.0;

b[i] = i * 2.0;

double start_time = omp_get_wtime();

// Parallelize the loop. Each thread works on a chunk of i.

#pragma omp parallel for

for(int i=0; i<N; i++) {

c[i] = a[i] + b[i];

double end_time = omp_get_wtime();

5
 // Check a result to ensure compiler didn't optimize away the loop

printf("c[%d] = %f\n", N-1, c[N-1]);

printf("OpenMP Vector Add Time: %f seconds\n", end_time - start_time);

free(a); free(b); free(c);

return 0;

3.5 Code Example 3: The Danger of Data Races and the Fix

Incorrect (Race Condition):

int sum = 0;

#pragma omp parallel for

for(int i=0; i<1000; i++) {

sum += i; // Multiple threads read-modify-write "sum" -> WRONG

Correct Solution A: Critical Section (Slow)

int sum = 0;

#pragma omp parallel for

for(int i=0; i<1000; i++) {

#pragma omp critical

sum += i;

Correct Solution B: Reduction (Fast)

6
int sum = 0;

#pragma omp parallel for reduction(+:sum)

for(int i=0; i<1000; i++) {

sum += i;

4. Part II: PThreads Programming

4.1 Core Model and Compilation

PThreads requires explicit management of thread handles and attributes. Compile with:

gcc -pthread -o prog prog.c

4.2 Essential Functions and Types

Function Purpose

pthread_t Data type for a thread identifier.

pthread_create(&tid, NULL, func, arg) Starts a new thread executing func(arg).

pthread_join(tid, NULL) Waits for a specific thread to terminate (Join).

pthread_mutex_t Data type for a mutual exclusion lock.

pthread_mutex_lock(&mutex) Acquires the lock (blocks if already held).

pthread_mutex_unlock(&mutex) Releases the lock.

Synchronization point where all threads must

pthread_barrier_t
wait for each other.

7
4.3 Mutex Locking Mechanism

A mutex ensures Mutually Exclusive access to a critical section.

4.4 Code Example 1: Creating and Joining Threads

This example shows the boilerplate required to pass arguments to threads safely.

#include <stdio.h>

#include <stdlib.h>

#include <pthread.h>

#define NUM_THREADS 4

// Thread function signature MUST be: void* func(void* arg)

void* worker(void* arg) {

int tid = *((int*)arg);

printf("PThread %d is executing.\n", tid);

return NULL;

8
int main() {

pthread_t threads[NUM_THREADS];

int thread_ids[NUM_THREADS];

for(int i=0; i<NUM_THREADS; i++) {

thread_ids[i] = i;

// Create thread: handle, attributes, function, argument

if(pthread_create(&threads[i], NULL, worker, &thread_ids[i]) != 0) {

perror("Failed to create thread");

exit(1);

// Wait for all threads to finish

for(int i=0; i<NUM_THREADS; i++) {

pthread_join(threads[i], NULL);

printf("All threads have joined. Exiting.\n");

return 0;

4.5 Code Example 2: Vector Addition with PThreads (Structure Packing)

Because pthread_create passes only one argument, we must pack multiple parameters into a struct.

9
#include <stdio.h>

#include <stdlib.h>

#include <pthread.h>

#include <time.h>

#define N 100000000

#define NUM_THREADS 4

// Struct to hold the "slice" of work for each thread

typedef struct {

double *a;

double *b;

double *c;

int start;

int end;

} thread_data_t;

void* vector_add(void* arg) {

thread_data_t* data = (thread_data_t*)arg;

for(int i = data->start; i < data->end; i++) {

data->c[i] = data->a[i] + data->b[i];

return NULL;

int main() {

10
 double *a = (double*)malloc(N * sizeof(double));

double *b = (double*)malloc(N * sizeof(double));

double *c = (double*)malloc(N * sizeof(double));

// Initialize

for(int i=0; i<N; i++) { a[i] = i; b[i] = i*2; }

pthread_t threads[NUM_THREADS];

thread_data_t tdata[NUM_THREADS];

struct timespec start, end;

clock_gettime(CLOCK_MONOTONIC, &start);

int chunk = N / NUM_THREADS;

for(int i=0; i<NUM_THREADS; i++) {

tdata[i].a = a;

tdata[i].b = b;

tdata[i].c = c;

tdata[i].start = i * chunk;

tdata[i].end = (i == NUM_THREADS - 1) ? N : (i+1) * chunk;

pthread_create(&threads[i], NULL, vector_add, &tdata[i]);

for(int i=0; i<NUM_THREADS; i++) {

pthread_join(threads[i], NULL);

11
 clock_gettime(CLOCK_MONOTONIC, &end);

double time_taken = (end.tv_sec - start.tv_sec) + (end.tv_nsec - start.tv_nsec) / 1e9;

printf("c[%d] = %f\n", N-1, c[N-1]);

printf("PThreads Vector Add Time: %f seconds\n", time_taken);

free(a); free(b); free(c);

return 0;

4.6 Code Example 3: Mutual Exclusion (Mutex) for a Shared Counter

#include <stdio.h>

#include <pthread.h>

#define ITERATIONS 1000000

int counter = 0;

pthread_mutex_t lock; // Global mutex

void* increment(void* arg) {

for(int i=0; i<ITERATIONS; i++) {

pthread_mutex_lock(&lock); // Enter Critical Section

counter++; // Safe update

pthread_mutex_unlock(&lock); // Exit Critical Section

return NULL;

12
}

int main() {

pthread_t t1, t2;

pthread_mutex_init(&lock, NULL); // Initialize mutex

pthread_create(&t1, NULL, increment, NULL);

pthread_create(&t2, NULL, increment, NULL);

pthread_join(t1, NULL);

pthread_join(t2, NULL);

printf("Final Counter Value: %d (Expected: %d)\n", counter, 2 * ITERATIONS);

pthread_mutex_destroy(&lock); // Clean up

return 0;

13
5. Benchmarking and Performance Considerations

5.1 Timing Your Code

· OpenMP: Always use omp_get_wtime() for portable, high-resolution wall-clock time.

· PThreads: Use clock_gettime(CLOCK_MONOTONIC, &ts) for reliable wall-clock time not
affected by system clock adjustments.

5.2 Speedup and Efficiency

· Speedup (S): 𝑆=𝑇 𝑠 𝑒𝑟 𝑖𝑎 𝑙 /𝑇 𝑝 𝑎𝑟 𝑎 𝑙𝑙𝑒 𝑙

· Efficiency (E): 𝐸=𝑆/ 𝑃 (where P is number of cores/threads)

Important Observation: You will notice that doubling the threads does not double the speed
(Amdahl's Law). The sequential portions of code (malloc, I/O) become the bottleneck.

5.3 False Sharing

A silent performance killer in both OpenMP and PThreads. Occurs when two threads modify
different variables that reside on the same cache line (typically 64 bytes). The cache coherence
protocol forces the cache line to bounce between cores, destroying performance.

False Sharing. Even though they touch different memory addresses, the hardware sees
contention for the same cache line.
Mitigation: Use padding (e.g., char pad[60]) or OpenMP's schedule(static, chunk_size) to spread work
across distinct cache lines.

6. Advantages and Limitations: A Comparative Analysis

OpenMP

Advantages Limitations

Less Control: Difficult to implement complex

Incremental Parallelism: Start with serial code,
pipeline patterns or producer-consumer
add one #pragma, and it's parallel.
queues.

14
 Compiler Dependent: If the compiler has a
Portability: Same code compiles and runs
bug in its OpenMP implementation, you are
(serially) without OpenMP compiler.
stuck.

Overhead Spikes: Implicit barriers at the end

Productivity: Much less code than PThreads for
of for loops can cause idle time if work is
loop-level parallelism.
unbalanced.

PThreads

Advantages Limitations

Fine-Grained Control: You can precisely Verbose & Error-Prone: Easy to forget
manage thread affinity, scheduling policies, and a pthread_join (resource leak) or
signal masks. a pthread_mutex_unlock (deadlock).

Rich Synchronization Primitives: Condition Steep Learning Curve: Managing thread

variables allow threads to sleep until signaled, arguments via struct packing and handling
essential for server applications. return values adds complexity.

Deterministic Patterns: Ideal for persistent Code Clutter: Parallelism logic dominates the
worker thread pools. codebase, obscuring the core algorithm.

7. Additional Expert Guidance for Beginners

7.1 Common Pitfalls and Debugging Tips

The "Parallel For" Private Variable Trap:

int i; // DANGER: declared outside.

#pragma omp parallel for

for(i=0; i<N; i++) { ... }

15
 Problem: i is shared by default! Thread 1 might set i=0, Thread 2 sets i=1, causing chaos.
Fix: Declare loop index inside the for statement: for(int i=0; ...) or explicitly list as private(i).

PThreads and Returning Local Pointers:

Never return a pointer to a local variable from a thread function. If you must return data,
allocate it dynamically (malloc) and let the joining thread free it.

Using printf for Debugging:

While printf is useful, it forces serialization (locks stdout), changing the timing of your
program and often hiding the race condition you are trying to find. Use it sparingly, and
consider logging to a memory buffer instead.

7.2 Recommended Environment

· OS: Linux (Ubuntu/Debian or WSL2 on Windows). The performance and tooling for
PThreads/OpenMP on native Linux is superior.
· Compiler: GCC (version 9+).
· Tool: htop or top (press 1) to visually see your threads saturating the CPU cores.

16