OpenMP Pragmas and Directives

OpenMP directives, or pragmas, are compiler instructions used in shared-memory

programming (C/C++ and Fortran) to execute code in parallel using multiple threads.
They are the mechanism for de ning parallel regions and sharing work among
threads.

Core Directives
Directive-Purpose-Key Feature
#pragma omp **parallel** -Forks a team of threads to execute the following
structured code block concurrently. - Creates the parallel region.
#pragma omp **for** - Distributes the iterations of a loop among the threads in the
team. -Must be inside a parallel region (or combined as parallel for).
#pragma omp **sections** - Divides a block of non-iterative tasks into independent
sections to be executed concurrently. - Used for function-level parallelism.
#pragma omp **single** - Speci es that the following code block is executed by only
one thread (not necessarily the master). - Often used for I/O or initialization within
the parallel region.

Synchronization Directives
Directive - Purpose
#pragma omp **barrier** Pauses all threads until every thread in the team has
reached that point.
#pragma omp **critical** - Restricts the execution of the following block to only one
thread at a time (mutual exclusion).
#pragma omp **atomic** - Guarantees that a simple memory update (e.g.,
assignment, addition) is performed without interruption.
#pragma omp **master** - Executes the following block only by the master thread
(thread ID 0).

Data Sharing Clauses (Used with Directives)

Clauses are modi ers that de ne how variables are handled within the parallel
region.

Clause - Variable Scope - Effect

shared(list) - Shared - Variables are accessible and writable by all threads.
private(list) - Local (Private) - Each thread gets its own uninitialized copy of the
variable.
reduction(op:list) - Local (Private) - Each thread calculates its local value, and the
results are combined using the speci ed operator (+, *, max, etc.) at the end.

TRAPEZOIDAL RULE
The Trapezoidal Rule approximates the area under the curve of a function $f(x)$
over an interval $[a, b]$ by dividing the interval into $n$ smaller subintervals of equal
width $h$
Formula and ConceptInterval
Division: The interval [a, b] is divided into n subintervals, each forming the base of a
trapezoid.
Width Calculation: The width of each trapezoid (or the step size) is: h = {b - a}/{n}
Area Approximation: The area of each subinterval is approximated by the area of a
trapezoid, which is the width multiplied by the average of the two heights (function
values) at the endpoints.
fi
fi
fi
fi
fi
 Area Approximation: The area of each subinterval is approximated by the area of a
trapezoid, which is the width multiplied by the average of the two heights (function
values) at the endpoints.
Total Integral: The approximate value of the de nite integral I is the sum of the areas of all
n trapezoids:

where x_0 = a, x_n = b, and x_i = a + ih.

Parallelization with MPI

The Trapezoidal Rule is an example of an embarrassingly parallel problem because the
calculation of the area over any one subinterval is entirely independent of the calculation over
any other subinterval.
Decomposition: The total range $[a, b]$ is divided among $p$ processes (where
$p$ is the number of processes).

Local Integral: Each process is assigned a local sub-interval $[a_{local}, b_{local}]

$. Each process then calculates the sum of the trapezoids within its own local
interval. This is an independent computation phase.

Communication (Reduction): The nal step requires the partial sums (local
integrals) calculated by each process to be combined into a single, total sum. This is
achieved using an MPI reduction operation, typically MPI_Reduce, which sums the
local results and delivers the nal total to the root process (rank 0).

SCOPE OF VARIABLES

The scope of variables in an OpenMP parallel region determines whether each thread has its
own copy or shares the original variable's memory location.

1. Shared Variables

• • De nition: Variables that refer to a single memory location

accessible by all threads in the team.

• • Purpose: Enables threads to cooperate and communicate intermediate

results.

• • Risk: Requires explicit synchronization (critical, atomic) to

prevent race conditions when multiple threads write to it simultaneously.

• • Clause: shared(list)

• • Default: Most variables referenced in a parallel region are shared by

default.

2. Private Variables

fi
fi
fi
fi
 2. Private Variables

• • De nition: Each thread gets its own local copy of the variable.

• • Purpose: Allows threads to perform calculations safely without

interfering with each other.

• • Clause Variations:
◦ ◦ private(list): Each copy is uninitialized. The original
value is not copied in or out.

◦ ◦ firstprivate(list): Each copy is initialized with the

original variable's value before the parallel region starts.

◦ ◦ lastprivate(list): The copy from the thread that

completes the nal logical iteration is copied back to update the original
shared variable.

◦ ◦ reduction(op:list): Private copies are used for

accumulating a result, and the nal results are mathematically combined into
the shared variable using an operator (e.g., +, *) at the end.

• • Default: Loop iteration variables (e.g., the i in a for loop) in a

#pragma omp for construct are automatically private.

REDUCTION CLAUSE

The OpenMP reduction clause is a specialized data sharing attribute used to safely
compute a single value from the values contributed by all threads in a team.

Reduction Clause

• • Syntax: reduction(operator: list)

• • Purpose: To prevent race conditions when a shared variable

is used for a cumulative calculation (like a sum or product) by multiple
threads.

• • Mechanism:
1 1 Private Copies: For each variable in the list, a private
copy is created for every thread, initialized to the identity value of the
speci ed operator (e.g., $0$ for addition, $1$ for multiplication).

2 2 Local Calculation: Each thread performs its work and

accumulates its partial result using its private copy.

3 3 Combination: At the end of the reduction region, the

private copies from all threads are combined using the speci ed
operator (e.g., summed together).
fi
fi
fi
fi
fi
 3 3 Combination: At the end of the reduction region, the
private copies from all threads are combined using the speci ed
operator (e.g., summed together).

4 4 Update: The nal combined result is stored back into the

original shared variable.

Common Reduction Operators

The clause supports standard associative and commutative operators:

• • Arithmetic: + (Summation), * (Product), - (Subtraction, used

carefully for differences).

• • Logical: & (Bitwise AND), | (Bitwise OR), && (Logical AND), ||

(Logical OR).

• • Min/Max: max (Maximum value), min (Minimum value).

LOOP CARRIED DEPENDENCY

A loop-carried dependency is a critical concept in parallel programming that

identi es a constraint preventing the safe parallelization of a loop.

De nition

A loop-carried dependency exists when an operation in one iteration of a loop

depends on the result of an operation in a previous iteration of the same loop.

Impact on Parallelization

• • Prevents Parallelism: When a loop has a loop-carried

dependency, the iterations cannot be executed concurrently or out of their
original order (i.e., in parallel) without potentially yielding an incorrect result.
The loop must remain sequential to guarantee that the data dependency is
satis ed.

• • Race Conditions: If a loop with a loop-carried dependency is

forced into a parallel region (e.g., using #pragma omp for), a race condition
will occur, as threads will try to read or write a shared variable in an order that
violates the logical sequence of the original serial code.

📝 Example

A common example of a loop-carried dependency is a simple recurrence relation

where an element of an array is calculated based on its predecessor:

A[i] = A[i-1] * c + b
In this loop, the calculation of $A[i]$ (current iteration) depends on the value of
$A[i-1]$ (previous iteration). If threads run this loop concurrently, the thread
calculating $A[i]$ might execute before the thread calculating $A[i-1]$ has nished
writing its result, leading to a wrong value.
fi
fi
fi
fi
fi
fi
 SCHEDULING, PRODUCERS AND CONSUMERS

Scheduling
Scheduling in parallel programming refers to the process of dividing the total parallel work
(e.g., the iterations of a loop) and assigning those pieces of work to individual threads or
processes for execution.

• • Goal: To achieve load balancing by distributing work evenly,

minimizing idle time, and thereby improving parallel ef ciency.

• •
OpenMP Application (Work-Sharing): In OpenMP, the
schedule clause (used with the #pragma omp for directive) speci es how
loop iterations are assigned to threads. Common types include:
◦ ◦ Static: Iterations are divided into xed-size chunks and
assigned to threads in advance (often round-robin). This is fast but poor for
loops with variable iteration times.

◦ ◦ Dynamic: Threads are assigned a small chunk of iterations at

runtime. When a thread nishes, it requests the next chunk. This improves
load balance but incurs higher overhead.

◦ ◦ Guided: A variant of dynamic scheduling where the chunk size

is large initially and progressively decreases.

Producers and Consumers (P&C)

The Producer-Consumer model is a classic parallel design pattern used for coordinating
data ow between different tasks.

• • Producers: Tasks or processes that generate data and place it into a

shared data structure (often called a buffer or queue).

• • Consumers: Tasks or processes that read and process data retrieved

from the shared buffer.

• • Coordination: This model is crucial in environments like OpenMP

(shared memory) and MPI (message passing) to manage synchronization and resource
ow.
◦ ◦ Shared Memory (OpenMP/Pthreads): Synchronization
mechanisms like mutexes or semaphores are necessary to ensure that
producers and consumers do not access the shared buffer simultaneously,
preventing data corruption.

◦ ◦ Distributed Memory (MPI): Producers use send routines to

transfer data (messages) to consumers, and consumers use receive routines to
obtain the data, with the MPI library managing the underlying coordination.

• • Bene t: Decouples the production rate from the consumption rate,

allowing concurrent execution and improving throughput.
fl
fl
fi
fi
fi
fi
fi
 • • Bene t: Decouples the production rate from the consumption rate,
allowing concurrent execution and improving throughput.

CACHES, CACHE COHERENCE, FALSE SHARING

The relationship between caches, cache coherence, and false sharing is critical to
performance in OpenMP (shared-memory) programs.

1. Caches

• • De nition: Small, fast memory components located close to each

processor core.

• • Purpose: To store copies of frequently accessed data from main

memory. Since cache access is much faster than main memory access, this drastically
reduces the time a core spends waiting for data, improving performance.

• • Mechanism: Data is moved between main memory and the cache in

units called cache lines (typically 32 to 256 bytes).

2. Cache Coherence

• • De nition: A mechanism implemented by hardware to ensure that all

processors in a shared-memory system see a consistent view of the shared data, even
when multiple processors have a copy of that data in their local caches.

• • Problem Addressed: When one core writes a new value to a shared

variable in its cache, other cores with a copy of the old value must be noti ed or
prevented from using the outdated data.

• • Resolution: Coherence protocols (like MESI) are used to track the

state of cache lines and ensure that a write by one core invalidates or updates the
copies held by other cores.

3. False Sharing

• • De nition: A performance degradation problem that occurs when two

or more threads are modifying variables that are logically independent but happen to
reside in the same cache line.

• • Mechanism:
1 1 Thread A modi es Variable X. The entire cache line
containing X and Y (and potentially Z) is marked as modi ed.

2 2 Thread B tries to modify Variable Y. Even though Y is

logically separate from X, because Y is in the same cache line, the cache
coherence protocol forces Thread B's cache to invalidate or update the entire
cache line from Thread A.

3 3 This repeated invalidation and data movement causes the

system to waste time transferring data between caches, signi cantly slowing
down the parallel program, despite the variables being independent.
fi
fi
fi
fi
fi
fi
fi
fi
 3 3 This repeated invalidation and data movement causes the
system to waste time transferring data between caches, signi cantly slowing
down the parallel program, despite the variables being independent.

• • Solution: Programmers must try to reorganize data structures to ensure

that variables frequently accessed by different threads are placed in different cache
lines.

TASKING AND THREAD SAFETY

Tasking
Tasking is an OpenMP feature that enables asynchronous (non-sequential) and dynamic
work assignment, which is particularly useful for parallelizing irregular or recursive
problems.

• • De nition: A task is a speci c unit of work (a structured block of

code) de ned by a thread but executed by any available thread in the team, often at a
later time.

• • Directive: #pragma omp **task** de nes the code block to

be executed as a task.

• • Execution: The thread that encounters the task directive usually

delegates its execution to the OpenMP runtime environment. This allows the de ning
thread to continue with its other work, leading to better load balancing for irregular
problems.

• • Synchronization: #pragma omp **taskwait** is used by a

thread to wait for all child tasks it generated to complete before continuing.

Thread Safety
Thread safety refers to the property of a code segment, function, or data structure that
guarantees correct results when executed concurrently by multiple threads.

• • De nition: A code segment is thread safe if, when executed by

multiple threads simultaneously, it produces the same result as if it were executed
sequentially (serially).

• • Unsafe Code: Occurs due to race conditions—when the outcome

depends on the unpredictable order in which threads access and modify shared
variables.

• • Achieving Safety (OpenMP): OpenMP provides explicit

synchronization directives to enforce thread safety where necessary:
◦ ◦ Using the private data clause to give each thread its own local
copy of a variable.

◦ ◦ Using synchronization directives like #pragma omp

**critical**, #pragma omp **atomic**, or #pragma omp
**barrier** to serialize access to shared resources.
fi
fi
fi
fi
fi
fi
fi
◦ ◦ Using synchronization directives like #pragma omp
**critical**, #pragma omp **atomic**, or #pragma omp
**barrier** to serialize access to shared resources.
◦ ◦ Using the reduction clause to safely combine commutative/
associative shared values.