PARALLEL COMPUTING(BCS702)

MODULE-2
GPU programming, Programming hybrid systems, MIMD systems, GPUs,
Performance – Speedup and efficiency in MIMD systems, Amdahl’s law, Scalability
in MIMD systems, Taking timings of MIMD programs, GPU performance.
TEXT BOOK1: CHAPTERS- 2.4.5 & 2.4.6, 2.5.1& 2.5.2, 2.6

GPU programming
GPUs are usually not “standalone” processors. This means they don’t normally run
an operating system or provide services like direct access to hard drives. So, when
we program a GPU, we also need to write code for the CPU (called the host) that
works with the GPU. The CPU is an ordinary processor that controls things.
The CPU and GPU usually have separate memory. This means the program running
on the CPU must:
• Allocate memory for both CPU and GPU.
• Copy data between them.
• Start the GPU program.
• Collect the results from the GPU program.
So GPU programming is called heterogeneous programming, because we are
programming two types of processors: CPU and GPU.
The GPU itself contains many processors, and each processor can run hundreds or
even thousands of threads (tiny programs running in parallel). In the systems we’ll
use:
• All processors share a large block of GPU memory.
• But each processor also has its own small, very fast memory.
• This small memory works like a cache, but the programmer has to manage it
directly.
The GPU threads are divided into groups. Inside a group, threads usually follow the
SIMD model (Single Instruction, Multiple Data). This means all threads in the group
run the same instruction on different data. Threads in different groups can run
independently.
However, threads in the same group may not always run in perfect lockstep. In other
words, not all threads always run the same instruction at the same time. But one
important rule is:
• A thread cannot move to the next instruction until all threads in the group
finish the current instruction.
1|Pa ge Dept. of CS&E
PARALLEL COMPUTING(BCS702)

If there’s a branch (like an if statement), some threads might take one path and others
might take another path. In that case, some threads will be idle (waiting) while others
are working.
For example, suppose:
• There are 32 threads in a SIMD group.
• Each thread has a private variable called rank_in_gp that goes from 0 to 31.
• Now, if the threads are running a piece of code that depends on rank_in_gp,
some threads might run one branch, and others might run a different branch.
This causes some threads to wait while others finish, which can reduce performance.
// Thread private variables
i n t rank_in_gp , my_x ;
...
i f ( rank_in_gp < 16)
my_x += 1 ;
else
my_x -= 1 ;
In this example, each thread has private variables such as rank_in_gp and my_x. The
variable rank_in_gp identifies the thread’s number in the group, while my_x is just a
value the thread works with. If the thread’s rank is less than 16, the thread will add 1
to my_x. If the rank is 16 or greater, the thread will subtract 1 from my_x.
When this code runs, only the threads with rank less than 16 are active, while the rest
must wait. After these finish, the threads with rank greater than or equal to 16 will
execute their instruction, and now the first half will be idle. This means that only half
the threads are working at a time. Since the GPU is built to run all threads together,
idling half the threads wastes resources. This situation is called branch divergence,
and it reduces efficiency. Therefore, programmers try to avoid code where threads in
the same group split into different branches.
Another difference between CPU and GPU programming is the way threads are
scheduled. CPUs use software to schedule threads, which adds some overhead.
GPUs, on the other hand, use a hardware scheduler, which is much faster and more
efficient. However, the GPU scheduler only allows a group of threads to move
forward when all threads in that group are ready. For example, before checking the
condition in the code, it is best if the value of rank_in_gp is already stored in a fast
register for each thread, so that execution is not delayed.

2|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

To make sure the GPU stays busy, programmers usually create a large number of
thread groups. This way, if some groups are not ready to run, perhaps they are waiting
for data from memory or for another instruction to finish, the scheduler can simply
switch to another group that is ready. This allows the GPU to always keep doing
useful work, even if some groups are stalled.
Programming hybrid systems
We should note that it is possible to program systems such as clusters of multicore
processors by using two types of APIs together: a shared-memory API for the nodes
themselves and a distributed-memory API for communication between different
nodes. However, this is usually done only for programs that need the very highest
performance, because using both APIs together makes programming much more
complex. Instead, most of the time, such systems are programmed with just one
distributed-memory API, which is used for both communication between nodes and
communication inside the nodes.

2.5 Input and output

2.5.1 MIMD systems
So far, we’ve mostly avoided the topic of input and output (I/O). There are two
reasons for this. The first is that parallel I/O—where multiple cores read from or
write to multiple disks or devices at the same time—is a huge subject, large enough
to fill an entire book. The second reason is that most of the programs we’ll write don’t
need much I/O. They usually read and write only small amounts of data, which can
easily be handled by the standard C I/O functions like printf, fprintf, scanf, and fscanf.
However, even using these standard C functions can create some problems. Since C
is a serial language, the standard does not say what should happen if multiple
processes try to use these functions at the same time. Threads that come from a single
process do share the standard input, output, and error streams (stdin, stdout, stderr).
But, as we’ve already seen, when multiple threads try to write to the same stream,
the results are unpredictable—we don’t know exactly what will happen.
For example, when we call printf from multiple processes or threads, we usually want
the output to appear on the same console where we started the program. Most systems
do this. But with processes, there is no guarantee. In some cases, only one process
may have access to stdout or stderr, and in very rare cases, no process may be able to
access them at all.
The situation with scanf is even less clear. Should the input from the keyboard be
divided among processes or threads? Or should only one process or thread be allowed
to use scanf? In most systems, at least one process (usually process 0) is allowed to

3|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

use scanf. Most systems also allow multiple threads to use it. But once again, there
are some systems that don’t allow any process to use scanf.
If multiple processes or threads can use stdin, stdout, or stderr, then the results are
usually nondeterministic. This means the order of the output might change every
time the program runs, and sometimes the output from one process or thread might
get mixed up with the output from another. Similarly, for input, different processes
or threads might read different parts of the input on each run, even if the input itself
doesn’t change.
To avoid these issues, we’ll follow some clear rules when writing parallel programs
that need I/O:
• In distributed-memory programs, only process 0 will use stdin. In shared-
memory programs, only the master thread or thread 0 will use stdin.
• In both distributed-memory and shared-memory programs, all processes and
threads can use stdout and stderr.
• Because the output to stdout is nondeterministic, usually only one process or
thread will be used for normal output. The main exception is when we are
debugging, where it can be useful to allow multiple processes or threads to
write to stdout.
• Only one process or thread will access any file other than stdin, stdout, or
stderr. For example, each process or thread can open its own private file, but
no two processes or threads should open the same file.
• Debugging output should always include the rank or ID of the process or
thread that is writing the output, so we know where the message came from.

2.5.2 GPUs

In GPU programming, most of the input and output (I/O) operations are handled by
the host code, which is the part of the program that runs on the CPU. Since the host
usually runs only a single process or thread, the standard C I/O functions (like printf
or scanf) work exactly the same way as they do in ordinary serial C programs.
However, there is an exception when it comes to debugging GPU code. Sometimes
we may want to directly write output from GPU threads for debugging purposes. In
the systems commonly used, each GPU thread is allowed to write to stdout. But
since many threads may attempt to print simultaneously, the order of output becomes
nondeterministic meaning that the sequence of printed lines is unpredictable, similar
to what happens in MIMD programs. At the same time, GPU threads generally do
not have access to other I/O resources such as stderr, stdin, or disk storage. This
restriction emphasizes that I/O in GPU programs is mainly the responsibility of the
host, while GPU threads are meant primarily for computation.

4|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

2.6 Performance

The main motivation for writing parallel programs, whether on CPUs or GPUs, is to
increase performance. However, before evaluating performance, it is important to
distinguish between different parallel architectures. In this section, we first examine
homogeneous MIMD systems, where all processor cores have the same
architecture. Later, we will separately discuss GPUs, since their structure and
performance behavior are different from traditional CPU-based systems.

2.6.1 Speedup and Efficiency in MIMD Systems

Ideally, the best a parallel program can achieve is to divide the work equally among
all available cores, without introducing any extra overhead. If this ideal case is
achieved, then running the program with p cores would make it run p times faster
compared to the serial version on a single core. For example, if the runtime of the
serial program is denoted as Tserial, and the runtime of the parallel program using
p cores is Tparallel, then in the best case: Tparallel = Tserial/p.

This condition is called linear speedup, since the speedup grows directly in
proportion to the number of cores used.

However, in practice, achieving perfect linear speedup is rare. The reason is that
running multiple threads or processes inevitably introduces overheads. For instance,
in shared-memory programs, there are often critical sections—parts of the code
where only one thread can execute at a time. To enforce this, programmers must use
synchronization tools such as mutexes. These mutex operations themselves add extra
time (overhead), and they also serialize the execution of the critical section, reducing
parallel efficiency. Similarly, in distributed-memory programs, data often needs to
be transmitted across a network, which is much slower than local memory access.
This communication cost is another form of overhead absent in a serial program.

Therefore, real-world parallel programs rarely achieve perfect linear speedup.

Moreover, as the number of threads or processes increases, these overheads tend to
grow. With more threads, there is a higher chance that multiple threads will compete
for the same critical section. With more processes, more data must be exchanged
across the network. As a result, the actual performance gain is usually less than
linear, and efficiency decreases with scale. So if we define the speedup of a parallel
program to be

5|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

When a parallel program achieves linear speedup, the speedup SSS is exactly equal
to the number of cores, ppp. In other words,
S=p
This means if a program runs on 8 cores, it would ideally be 8 times faster than
running on a single core. However, in practice, things are not so perfect. As the
number of cores ppp increases, the parallel overhead also tends to grow. This
overhead could come from synchronization, communication, or waiting for data. As
a result, the actual speedup SSS becomes a smaller and smaller fraction of the ideal
value ppp.
Another way to describe this is by looking at the ratio S/pS/pS/p. This ratio measures
how close the achieved speedup is to the ideal linear speedup. For example, if 8 cores
only give a speedup of 6, then S/p=6/8=0.75. As we increase p, this fraction S/p
usually decreases, because the overheads become more significant at larger scales.
This ratio S/p is very important in parallel computing, and it is called the efficiency
of the parallel program. Efficiency essentially tells us how effectively the processors
are being used. A value close to 1 means nearly perfect utilization, while smaller
values indicate wasted potential due to overhead.
If we substitute the formula for speedup into the expression for efficiency, we get:

Thus, efficiency provides a normalized measure of performance, allowing us to

compare different parallel programs or systems, regardless of the number of cores
being used.
If the serial run-time is measured on the same type of core that the parallel system is
using, then efficiency can be directly interpreted as the average utilization of the
parallel cores in solving the problem. In other words, efficiency tells us what fraction
of the time each core is actually doing useful work on the original problem, rather
than waiting, synchronizing, or dealing with communication.
So, if the efficiency is 0.8 (or 80%), it means that on average, each core spends 80%
of its time working on the computation, while the remaining 20% is lost to parallel
overhead. The overhead could come from delays such as waiting to enter a critical
section, transferring data across the network, or handling synchronization between
processes.
This idea can be seen mathematically by multiplying the efficiency with the parallel
run-time. Since efficiency is defined as

6|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

This means that, on average, each process spends (3/4)* 4=3ms actually solving the
original problem, while the remaining 4−3=1 ms is lost to parallel overhead.
Many parallel programs are designed by dividing the work of the serial program
among multiple processes or threads and then adding the extra work required for
synchronization or communication. If we let Toverhead denote this additional cost,
we often write the parallel run-time as:

This formula makes it clear that efficiency is simply the fraction of the parallel run-
time spent solving the original problem. The first term Tserial/p represents useful
work, while Toverhead represents wasted time caused by coordination among
processes.

It is also important to remember that performance depends not only on the number
of processes p, but also on the size of the problem being solved. Both Tserial and
Tparallel change as the problem size changes. For example, if we halve the problem
size or double it, the speedups and efficiencies will also change. This effect is shown
in Tables 2.4 and 2.5 of the text, with the corresponding graphs in Figures 2.18 and
2.19. In general, when the problem size increases while the number of processes
remains fixed, speedup and efficiency both improve. The reason is that the serial
work grows much faster than the parallel overhead. Thus, the fraction of time wasted
on overhead becomes smaller relative to the useful work.
Finally, we need to decide what serial run-time Tserial should be used when reporting
speedup and efficiency. Some authors prefer to use the run-time of the fastest
available serial algorithm on the fastest processor. Others, however, argue that
efficiency is best understood as the utilization of the processors in the parallel system,

7|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

and so they use the serial program corresponding to the parallel version, run on a
single core of the same system.

For example, if we are studying a parallel shell sort, the first approach might compare
it to a fast radix sort or quicksort, while the second approach would compare it to a
serial shell sort. In this book, we will generally follow the second approach, since it
makes efficiency directly reflect how well the parallel system is using its own
processors.

Table 2.5 – Speedups and efficiencies of a parallel program on different problem

sizes

8|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

This table shows the same program but tested with different problem sizes: Half-
size, Original size, and Double size.
1. Half-size problem
• With fewer computations, overhead dominates more.
• For p=16, speedup is only 6.2, and efficiency drops to 0.39 (39%).
• If the problem is too small, adding many processors wastes resources.
2. Original-size problem
• These values are the same as Table 2.4.
• At p=16, speedup is 10.8 with efficiency 0.68.
• A medium-size problem gives reasonable speedup and efficiency.
3. Double-size problem
• With larger computations, overhead becomes less significant.
• For p=16, speedup improves to 14.2 and efficiency is 0.89 (very high).
• Bigger problems benefit more from parallelism.

Amdahl’s Law

In the 1960s, Gene Amdahl introduced a principle that has become one of the
most widely cited results in parallel computing. This principle, called Amdahl’s
law, gives us a way to estimate the maximum speedup we can obtain when
converting a serial program into a parallel one. The key idea is simple: even if a
large portion of a program can be parallelized, the parts that cannot be parallelized
will eventually dominate the total execution time as we increase the number of
processors.

Let us understand this with an example. Suppose we take a program that, when
executed on a single processor, has a serial runtime of Tserial = 20 seconds. We
manage to parallelize 90% of the code, leaving 10% as inherently serial. If the
parallelized section can achieve perfect speedup, then using p processors, the
time taken for this section will be:

0.9×Tserial/p=18/p

The non-parallel section, which is 10% of the program, will always take the same
amount of time regardless of the number of processors:

0.1×Tserial=2

Therefore, the total parallel runtime becomes:

9|Pa ge Dept. of CS&E

PARALLEL COMPUTING(BCS702)

Tparallel=0.9×Tserial/p+0.1×Tserial=18/p+2

Now, the speedup S is defined as the ratio of the serial runtime to the parallel
runtime:

When the number of processors p increases, the parallel portion 18/p becomes very
small, almost approaching zero. However, the serial part (2 seconds) remains
unchanged. Thus, the best possible runtime will always be greater than or equal to 2
seconds, no matter how many processors we add. Hence, the maximum possible
speedup is:

This means that even with thousands of processors, the maximum achievable
speedup for this program is 10.
In general terms, if a fraction r of the program is inherently serial, then the maximum
speedup that can ever be achieved is:

For our example, r = 0.1, so the maximum speedup is 10. If we imagine a program in
which only 1% of the work is serial (r = 1/100), then no matter how many processors
we use, the speedup will never be more than 100.
This result may look discouraging, but it should not be misunderstood. Amdahl’s law
assumes that the problem size is fixed. In many real-world applications, however,
when we increase the number of processors, we also increase the problem size (for
example, simulating more particles, processing more images, or solving larger
equations). In such cases, the proportion of the inherently serial section often
becomes smaller as the problem grows. This observation is formalized in Gustafson’s
law, which suggests that speedup can keep increasing as the problem size scales up
with more processors.
Also, in practice, a speedup of 5, 10, or even less can be extremely valuable. For
example, if a simulation that takes 10 hours can be reduced to 1 hour, that is still a
huge improvement for scientists and engineers. Thus, while Amdahl’s law shows a
limit, it does not mean parallel programming is unhelpful; it only reminds us that we
must always account for the serial fraction of code.

10 | P a g e Dept. of CS&E
PARALLEL COMPUTING(BCS702)

Scalability in MIMD Systems

The concept of scalability describes how well a parallel program can make use of
additional computing resources. In simple terms, a program is said to be scalable if,
when we increase the number of processors or threads, the program’s performance
improves in a predictable and efficient way. More formally, scalability refers to the
ability of a program to maintain its efficiency when the number of processors and the
size of the problem are increased together.
Let us look at an example to understand this more clearly. Suppose the serial runtime
of a problem of size n is:

Tserial=n
and the parallel runtime is:
Tparallel=n/p+1

Here, n/p represents the portion of the work that gets divided among p processors,
while the constant 1 represents the part of the work that cannot be parallelized (or
some overhead cost of parallelization).
The efficiency E of the program is defined as:

Now suppose we increase the number of processors by a factor of k, so the number

of processors becomes kp. At the same time, we also increase the problem size by a
factor of x, making it xn. To check whether the program is scalable, we ask: is it
possible to choose x so that the efficiency remains the same?
We set up the equation:

11 | P a g e Dept. of CS&E
PARALLEL COMPUTING(BCS702)

This shows that if we increase the problem size at the same rate as we increase the
number of processors, then the efficiency remains unchanged. In other words, the
program is scalable.
There are two special types of scalability:
• Strong scalability: If efficiency stays constant even when the problem size
is fixed and only the number of processors is increased, the program is
strongly scalable.
• Weak scalability: If efficiency stays constant only when the problem size is
increased at the same rate as the number of processors, the program is
weakly scalable.
In our example, the program is weakly scalable because we had to increase the
problem size in proportion to the number of processors to maintain efficiency.
Scalability is important in practice because it tells us how a parallel algorithm will
behave on larger and larger systems. If a program is weakly scalable, it can still
perform well when run on thousands of processors, provided the problem size also
grows. Strong scalability, on the other hand, is more difficult to achieve but is
highly desirable, since it means we can solve a fixed-size problem faster simply by
adding more processors.

Taking Timings of MIMD Programs

When working with parallel programs, one of the fundamental tasks is to measure
how long the program takes to run. These timings allow us to evaluate performance,
detect inefficiencies, and compare different implementations. However, taking
timings in MIMD (Multiple Instruction, Multiple Data) systems is not always
straightforward. There are several important considerations that programmers must
keep in mind.
The first point to understand is that we measure timings for two different reasons.
During program development, timings are often used for debugging and analysis. For
instance, in a distributed-memory program, a process may spend a significant amount
of time waiting for messages to arrive from other processes. If this waiting time is
large, it suggests there may be a design flaw in the communication pattern, or an
inefficient implementation of message passing. In such cases, we need detailed
timings that show how much time is being spent in different sections of the code. On
the other hand, once development is complete, our main goal is to measure the
overall performance of the program. In this case, we usually only need a single
timing value that reports the total runtime of the important section of the code.
The second point is that we are usually not interested in the total program runtime
from start to finish. For example, consider a program that reads a dataset, sorts it
using bubble sort, and then prints the results. If we want to study the efficiency of the

12 | P a g e Dept. of CS&E
PARALLEL COMPUTING(BCS702)

sorting algorithm, the relevant timing is only the time taken to sort the keys, not the
time for input and output operations. Therefore, generic operating system tools like
the Unix time command, which report the total program execution time, are not
suitable for this type of measurement.
The third important point is the distinction between CPU time and wall clock time.
The standard C function clock() returns CPU time, which is the amount of time the
processor spends executing the program’s instructions. This includes user-written
code, library functions (like pow or sin), and operating system calls (like printf and
scanf). However, CPU time does not include the periods when the program is idle.
This becomes problematic in parallel programs, especially distributed-memory ones.
For example, if a process calls a receive function but the matching send has not yet
been issued, the operating system may put the process to sleep. This idle waiting time
will not be counted as CPU time, but from the perspective of overall program
performance, it must be included. Ignoring it would give a misleading picture.
For this reason, the most appropriate measurement in parallel programs is wall clock
time. This is the actual elapsed time from the start of the timed section to its finish,
similar to starting a stopwatch when the code begins and stopping it when it ends. In
practice, we insert code into the program such as:
double start, finish;
...
start = Get_current_time();
/* Code that we want to time */
...
finish = Get_current_time();
printf("The elapsed time = %e seconds\n", finish - start);
Here, Get_current_time() is a placeholder function that returns the wall clock time in
seconds. The actual function depends on the API. For example, in MPI, we can use
MPI_Wtime(), and in OpenMP, we can use omp_get_wtime().
An additional issue is timer resolution. Resolution is the smallest measurable unit
of the timer. Some timers have millisecond resolution (10⁻³ seconds), while modern
processors can execute billions of instructions per second (with times in
nanoseconds, 10⁻⁹ seconds). If the timer’s resolution is too coarse, it may require
millions of instructions before reporting a nonzero time. Most APIs provide either a
function that reports the resolution or a guarantee about the timer’s accuracy, and
programmers must be aware of this.
Timing in parallel programs has another complexity: multiple processes or threads
are executing simultaneously. If each one independently reports its elapsed time,

13 | P a g e Dept. of CS&E
PARALLEL COMPUTING(BCS702)

we will obtain multiple values, one per process. What we actually want, however, is
a single value that reflects the elapsed time from when the first process starts to when
the last process finishes. Since clocks on different nodes are not synchronized, this
exact measurement is difficult. Instead, we adopt an approximation. We first
synchronize all processes using a barrier, then each process measures its own elapsed
time, and finally we compute the maximum elapsed time across all processes. The
code looks like this:
shared double global_elapsed;
private double my_start, my_finish, my_elapsed;
...
/* Synchronize all processes/threads */
Barrier();
my_start = Get_current_time();
/* Code that we want to time */
...
my_finish = Get_current_time();
my_elapsed = my_finish - my_start;
/* Find the maximum across all processes */
global_elapsed = Global_max(my_elapsed);
if (my_rank == 0)
printf("The elapsed time = %e seconds\n", global_elapsed);
This method ensures that the reported time reflects the slowest process or thread,
which is effectively the actual elapsed runtime.
Another practical consideration is variability in timings. Even if we run the program
with the same input on the same system multiple times, the measured times will
usually vary slightly. This variability may be due to operating system scheduling,
background processes, or network delays. Instead of reporting the mean or median
runtime, it is common practice to report the minimum runtime, since no external
factor can make a program run faster than its true best performance.
Finally, it is worth noting that running multiple threads per core increases
variability and adds scheduling overhead, so in practice we rarely do so. Also,
input/output (I/O) operations are typically excluded from performance timings, since
most high-performance applications are not optimized for I/O speed.

14 | P a g e Dept. of CS&E
PARALLEL COMPUTING(BCS702)

GPU Performance
In MIMD systems, we generally evaluate performance by comparing the runtime of
a parallel program against the runtime of its serial version on the same type of core.
However, things are different for GPU (Graphics Processing Unit) programs. GPUs
are designed with massively parallel cores that are very different from conventional
CPU cores, so the traditional notions of speedup, efficiency, and scalability do not
always apply in the same way.
For example, one often sees reports that a GPU program achieves a speedup of 100×
or more compared to a CPU program. While these results are impressive, they are
not directly comparable to MIMD performance measures. This is because GPU cores
and CPU cores have fundamentally different architectures and execution models. As
a result, the formal definition of efficiency (speedup divided by number of
processors) is not meaningful for GPUs, nor is the idea of linear speedup relative to
a serial CPU program.
That said, researchers still use the informal term scalability when talking about
GPUs. A GPU program is called scalable if increasing the size of the GPU (i.e., using
more GPU cores or a more powerful GPU) results in performance improvements
compared to a smaller GPU.
Amdahl’s law can still apply in the context of GPU programs, but only under certain
conditions. If the inherently serial portion of the program must run on a conventional
CPU, then the same bound applies: if a fraction r of the program is serial, the
maximum possible speedup is 1/r. This means that even with a powerful GPU, the
non-parallelizable part running on the CPU will limit the total speedup. As with
MIMD programs, the caveats remain the same: the serial fraction may shrink as
problem sizes increase, so real-world speedups can still be very large, and even
modest improvements may be valuable.
The principles of timing GPU programs are also similar to those for MIMD
programs. Typically, a GPU program is launched and managed by a CPU. Therefore,
we can measure GPU performance simply by using the CPU’s timer. We start the
timer just before launching the GPU kernel and stop it right after the kernel completes
execution. This gives us the wall clock time for the GPU section of the program.
In more complex cases, such as when multiple CPU-GPU pairs are used, or when
only specific kernels on the GPU need to be timed, additional care must be taken.
Nonetheless, the general practice is the same: report wall clock time, avoid CPU-only
measures like clock(), and always be aware of timer resolution and variability.

15 | P a g e Dept. of CS&E