Pune Vidyarthi Griha’s

COLLEGE OF ENGINEERING, NASHIK – 3.

“PARALLEL PROGRAMMING”

By
Prof. Anand N. Gharu
(Assistant Professor)
PVGCOE Computer Dept.

20th July 2018

.
CONTENTS :-
1. Principles of Parallel Algorithm Design: Preliminaries

2. Decomposition Techniques

3. Characteristics of Tasks and Interactions

4. Mapping Techniques for Load Balancing

5. Methods for Containing Interaction Overheads

6. Parallel Algorithm Models

7. The Age of Parallel Processing

8. the Rise of GPU Computing

9. A Brief History of GPUs, Early GPU.

 Contents
• Introduction to Parallel Algorithms
– Tasks and Decomposition
– Processes and Mapping
– Processes Versus Processors

• Decomposition Techniques
– Recursive Decomposition
– Recursive Decomposition
– Exploratory Decomposition
– Hybrid Decomposition

• Characteristics of Tasks and Interactions

– Task Generation, Granularity, and Context
– Characteristics of Task Interactions.
 Contents
• Mapping Techniques for Load Balancing
– Static and Dynamic Mapping

• Methods for Minimizing Interaction Overheads

– Maximizing Data Locality
– Minimizing Contention and Hot-Spots
– Overlapping Communication and Computations
– Replication vs. Communication
– Group Communications vs. Point-to-Point Communication

• Parallel Algorithm Design Models

– Data-Parallel, Work-Pool, Task Graph, Master-Slave,
Pipeline, and Hybrid Models
Principles of Parallel Algorithm
Design
❖ Parallel algorithm accepts inputs from user and

execute several instructions simultaneously on

different processing units.

❖ The output from the different units are combined to form

the overall output of the algorithm.

Principles of Parallel Algorithm
Design
Preliminaries: Decomposition, Tasks, and
Dependency Graphs
Preliminaries: Decomposition, Tasks, and
Dependency Graphs
 Preliminaries: Decomposition, Tasks, and
Dependency Graphs
• The first step in developing a parallel algorithm is to decompose
the problem into tasks that can be executed concurrently

• A given problem may be docomposed into tasks in many different

ways.

• Tasks may be of same, different, or even interminate sizes.

• A decomposition can be illustrated in the form of a directed graph

with nodes corresponding to tasks and edges indicating that the
result of one task is required for processing the next. Such a graph
is called a task dependency graph.
 Example: Multiplying a Dense Matrix with a
Vector
A b y
0 1 n
Task 1
2

n-1 Task n

Computation of each element of output vector y is independent of other elements. Based on this, a
dense matrix-vector product can be decomposed into n tasks. The figure highlights the portion of the
matrix and vector accessed by Task 1.

Observations: While tasks share data (namely, the vector b), they do not
have any control dependencies – i.e., no task needs to wait for the (partial)
completion of any other. All tasks are of the same size in terms of number
of operations. Is this the maximum number of tasks we could decompose
this problem into?
 Example: Database
Query Processing
Consider the execution of the query:
MODEL = ‘‘CIVIC’’ AND YEAR = 2001 AND
(COLOR = ‘‘GREEN’’ OR COLOR = ‘‘WHITE)

on the following database:

ID# Model Year Color Dealer Price
4523 Civic 2002 Blue MN $18,000
3476 Corolla 1999 White IL $15,000
7623 Camry 2001 Green NY $21,000
9834 Prius 2001 Green CA $18,000
6734 Civic 2001 White OR $17,000
5342 Altima 2001 Green FL $19,000
3845 Maxima 2001 Blue NY $22,000
8354 Accord 2000 Green VT $18,000
4395 Civic 2001 Red CA $17,000
7352 Civic 2002 Red WA $18,000
 Example: Database Query Processing
The execution of the query can be divided into subtasks in various ways.
Each task can be thought of as generating an intermediate table of entries
that satisfy a particular clause.
ID# Year
ID# Model ID# Color
7623 2001
4523 Civic 6734 2001 7623 Green
6734 Civic ID# Color 9834 Green
5342 2001
4395 Civic 3845 2001 3476 White 5342 Green
4395 2001 6734 White 8354 Green
7352 Civic

Civic 2001 White Green

ID# Color
3476 White
ID# Model Year 7623 Green
6734 Civic 2001 Civic AND 2001 White OR Green 9834 Green
4395 Civic 2001 6734 White
5342 Green
8354 Green

Civic AND 2001 AND (White OR Green)

ID# Model Year Color

6734 Civic 2001 White

Decomposing the given query into a number of tasks. Edges in this graph
denote that the output of one task is needed to accomplish the next.
 Example: Database Query Processing
Note that the same problem can be decomposed into subtasks in other ways as
well.
ID# Year
ID# Model ID# Color
7623 2001
4523 Civic 6734 2001 7623 Green
6734 Civic ID# Color 9834 Green
5342 2001
4395 Civic 3845 2001 3476 White 5342 Green
7352 Civic 4395 2001 6734 White 8354 Green

Civi 2001 White Green

c

ID# Color
3476 White
White OR 7623 Green
Green 9834 Green
6734 White
5342 Green
8354 Green

2001 AND (White or ID# Color Year

Green) 7623 Green 2001
6734 White 2001
5342 Green 2001

Civic AND 2001 AND (White OR

Green)
ID# Model Year Color
6734 Civic 2001 White

An alternate decomposition of the given problem into subtasks,

along with their data dependencies.
Different task decompositions may lead to significant differences with
respect to their eventual parallel performance.
Granularity of Task Decompositions
• The number of tasks into which a problem is decomposed determines
its granularity.

• Decomposition into a large number of tasks results in fine- grained

decomposition and that into a small number of tasks results in a coarse
grained decomposition.
0 1A ... b n y
Task 1

Task 2

Task 3

Task 4

A coarse grained counterpart to the dense matrix-vector product

example. Each task in this example corresponds to the computation of three
elements of the result vector.
 Degree of Concurrency
• The number of tasks that can be executed in parallel is the
degree of concurrency of a decomposition.

• Since the number of tasks that can be executed in parallel may change
over program execution, the maximum degree of concurrency is the
maximum number of such tasks at any point during execution. What is the
maximum degree of concurrency of the database query examples?

• The average degree of concurrency is the average number of tasks that

can be processed in parallel over the execution of the program.
Assuming that each tasks in the database example takes identical
processing time, what is the average degree of concurrency in each
decomposition?

• The degree of concurrency increases as the decomposition becomes finer

in granularity and vice versa.
 Critical Path Length
• A directed path in the task dependency graph represents a
sequence of tasks that must be processed one after the other.

• The longest such path determines the shortest time in which the
program can be executed in parallel.

• The length of the longest path in a task dependency graph is

called the critical path length.
 Critical Path Length
Consider the task dependency graphs of the two database
query decompositions:
Task 4 Task 3 Task 2 Task 1 Task 4 Task 3 Task 2 Task 1

10 10 10 10 10 10 10 10

6 Task 5
9 Task 6 6 Task 5
11 Task 6

8 Task 7 7 Task 7

(a) (b)

What are the critical path lengths for the two task dependency graphs? If each
task takes 10 time units, what is the shortest parallel execution time for each
decomposition? How many processors are needed in each case to achieve this
minimum parallel execution time? What is the maximum degree of concurrency?
Task Dependency Graphs
Task Dependency Graphs
Task Dependency Graphs
 Task Interaction Graphs
• Subtasks generally exchange data with others in a
decomposition. For example, even in the trivial decomposition
of the dense matrix-vector product, if the vector is not
replicated across all tasks, they will have to communicate
elements of the vector.

• The graph of tasks (nodes) and their interactions/data

exchange (edges) is referred to as a task interaction
graph.

• Note that task interaction graphs represent data dependencies,

whereas task dependency graphs represent control dependencies.
 Task Interaction Graphs: An Example
Consider the problem of multiplying a sparse matrix A with a vector b.
The following observations can be made:
• As before, the computation of each element of the result vector can be viewed as an
independent task.
• Unlike a dense matrix-vector product though, only non-zero elements of matrix A
participate in the computation.
• If, for memory optimality, we also partition b across tasks, then one can see that the
task interaction graph of the computation is identical to the graph of the matrix A (the
graph for which A represents the adjacency structure).
A b 0
2 3
1
Task 0 0 1 2 3 4 5 6 7 8 9 1011

5
4
4 6

7
8

9
Task 11 8 10 11

(a) (b)
 Processes and Mapping
• In general, the number of tasks in a decomposition exceeds the
number of processing elements available.

• For this reason, a parallel algorithm must also provide a

mapping of tasks to processes.

Note: We refer to the mapping as being from tasks to processes, as

opposed to processors. This is because typical programming APIs, as we
shall see, do not allow easy binding of tasks to physical processors.
Rather, we aggregate tasks into processes and rely on the system to map
these processes to physical processors. We use processes, not in the UNIX
sense of a process, rather, simply as a collection of tasks and associated
data.
 Processes and Mapping
• Appropriate mapping of tasks to processes is critical to the
parallel performance of an algorithm.

• Mappings are determined by both the task dependency and task

interaction graphs.

• Task dependency graphs can be used to ensure that work is

equally spread across all processes at any point (minimum idling
and optimal load balance).

• Task interaction graphs can be used to make sure that processes

need minimum interaction with other processes (minimum
communication).
 Processes and Mapping
An appropriate mapping must minimize parallel execution time by:

• Mapping independent tasks to different processes.

• Assigning tasks on critical path to processes as soon as they become

available.

• Minimizing interaction between processes by mapping tasks with dense

interactions to the same process.

Note: These criteria often conflict with each other. For example, a
decomposition into one task (or no decomposition at all) minimizes
interaction but does not result in a speedup at all! Can you think of other
such conflicting cases?
Processes and Mapping: Example
Task 4 Task 3 Task 2 Task 1 Task 4 Task 3 Task 2 Task 1

10 10 10 10 10 10 10 10
P3 P2 P1 P0 P3 P2 P1 P0

P0 6 Task 5
P2 9 Task 6 P0 6 Task 5
P0 11 Task 6

P0 8 Task 7 P0 7 Task 7

(a) (b)

Mapping tasks in the database query decomposition to processes.

These mappings were arrived at by viewing the dependency graph in
terms of levels (no two nodes in a level have dependencies). Tasks
within a single level are then assigned to different processes.
Processes and Mapping: Example

(a) (b)
Processes and Mapping: Example

(a) (b)
Decomposition Techniques
So how does one decompose a task into various subtasks?
While there is no single techniques that works for all problems,
we present a set of commonly used techniques that apply to broad
classes of problems. These include:

• recursive decomposition

• data decomposition

• exploratory decomposition

• speculative decomposition
Recursive Decomposition
• Generally suited to problems that are solved using the divide-
and-conquer strategy.

• A given problem is first decomposed into a set of sub-

problems.

• These sub-problems are recursively decomposed further until

a desired granularity is reached.
 Recursive Decomposition: Example
A classic example of a divide-and-conquer algorithm on which we
can apply recursive decomposition is Quicksort.
5 12 11 1 10 6 8 3 7 4 9 2

1 3 4 2 5 12 11 10 6 8 7 9

1 2 3 4 5 6 8 7 9 12 11 10

1 2 3 4 5 6 7 8 9 10 12 11

5 6 7 8 10 11 12

11 12

In this example, once the list has been partitioned around the pivot, each sublist can be
processed concurrently (i.e., each sublist represents an independent subtask). This can be
repeated recursively.
 Data Decomposition
• Identify the data on which computations are performed.

• Partition this data across various tasks.

• This partitioning induces a decomposition of the problem.

• Data can be partitioned in various ways – this critically

impacts performance of a parallel algorithm.
 Data Decomposition: Output
Data Decomposition
• Often, each element of the output can be computed
independently of others (but simply as a function of the input).

• A partition of the output across tasks decomposes the problem

naturally.
Output Data Decomposition: E.g.
Consider the problem of multiplying two n × n matrices A and B to
yield matrix C. The output matrix C can be partitioned into four tasks
as follows:

. Σ . Σ . Σ
A 1 ,1 A 1 ,2 B 1 ,1 B 1 ,2 C 1 ,1 C 1 ,2
. →
A 2 , 1 A 2 ,2 B 2 , 1 B 2 ,2 C 2 , 1 C 2 ,2

Task 1: C1,1 = A1,1B1,1 + A1,2B2,1

Task 2: C1,2 = A1,1B1,2 + A1,2B2,2

Task 3: C2,1 = A2,1B1,1 + A2,2B2,1

Task 4: C2,2 = A2,1B1,2 + A2,2B2,2

Output Data Decomposition: E.g.
Exploratory Decomposition
• In many cases, the decomposition of the problem goes hand- in-
hand with its execution.

• These problems typically involve the exploration (search) of a

state space of solutions.

• Problems in this class include a variety of discrete optimization

problems (0/1 integer programming.), theorem proving, game
playing, etc.
Exploratory Decomposition: E.g.
A simple application of exploratory decomposition is in the solution to a
15 puzzle (a tile puzzle). We show a sequence of three moves that
transform a given initial state (a) to desired final state (d).

1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
5 6 8 5 6 7 8 5 6 7 8 5 6 7 8
9 10 7 11 9 10 11 9 10 11 9 10 11 12
13 14 15 12 13 14 15 12 13 14 15 12 13 14 15

(a) (b) (c) (d)

Of-course, the problem of computing the solution, in general, is

much more difficult than in this simple example.
Exploratory Decomposition: E.g.
The state space can be explored by generating various successor states

1 2 3 4
5 6 7 8
9 1 1
0 15 121
13 14
1 2 3 4 1 23 4
task 4

5 6 7 8 5 67
9 10 9 10 11 8
11 15 12
13 14 13 14 15 12
1 2 3
4
5
9 6 7
8
of the current state and to view them as independent tasks.

13 14 15
10 11
12
1 2 3 4
5 6 7 8
9 1 1
0 15 121
13 14
1 2 3 4
5 6 7 8
1 2 3 4 9 10
5 6 7 8 1112
13 14 15
9 10 11
1 2 3 4
13 14 15 12
5 7 8
task 3

9 6 10
1112
13 14 15
1 2 3 4
1 2 3 4 5 6 7 8
9 14 10
5 6 7 8 1 11
3 15 12
9 10 11 1 2 3 4
5 6 8
13 14 15 12
9 1 7
0 151112
13 14
1 2 3 4
5 6 8
1 2 3 4 9 1 7 1
0 15 121
13 14
5 6 8
9 10 7 11

task 2
1 2 4
13 14 15 12
5 6 3 8
9 1 7 11
0 15 12
13 14
1 2 3 4
5 6 7 8
9 1 1
0 15 121
13 14
1 2 3
4
5
9 6 7
8
13 14 12
10 15
1 2 3 1 211 3 4
4
5 5 6 7 8
9 6 7

task 1
9 10 15
8 11
13 14 1 1
10 15 2 3 14 12
11 1 2 3 4
5 6 7 8
9 1 1
0 15 121
13 14
Speculative Decomposition
• In some applications, dependencies between tasks are not known
a-priori.

• For such applications, it is impossible to identify independent tasks.

• There are generally two approaches to dealing with such

applications: conservative approaches, which identify
independent tasks only when they are guaranteed to not have
dependencies, and, optimistic approaches, which schedule tasks
even when they may potentially be erroneous.

• Conservative approaches may yield little concurrency and

optimistic approaches may require roll-back mechanism in the
case of an error.
 Speculative Decomposition
A classic example of speculative decomposition is in discrete event
simulation.

• The central data structure in a discrete event simulation is a time-ordered

event list.

• Events are extracted precisely in time order, processed, and if required,

resulting events are inserted back into the event list.

• Consider your day today as a discrete event system – you get up, get ready,
drive to work, work, eat lunch, work some more, drive back, eat dinner, and
sleep.

• Each of these events may be processed independently, however, in driving

to work, you might meet with an unfortunate accident and not get to work
at all.
Speculative Decomposition: Example
Another example is the simulation of a network of nodes (for instance,
an assembly line or a computer network through which packets pass).
The task is to simulate the behavior of this network for various inputs
and node delay parameters (note that networks may become unstable
for certain values of service rates, queue sizes, etc.).

A D
System
Inputs

E G I

System
Output
B
F H

System Components
 Characteristics of
Tasks & Interaction
 Characteristics of Tasks
Once a problem has been decomposed into independent tasks, the
characteristics of these tasks critically impact choice and
performance of parallel algorithms. Relevant task characteristics
include:

• Task generation.

• Task sizes.

• Size of data associated with tasks.

Characteristics of Tasks & Interaction
Characteristics of Tasks & Interaction
 Characteristics of Task
Interactions
Tasks may communicate with each other in various ways. The
associated dichotomy is:

• Static interactions: The tasks and their interactions are known a-priori.
These are relatively simpler to code into programs.

• Dynamic interactions: The timing or interacting tasks cannot be

determined a-priori. These interactions are harder to code, especitally,
as we shall see, using message passing APIs.
 Characteristics of Task
Interactions
• Regular interactions: There is a definite pattern (in the graph
sense) to the interactions. These patterns can be exploited for
efficient implementation.

• Irregular interactions: Interactions lack well-defined

topologies.
 Mapping Techniques
• Once a problem has been decomposed into concurrent tasks,
these must be mapped to processes (that can be executed on a
parallel platform).

• Mappings must minimize overheads.

• Primary overheads are communication and idling.

• Minimizing these overheads often represents contradicting

objectives.

• Assigning all work to one processor trivially minimizes

communication at the expense of significant idling.
Mapping Techniques for Minimum Idling
Mapping must simultaneously minimize idling and load
balance. Merely balancing load does not minimize idling.
finish
start synchronization finish start synchronization

P1 1 5 9 P1 1 2 3

P2 2 6 10 P2 4 5 6

P3 3 7 11 P3 7 8 9

P4 4 8 12 P4 10 11 12

t=0 t=2 t=3 t=0 t=3 t=6

(a (b)
)
Mapping Techniques for Minimum Idling
Mapping techniques can be static or dynamic.

• Static Mapping: Tasks are mapped to processes a-priori. For this to

work, we must have a good estimate of the size of each task. Even in
these cases, the problem may be NP complete.

• Dynamic Mapping: Tasks are mapped to processes at runtime. This

may be because the tasks are generated at runtime, or that their sizes
are not known.

Other factors that determine the choice of techniques include the size of
data associated with a task and the nature of underlying domain.
Schemes for Static Mapping
• Mappings based on data partitioning.

• Mappings based on task graph partitioning.

• Hybrid mappings.
Mappings Based on Data Partitioning
We can combine data partitioning with the “owner- computes” rule to
partition the computation into subtasks. The simplest data decomposition
schemes for dense matrices are 1-D block distribution schemes
.
row-wise distribution column-wise distribution
P8 P0
P9 P1
P 10 P2
P 11 P3
P0 P1 P2 P3 P4 P5 P6 P7
P 12 P4
P 13 P5
P 14
P6

P 15
P7
Block Array Distribution Schemes
Block distribution schemes can be generalized to higher
dimensions as well.

P0 P1 P2 P3
P0 P1 P2 P3 P4 P5 P6 P7
P4 P5 P6 P7

P8 P9 P10 P11
P8 P9 P10 P11 P12 P13 P14 P15

P12 P13 P14 P15

(a) (b)
Block Array Distribution Schemes: Examples
• For multiplying two dense matrices A and B, we can partition
the output matrix C using a block decomposition.

• For load balance, we give each task the same number of elements
of C. (Note that each element of C corresponds to a single dot
product.)

• The choice of precise decomposition (1-D or 2-D) is determined

by the associated communication overhead.

• In general, higher dimension decomposition allows the use of

larger number of processes.
Data Sharing in Dense Matrix Multiplication
A B C
P0
P1
P2
P3
P4
P5
P6
P7
X = P8
P9
P 10
P1 2 P 11
P 14 P 13
P 15
(a)

A B C

P0 P1 P2 P3

P4 P5 P6 P7
X =
P8 P9 P10 P 11

P 12 P13 P14 P 15

(b)
Graph Partitioning Dased Data Decomposition
• In case of sparse matrices, block decompositions are more complex.

• Consider the problem of multiplying a sparse matrix with a vector.

• The graph of the matrix is a useful indicator of the work (number of

nodes) and communication (the degree of each node).

• In this case, we would like to partition the graph so as to assign equal

number of nodes to each process, while minimizing edge count of the
graph partition.
Partitioning the Graph of Lake Superior

Random Partitioning

Partitioning for minimum edge-cut.

 Hierarchical Mappings
• Sometimes a single mapping technique is inadequate.

• For example, the task mapping of the binary tree

(quicksort) cannot use a large number of processors.

• For this reason, task mapping can be used at the top

level and data partitioning within each level.
Hierarchical Mapping: Example
An example of task partitioning at top level with data
partitioning at the lower level.
P0 P1 P4 P5
P2 P3 P6 P7

P0 P1 P4 P5
P2 P3 P6 P7

P0 P1 P2 P3 P4 P5 P6 P7

P0 P1 P2 P3 P4 P5 P6 P7
 Schemes for Dynamic
Mapping
• Dynamic mapping is sometimes also referred to as
dynamic load balancing, since load balancing is the
primary motivation for dynamic mapping.

• Dynamic mapping schemes can be centralized or

distributed.
Centralized Dynamic Mapping
• Processes are designated as masters or slaves.

• When a process runs out of work, it requests the master for more work.

• When the number of processes increases, the master may become

the bottleneck.

• To alleviate this, a process may pick up a number of tasks (a chunk) at

one time. This is called Chunk scheduling.

• Selecting large chunk sizes may lead to significant load

imbalances as well.

• A number of schemes have been used to gradually decrease chunk size

as the computation progresses.
Distributed Dynamic Mapping
• Each process can send or receive work from other
processes.

• This alleviates the bottleneck in centralized schemes.

• There are four critical questions: how are sensing and

receiving processes paired together, who initiates work
transfer, how much work is transferred, and when is a
transfer triggered?

• Answers to these questions are generally application

specific. We will look at some of these techniques later in
this class.
Minimizing Interaction Overheads
• Maximize data locality: Where possible, reuse intermediate
data. Restructure computation so that data can be reused in
smaller time windows.

• Minimize volume of data exchange: There is a cost associated

with each word that is communicated. For this reason, we must
minimize the volume of data communicated.

• Minimize frequency of interactions: There is a startup cost

associated with each interaction. Therefore, try to merge
multiple interactions to one, where possible.

• Minimize contention and hot-spots: Use decentralized

techniques, replicate data where necessary.
 Minimizing Interaction
Overheads (continued)
• Overlapping computations with interactions: Use non-
blocking communications, multithreading, and
prefetching to hide latencies.

• Replicating data or computations.

• Using group communications instead of point-to-point

primitives.

• Overlap interactions with other interactions.

 Parallel Algorithm Models
An algorithm model is a way of structuring a parallel algorithm by
selecting a decomposition and mapping technique and applying the
appropriate strategy to minimize interactions.
• Data Parallel Model: Tasks are statically (or semi-statically) mapped
to processes and each task performs similar operations on different
data.
Data Parallel Model
 Task Graph Model
• Task Graph Model: Starting from a task dependency graph,
the interrelationships among the tasks are utilized to promote
locality or to reduce interaction costs.
Task Graph Model
 Master Slave Model
• Master-Slave Model: One or more processes generate work and allocate
it to worker processes. This allocation may be static or dynamic.
Master Slave Model
 Pipeline / Producer-Comsumer Model
• Pipeline / Producer-Comsumer Model: A stream of data is passed
through a succession of processes, each of which perform some task on
it.
 Pipeline / Producer-Comsumer Model
• Pipeline / Producer-Comsumer Model: A stream of data is passed
through a succession of processes, each of which perform some task on
it.
Pipeline / Producer-Comsumer Model
History of GPU
History of GPU
History of GPU
History of GPU
 History of GPU
General-purpose computing on a GPU (Graphics Processing Unit), better
known as GPU programming, is the use of a GPU together with a CPU
(Central Processing Unit) to accelerate computation in applications
traditionally handled only by the CPU. Even though GPU programming has
been practically viable only for the past two decades, its applications now
include virtually every industry.

For example, GPU programming has been used to accelerate video, digital
image, and audio signal processing, statistical physics, scientific
computing, medical imaging, computer vision, neural networks and deep
learning, cryptography, and even intrusion detection, among many other
areas.
CPU vs GPU
 CPU vs GPU
[Link] CPU GPU

CPU stands for Central Processing While GPU stands for Graphics Processing
1.
Unit. Unit.
CPU consumes or needs more While it consumes or requires less memory
2.
memory than GPU. than CPU.
The speed of CPU is less than
3. While GPU is faster than CPU’s speed.
GPU’s speed.
4. CPU contain powerful cores. While it contain more weak cores.

CPU is suitable for serial While GPU is not suitable for serial
5.
instruction processing. instruction processing.

CPU is not suitable for parallel While GPU is suitable for parallel
6.
instruction processing. instruction processing.

7. CPU emphasis on low latency. While GPU emphasis on high throughput.

 THANK YOU !!!!
3
My Blog : [Link] 1

Email : [Link]@[Link]

PROF. ANAND GHARU