Parallel Processing
Definition - What does Parallel Processing mean?
Parallel Processing: is the simultaneous use of more than one CPU (processor) to execute a
program. Parallel processing makes a program run faster because there are more CPUs running it
Also, parallel processing is a method of simultaneously breaking up and running program tasks
on multiple microprocessors, thereby reducing processing time. Parallel processing may be
accomplished via a computer with two or more processors or via a computer network.
Parallel processing is also called parallel computing. Supercomputers commonly have hundreds
of thousands of microprocessors for this purpose. Parallel processing should not be confused
with concurrency, which refers to multiple-tasks that run simultaneously.
Instruction-level parallelism (ILP): is a measure of how many of the operations in a computer
program can be performed simultaneously. The potential overlap among instructions is called
instruction level parallelism.
Micro-architectural techniques that are used to exploit ILP (Instruction Level Parallelism)
include:
i. Instruction pipelining is a technique that implements a form of parallelism called
instruction-level parallelism within a single processor. It therefore allows faster CPU
throughput (the number of instructions that can be executed in a unit of time).
Instruction pipelining where the execution of multiple instructions can be partially
overlapped.
ii. Superscalar describes a microprocessor design that makes it possible for more than
one instruction at a time to be executed during a single clock cycle . In a superscalar
design, the processor or the instruction compiler is able to determine whether an
instruction can be carried out independently of other sequential instructions, or
whether it has a dependency on another instruction and must be executed in sequence
1
with it. The processor then uses multiple execution units to simultaneously carry out
two or more independent instructions at a time.
iii. Out-of-order execution where instructions execute in any order that does not violate
data dependencies. In this paradigm, a processor executes instructions in an order
governed by the availability of input data, rather than by their original order in a
program. In doing so, the processor can avoid being idle while waiting for the
preceding instruction to be completed. The processor therefore, retrieve data for the
next instruction in a program and processing the next instructions which are able to
run immediately and independently
iv. Speculative execution is an optimization technique where a computer system
performs some task that may not be actually needed. The main idea is to do work
before it is known whether that work will be needed at all, so as to prevent a delay
that would have to be incurred by doing the work after it is known whether it is
needed. If it turns out the work was not needed after all, any changes made by the
work are reverted and the results are ignored.
Flynn’s Taxonomy of Computer Architecture
In addition to increasing parallelism by finding instructions which can execute at the same time,
processor architects have introduced special instructions.
The most popular taxonomy of computer architecture was defined by Flynn in 1966. Flynn’s
classification scheme was based on the notion of a steam of information. There two types of
information that flows into a processor; instruction and data. Instruction stream is defined as the
sequence of instruction performed by the processing unit.
The data stream is defined as the data traffic exchanged between the memory and the processing
unit. According to Flynn’s classification, either of the instruction or data streams can be single or
multiple. This leads to four distinct categories of computer architectures:
1. Single-instruction single-data streams (SISD)
2. Single-instruction multiple-data streams (SIMD)
2
3. Multiple-instruction single-data streams (MISD)
4. Multiple-instruction multiple-data streams (MIMD)
Flynn’s classifications
3
Figure a-d: The four architecture classes resulting from Flynn’s taxonomy. (a) SISD (b)
SIMD, (c) MISD, (d) MIMD