INTRODUCTION TO PARALLEL COMPUTING
Learning Objectives
After studying this unit, students will be able to:
Understand the concept of parallel computing.
Explain the need for parallel processing.
Differentiate between sequential and parallel computing.
Describe Flynn's classification of computer architectures.
Analyze various parallel computer models.
Calculate performance metrics such as speedup and efficiency.
Understand Amdahl's Law and scalability.
Identify real-world applications of parallel computing.
1. Introduction to Parallel Computing
Parallel computing is a computational technique in which multiple processors execute several tasks
simultaneously to solve a problem faster.
Traditional computers execute instructions sequentially, one after another. As computational requirements
increased, parallel computing emerged as an effective solution to improve performance.
Definition
Parallel Computing is the simultaneous execution of multiple computations using multiple processing
elements to solve a problem efficiently.
Example
Consider adding the numbers from 1 to 1,000,000.
Sequential Approach
One processor performs all additions.
Execution time is high.
Parallel Approach
Divide numbers into groups.
Multiple processors compute partial sums simultaneously.
Results are combined.
Execution time is significantly reduced.
2. Evolution of Parallel Computing
First Generation (1940–1950)
Vacuum tube computers.
Sequential processing.
Examples:
IBM early systems.
Second Generation (1950–1960)
Transistor-based computers.
Improved performance.
Third Generation (1960–1970)
Integrated circuits.
Multiprogramming concepts introduced.
Fourth Generation (1970–1990)
Microprocessors.
Multiprocessor systems emerged.
Modern Era
Multi-core processors.
GPUs.
Cloud computing.
High-performance computing clusters.
3. Need for Parallel Computing
Modern applications require enormous computational power.
Reasons
1. Faster Execution
Large problems can be solved quickly.
2. Handling Big Data
Applications generate huge amounts of data.
3. Scientific Simulations
Examples:
Weather forecasting
Climate modeling
Space research
4. Artificial Intelligence
Machine learning models require parallel processing.
5. Cost Efficiency
Using multiple low-cost processors may be cheaper than one supercomputer.
4. Sequential Processing vs Parallel Processing
Feature Sequential Parallel
Execution One task at a time Multiple tasks simultaneously
Speed Slow Fast
Processor Count One Multiple
Scalability Limited High
Cost Lower Higher
Sequential Processing Diagram
Task A → Task B → Task C → Task D
Parallel Processing Diagram
Processor 1 → Task A
Processor 2 → Task B
Processor 3 → Task C
Processor 4 → Task D
5. Flynn's Classification of Computer Architectures
In 1966, Michael J. Flynn proposed a classification based on instruction streams and data streams.
Categories
SISD (Single Instruction Single Data)
Single Processor
|
Instruction
|
Single Data
Characteristics:
Traditional computers.
Sequential execution.
Examples:
Single-core systems.
SIMD (Single Instruction Multiple Data)
Instruction
|
--------------------------------
| | | |
D1 D2 D3 D4
Characteristics:
Same instruction on multiple data items.
Vector processing.
Examples:
Graphics processing.
MISD (Multiple Instruction Single Data)
Instruction 1
Instruction 2
Instruction 3
|
Same Data
Rarely used.
Applications:
Fault-tolerant systems.
MIMD (Multiple Instruction Multiple Data)
Processor 1 → Data 1
Processor 2 → Data 2
Processor 3 → Data 3
Characteristics:
Most modern parallel computers.
Independent execution.
Examples:
Multi-core processors.
Distributed systems.
6. Parallel Computer Models
Shared Memory Model
All processors access common memory.
CPU1
CPU2 ---- Shared Memory
CPU3
Advantages:
Easy communication.
Disadvantages:
Memory contention.
Distributed Memory Model
Each processor has private memory.
CPU1 ↔ CPU2 ↔ CPU3
Advantages:
Highly scalable.
Disadvantages:
Complex communication.
Hybrid Model
Combines shared and distributed memory systems.
Used in:
Supercomputers.
Large data centers.
7. Performance Metrics
Speedup
Measures performance improvement.
Formula:
Speedup=\frac{T_s}{T_p}
Where:
Ts = Sequential execution time
Tp = Parallel execution time
Example
Sequential time = 100 seconds
Parallel time = 20 seconds
Speedup = 100 / 20 = 5
Efficiency
Measures processor utilization.
Efficiency=\frac{Speedup}{Number\ of\ Processors}
Example:
Speedup = 8
Processors = 10
Efficiency = 0.8 = 80%
Throughput
Number of tasks completed per unit time.
Scalability
Ability of a system to maintain performance as processors increase.
8. Amdahl's Law
Amdahl's Law predicts the maximum improvement achievable through parallelization.
Speedup=\frac{1}{(1-P)+\frac{P}{N}}
Where:
P = Parallelizable portion
N = Number of processors
Solved Problem
A program has:
80% parallelizable code
4 processors
Calculate speedup.
Solution:
P = 0.8
N=4
Speedup
= 1 / [(1 − 0.8) + (0.8 / 4)]
= 1 / (0.2 + 0.2)
= 2.5
Maximum speedup = 2.5
9. Applications of Parallel Computing
Scientific Research
Nuclear simulations
Astronomy
Weather Forecasting
Complex atmospheric calculations.
Artificial Intelligence
Deep learning training.
Medical Imaging
MRI and CT scan processing.
Financial Modeling
Risk analysis and stock prediction.
Gaming
Real-time rendering.
Advantages of Parallel Computing
1. Faster execution.
2. Better resource utilization.
3. Handles large datasets.
4. Increased reliability.
5. Supports real-time applications.
Limitations
1. Complex programming.
2. Synchronization overhead.
3. Communication cost.
4. Load balancing issues.
5. Increased hardware cost.
Unit Summary
Parallel computing performs multiple computations simultaneously.
Flynn classified architectures into SISD, SIMD, MISD, and MIMD.
Shared and distributed memory are major parallel models.
Performance is measured using speedup, efficiency, throughput, and scalability.
Amdahl's Law predicts theoretical speedup limits.
Parallel computing powers AI, cloud computing, simulations, and big data applications.