Introduction to Parallel and
Distributed Computing
Parallel and distributed computing are essential concepts
in modern computing systems.
These approaches enhance performance and efficiency by
dividing tasks and distributing them across multiple
processors or computers.
by Francis Saa-Dittoh
What is Parallel
Computing?
Parallel Computing
The practice of dividing a program or computational
task into smaller parts, each of which can be
executed simultaneously on different processors or
cores within a single system.
Example: Image Processing
Image processing on GPUs where each pixel can be
processed independently, enabling faster processing
times by leveraging multiple cores in parallel.
What is Distributed Computing?
Distributed Computing
The use of a network of separate computers (nodes) that work together to complete a task. Each
computer in a distributed system performs a portion of the computation and communicates with
others over a network.
Example: Cloud Storage Services
Cloud storage services like Google Drive and Dropbox, where files are stored and processed
across multiple servers.
Goals of Parallel and Distributed Computing
Increased Scalability Fault Tolerance and Resource Sharing
Performance Reliability
Ensure the system or Use shared
Achieve faster application can Design systems that resources effectively
computational results handle growth, can recover from to improve cost-
by executing multiple whether in terms of failures (e.g., efficiency and
processes or threads data volume, hardware, network) to utilization rates
simultaneously. computational power, ensure consistent (common in
or user count. availability. distributed
computing).
Importance of Parallel and
Distributed Computing
Big Data and Machine High-Performance
Learning Applications
Processing massive Applications in weather
datasets and training forecasting, financial
machine learning modeling, scientific
models require high- research, and
performance simulations require
computing resources. tremendous computing
power.
Real-Time Systems
Systems such as autonomous vehicles and financial
trading algorithms rely on fast, parallel processing for
immediate data analysis and decision-making.
Differences Between Parallel and Distributed
Systems
Feature Parallel Computing Distributed Computing
System Type Single system (multi-core Network of separate systems
CPU, GPU)
Communication Shared memory (typically) Network-based
communication
Data Access Shared memory or cache Each node has its own
memory
Common Uses Fast computations within a Large-scale data processing
single device
Example Multi-threaded applications Cloud computing platforms
Blurring Lines
Many modern systems are a hybrid of parallel and
distributed architectures
A cloud data center may use distributed computing
across servers while each server uses parallel
processing (multi-core CPUs and GPUs) for computation
within itself.
Supercomputers like IBM’s Summit are a combination of
parallel processing units (e.g., GPUs) distributed across
thousands of nodes.
Real-World Examples
Parallel Computing Distributed Computing
An animation studio rendering CGI graphics An e-commerce platform where different servers
using a cluster of GPUs where each frame or handle the database, front-end, and
pixel can be processed independently. recommendation engine, distributing tasks for
efficiency.
Applications in Scientific
Research
Genomics Climate Modeling
1 2
Parallel and Climate models
distributed simulate complex
computing are weather patterns
essential for and predict future
analyzing massive climate change.
datasets in
genomics research.
Physics Simulations
3
Parallel computing enables scientists to simulate
complex physical phenomena, such as particle
collisions.
Applications in Financial
Modeling
High-Frequency Fraud Detection
Trading
Distributed systems
Parallel computing can process large
enables real-time datasets to identify
analysis of financial patterns and anomalies
markets for fast trading indicative of fraud.
decisions.
Risk Analysis
Parallel processing allows for complex simulations
and calculations to assess financial risks.
Applications in Image and Signal Processing
Satellite Imaging Face Recognition Video Processing
Parallel and distributed Social media platforms use Parallel computing enables real-
computing is used to process parallel processing to analyze time analysis and processing of
vast amounts of data from images and identify faces. video streams.
satellites.
Applications in Data
Centers and Cloud
Computing
Distributed Systems
Distributed systems are foundational for cloud services.
Global Access
Enable efficient storage and access to user data
worldwide.
Data Redundancy
Distribute user data across many servers globally.
Applications in AI and Machine Learning
Neural Network Training Large-Scale Language Models
Training complex neural networks requires Training large-scale language models that involve
intensive computation, which is sped up through billions of parameters typically use parallel
parallel processing on GPUs. processing to reduce training time.
Key Concepts in Parallel and Distributed
Computing
Speedup and Efficiency
Speedup is the ratio of time taken to complete a task on a single processor to the time taken on
multiple processors.
Efficiency measures the use of computing resources effectively. It’s calculated as the speedup divided
by the number of processors.
Where:
Ts(n) is the time it takes to complete the task on a single processor (serial execution time).
Tp(n,p) is the time it takes to complete the task on p processors (parallel execution time).
Amdahl’s Law
Amdahl’s Law is a theoretical principle that defines the
potential speedup of a program based on the portion that
can be parallelized.
The formula for Amdahl’s Law is: Speedup = 1 / (1 - p +
p/n), where p is the parallelizable portion, and n is the
number of processors.
Amdahl’s Law implies that no matter how many processors
are added, the non-parallelizable part limits the maximum
speedup.