Vth SEM Cloud Computing
8 UNIT 2
Parallel and Distributed Systems:
1. Parallel and Distributed Systems Introduction,
2. Parallel Computing,
3. Architecture,
4. Distributed Systems,
5. Communication Protocol and Process Coordination,
6. logical Clocks,
7. Message Delivery Rules,
8. Concurrency,
9. Atomic Actions,
10. Consensus Protocols,
11. Modeling Concurrency with Petri Nets,
12. Client-Server Paradigm
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Parallel Systems
Definition:
A computing architecture where multiple processors (CPUs, cores, or GPUs) work simultaneously to execute different
parts of the same problem or task.
It Break the problem into smaller subtasks that can be processed in parallel to reduce the total execution time.
Memory Model: Usually shared memory (all processors access the same global memory space), but can also use
distributed memory in some hybrid models.
Communication:
o Very fast, through system buses or interconnects (e.g., InfiniBand, high-speed crossbar switches).
o Lower latency than distributed systems.
Real-World Examples:
o Weather Simulation → supercomputers like Cray XC40
o 3D Rendering → parallel GPU rendering
o Genomic Data Analysis → parallel algorithms to map DNA sequences.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
2. Importance in Cloud Computing
AI Training: Massive datasets processed in parallel (e.g., GPUs in parallel clusters).
Big Data Analytics: Hadoop and Spark use distributed computing to process terabytes of data.
Content Delivery: Netflix uses distributed servers worldwide to reduce latency.
Scientific Simulations: Weather prediction models run on parallel supercomputers.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Example – Weather Simulation using Supercomputers (Cray XC40)
Why Parallel?
Weather prediction models require processing terabytes of atmospheric data (temperature, humidity, wind speed,
satellite imagery) in real time.
A single processor would take days to compute forecasts, which would make them useless for immediate decision-
making.
How it Works in Parallel Systems:
o The entire geographical area is divided into smaller grids.
o Each processor handles the simulation for its assigned grid.
o Processors share data at boundaries to maintain accuracy (shared memory communication).
o All computations happen simultaneously, drastically reducing runtime.
Real System:
o Cray XC40 (used by the Indian Meteorological Department and NASA) can deliver petaflop-level performance.
o Used for monsoon prediction, cyclone tracking, and climate change studies.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Example – Genomic Data Analysis (Parallel Systems)
Why Parallel?
Modern DNA sequencing generates massive datasets — mapping the human genome involves analyzing over 3
billion base pairs.
If done sequentially, mapping could take weeks or months.
Parallel computing reduces this to hours or days.
How it Works in Parallel Systems:
1. Data Partitioning
The genome dataset is split into smaller chunks of sequences.
2. Parallel Processing
Each processor runs sequence alignment algorithms (e.g., BLAST, Bowtie) on its assigned chunk
simultaneously.
3. Intermediate Results
Processors share partial results through shared memory or a high-speed interconnect.
4. Combination
Results are merged to form the complete genome map.
Real Systems Used:
o Illumina BaseSpace and Amazon EC2 HPC instances for cloud-based parallel genome analysis.
o National Institutes of Health (NIH) supercomputers for large-scale DNA mapping projects.
Applications:
o Identifying genetic markers for diseases.
o Personalizing medical treatments.
o Tracing evolutionary biology and ancestry.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Example – IBM Blue Gene (Parallel Systems)
Why Parallel?
IBM Blue Gene was designed for massively parallel processing to tackle problems requiring enormous computing
power — such as molecular dynamics, quantum chemistry, and nuclear simulations.
How it Works in Parallel Systems:
1. Massive Processor Count
Blue Gene/P had up to 294,912 processor cores.
Blue Gene/Q increased this to 1.6 million cores.
2. Tightly Coupled Architecture
Processors were interconnected via a high-speed 5D torus network, enabling fast communication
between cores.
3. Shared Data Processing
Tasks were split into smaller sub-tasks and processed in parallel, with results exchanged at high speed.
4. Optimized for Scientific Workloads
Specialized for floating-point heavy calculations and simulations.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Performance:
o Blue Gene/Q achieved 20 petaflops peak performance.
o Ranked #1 on the TOP500 list of supercomputers in multiple years.
Applications:
o Protein folding research for medical breakthroughs.
o Climate and weather modeling.
o Astrophysics simulations.
o Nuclear weapons safety testing (without actual detonations).
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
2.
Distributed Systems
Definition:
A network of independent computers (nodes) that collaborate to achieve a common goal, appearing to the user as a
single system.
Key Idea: Each node has its own memory and processor, and communicates through network protocols.
Memory Model: Distributed memory (each node stores data locally).
Communication:
o Via message passing over a network (TCP/IP, RPC, gRPC).
o Higher latency compared to parallel systems due to network delays.
Real-World Examples:
o Google Search Infrastructure → millions of servers processing queries
o Netflix Content Delivery Network → distributed servers worldwide
o Hadoop/Spark Clusters → distributed data processing.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Example – Google Search Infrastructure (Distributed Systems)
Why Distributed?
Google processes billions of search queries per day from users all over the world.
A single computer — or even a single data center — could never handle that scale due to:
o Massive data storage needs (hundreds of petabytes)
o High availability requirements (24/7 uptime)
o Low latency expectations (fractions of a second per search)
How it Works in Distributed Systems:
1. Geographically Distributed Data Centers
Google has servers spread across multiple continents.
Each data center stores a portion of the web index.
2. Load Balancing
Incoming queries are routed to the nearest available server to reduce latency.
3. Replication & Fault Tolerance
Data is replicated across locations so if one server fails, others can take over.
4. MapReduce for Processing
Large search and indexing tasks are split into smaller chunks and run across thousands of machines.
Real System:
o Runs on tens of thousands of commodity servers, not specialized supercomputers.
o Uses Google File System (GFS) and Bigtable to store and manage the search index.
o Communication is over high-speed internet backbone — not internal shared memory.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Example – Netflix Content Delivery Network (Distributed Systems)
Why Distributed?
Netflix streams millions of hours of video every day to users across the globe.
If all users tried to stream from a single central server, it would:
o Cause network congestion
o Increase buffering and latency
o Overload the server and cause downtime
How it Works in Distributed Systems:
1. Global Server Network
Netflix uses thousands of distributed servers located in data centers worldwide.
Many are placed in Internet Service Providers’ (ISP) networks directly to reduce distance to the user.
2. Edge Servers
Popular shows and movies are cached in regional servers close to viewers (edge computing).
When you press “Play,” your video comes from the nearest server, not from across the world.
3. Load Balancing
Requests are routed dynamically to the server with least load and lowest latency.
4. Fault Tolerance
If one server fails, the system reroutes requests to another nearby server.
Real System:
o Netflix’s CDN is called Open Connect.
o Uses distributed data replication to store copies of video files across locations.
o Relies on internet backbone connections and optimized routing to minimize buffering.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Differences Between Parallel & Distributed Systems
Feature Parallel Systems Distributed Systems
Coupling Tightly coupled (processors in same physical Loosely coupled (nodes spread across
location) locations)
Memory Shared (or hybrid) memory Distributed memory
Communication High-speed internal interconnects Network protocols (LAN/WAN, Internet)
Scalability Limited by hardware constraints Highly scalable by adding more nodes
Main Goal Speed up computations Resource sharing, fault tolerance,
scalability
Latency Low (nanoseconds to microseconds) Higher (milliseconds)
Examples Cray supercomputers, GPU farms AWS cloud, Google Datacenter, Hadoop
clusters
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT
Vth SEM Cloud Computing
Importance in Modern Computing
Parallel and distributed systems form the backbone of high-performance and large-scale computing:
1. Artificial Intelligence (AI) & Machine Learning
o Training large deep learning models (e.g., GPT, BERT) using GPU/TPU clusters.
o Parallel matrix multiplication in neural networks.
2. Big Data Analytics
o Distributed frameworks like Hadoop and Spark process petabytes of data efficiently.
3. Scientific Simulations
o Climate models, space research, earthquake prediction.
4. Real-Time Applications
o Multiplayer online gaming, stock market trading systems.
5. Cloud Computing
o Services like AWS, Azure, GCP use distributed architectures for global accessibility and fault tolerance.
6. High-Performance Graphics
o Parallel GPU rendering for animation, movies, and VR.
DRONACHARYA COLLEGE OF ENGINEEERING CSE DEPARTMENT