devices
Introduction to Parallel & Distributed
Systems
Parallel and Distributed Computing
Introduction to Computing
Computing is the process to complete a goal- Key Aspects of Computing
oriented task by using computer technology.
Computing may include the design and development architecture Structuring memory Processing
of software and hardware systems for a broad range
of purposes. storage Managing info Information
Parallel Computing Systems
compare_arrows Serial vs. Parallel
memory Definition
Simultaneous execution of a single task (split up and linear_scale Serial Processing
adapted) on multiple processors to obtain results faster.
Tasks executed sequentially
Single processor at a time
Simple to implement
lightbulb Key Concepts
Slower for complex tasks
speed Simultaneous Execution
content_cut Divide and Conquer
Coordination
view_comfy Parallel Processing
sync
Tasks executed simultaneously
timer Faster Results
Multiple processors working together
Complex coordination required
Faster execution for large problems
Parallel Computing Architecture
memory Parallel Computing Architecture developer_board Multi-core Processors
Computer with more than one processor (few to Single chip with multiple processing units
thousands) available for processing. Processors share the same memory
speed Multiple processors execute tasks simultaneously Lower power consumption
hub Processors may share memory or have distributed Brings parallel computing to desktop computers
memory
sync_alt Communication via shared memory or message passing
dns Multi-processor Systems
Multiple chips with one or more cores each
devices Commercial Examples May have shared or distributed memory
Higher computational power
developer_board Intel Core i7/i9 developer_board AMD Ryzen
Used in servers and high-performance computing
memory GPU Computing cloud Cloud Servers
Distributed Systems
hub Definition cases Real-World Examples
A system in which hardware or software components
located at networked computers communicate and school Online Education Systems
coordinate their actions only by passing messages.
Students and teachers connect from
different locations
Real-time collaboration through
message passing
category Key Characteristics
cloud Cloud Computing
lan Networked send Message Passing
Computers Resources distributed across multiple
data centers
Services coordinated through APIs and
sync Coordination public Geographic messages
Distribution
Why use Parallel Computing?
Key advantages of parallel computing over traditional serial approaches
public schedule expand
Model Complex Phenomena Save Time & Money Solve Larger Problems
arrow_right Better suited for real-world arrow_right Shorter completion time with arrow_right Handle impractical/impossible
simulations more resources serial tasks
arrow_right Handle interrelated events arrow_right Built from commodity arrow_right Process millions of transactions
simultaneously components per second
sync cloud memory
Provide Concurrency Non-Local Resources Better Hardware Use
arrow_right Single resource: one thing at a arrow_right Utilize wide area networks arrow_right Modern computers have multiple
time cores
arrow_right Leverage Internet resources
arrow_right Multiple resources: many things when needed arrow_right Serial programs waste computing
simultaneously power
Why use Distributed Computing?
Key advantages and motivations for distributed systems
history settings savings
Historical Functional Economical
arrow_right Independent resources now need arrow_right Access special-function arrow_right Cost-effective small computers
to work together hardware/software vs. large one
arrow_right Evolution from isolated arrow_right No need to duplicate resources on arrow_right Flexible configuration - add
workstations to networks every node resources as needed
verified schedule build
Reliability Availability Maintenance
arrow_right Replication of data and arrow_right Access resources 24/7 across the arrow_right No system downtime for
functionality system upgrades
arrow_right System continues if one arrow_right Alternate resources when one is arrow_right Components can be serviced
component fails unavailable individually
Challenges in Distributed Computing
Distributed computing brings advantages but also introduces new problems that must be addressed
sync_problem schedule
Data Consistency Clock Synchronization
Keeping multiple copies of data consistent across different Keeping the clocks on different machines in the system
nodes in the system. synchronized.
error_outline Replication creates synchronization issues error_outline Physical clocks drift at different rates
error_outline Concurrent updates can lead to conflicts error_outline Network latency affects time synchronization
error_outline Requires complex consensus algorithms error_outline Critical for ordering events correctly
warning Other Potential Problems
security Security & Authentication bug_report Fault Tolerance speed Performance
scatter_plot Scalability hub Network Partitioning settings_ethernet Heterogeneity
Comparison: Parallel vs Distributed Computing (Part 1)
Key differences between parallel and distributed computing approaches
memory Parallel Computing hub Distributed Computing
1 Operations 1 System Components
Many operations performed simultaneously Located at different locations
speed location_on
2 Computer Requirements 2 Computer Requirements
Single computer is required Uses multiple computers
computer devices
3 Processing 3 Processing
Multiple processors perform multiple operations Multiple computers perform multiple operations
developer_board dns
Comparison: Parallel vs Distributed Computing (Part 2)
Key differences between parallel and distributed computing approaches
memory Parallel Computing hub Distributed Computing
4 Memory Type 4 Memory Type
May have shared or distributed memory Has only distributed memory
storage cloud
5 Communication
5 Communication
Computers communicate through message passing
Processors communicate through bus
settings_input_component send
6 System Improvement
6 System Improvement
Improves scalability, fault tolerance and resource
Improves system performance sharing
speed trending_up