Parallel Computing
Parallel computing is a computing model in which multiple processors or cores work simultaneously
on different parts of the same problem. The goal is to reduce execution time by dividing a task into
smaller sub-tasks that run at the same time, usually within a single system with shared memory.
Distributed Computing
Distributed computing is a computing model where multiple independent computers (nodes)
connected by a network work together to solve a problem. Each system has its own memory and
processor, and communication happens through message passing.
Difference between Parallel and Distributed Computing
Parallel computing typically runs on a single machine with shared memory, whereas distributed
computing runs on multiple machines with separate memory. Parallel systems are tightly coupled,
while distributed systems are loosely coupled. Communication in parallel systems is faster
compared to distributed systems.
Data Parallelism
In data parallelism, the same operation is performed simultaneously on different pieces of data.
Example: Applying the same image filter to different parts of an image at the same time.
Task Parallelism
In task parallelism, different tasks or operations are executed in parallel.
Example: In a web server, one thread handles user authentication while another processes
database queries.
Flynn’s Taxonomy
SISD (Single Instruction, Single Data): A traditional single-core computer executing one
instruction stream on one data stream. Example: Basic personal computer.
SIMD (Single Instruction, Multiple Data): A system that performs the same instruction on multiple
data items simultaneously. Example: GPU performing vector operations.
MISD (Multiple Instruction, Single Data): Multiple instructions operating on the same data
stream. Rare in practice. Example: Fault-tolerant systems using redundant computations.
MIMD (Multiple Instruction, Multiple Data): Multiple processors executing different instructions on
different data. Example: Modern multi-core processors or distributed systems.