AI Questions Parallel Computing

Part 1: Multiple Choice Questions (MCQs)

Q1. Traditionally, software has been written for which type of computation?
A) Parallel computation
B) Serial computation
C) Quantum computation
D) Distributed computation

Correct Answer: B

Q2. In a serial computation, how are instructions executed?

A) Simultaneously
B) Out of order
C) One after another
D) In batches of four

Correct Answer: C

Q3. What is the definition of parallel computing in its simplest sense?

A) The use of a single CPU to switch rapidly between tasks
B) The simultaneous use of multiple compute resources to solve a computational problem
C) The sequential execution of broken-down program parts
D) The use of specialized GPUs for graphics rendering only

Correct Answer: B

Q4. Which of the following is NOT listed as a potential compute resource for parallel
computing?
A) A single computer with multiple processors
B) An arbitrary number of computers connected by a network
C) A standalone mechanical calculator
D) A single computer with specialized resources like a GPU or FPGA

Correct Answer: C
Q5. A computational problem suitable for parallel computing must have which
characteristic?
A) It can be solved in less time with a single resource than multiple resources
B) It cannot be broken into discrete pieces of work
C) It requires instructions to be executed sequentially only
D) It can execute multiple program instructions at any moment in time

Correct Answer: D

Q6. Parallel computing attempts to emulate what aspect of the natural world?
A) Linear progression of singular events
B) Static, unchanging environments
C) Many complex, interrelated events happening at the same time
D) The serial processing of information by a single brain

Correct Answer: C

Q7. Which of the following is an example of a "real world" parallel event mentioned in the
lectures?
A) Reading a book line by line
B) Automobile assembly line
C) Writing a letter
D) Running a single-threaded application

Correct Answer: B

Q8. Which commercial application is listed as a driving force for faster computers?
A) Simple text editing
B) Basic calculator apps
C) Oil exploration
D) Solitaire

Correct Answer: C
Q9. Ultimately, parallel computing is an attempt to maximize which scarce commodity?
A) Time
B) Money
C) Silicon
D) Electricity

Correct Answer: A

Q10. What is a primary reason for using parallel computing besides saving time?
A) To increase the complexity of the code
B) To provide concurrency (do multiple things at the same time)
C) To reduce the number of necessary programmers
D) To slow down the clock cycle of the CPU

Correct Answer: B

Q11. How can parallel computing help overcome memory constraints?

A) By compressing data into smaller packets
B) By using the memories of multiple computers for large problems
C) By increasing the size of the hard drive
D) By slowing down the processor to read data properly

Correct Answer: B

Q12. What is the absolute limit for data transmission speed through hardware?
A) The speed of sound
B) 9 cm/nanosecond
C) The speed of light (30 cm/nanosecond)
D) 100 meters/second

Correct Answer: C

Q13. What is the transmission limit of copper wire?

A) 30 cm/nanosecond
B) 9 cm/nanosecond
C) 50 cm/nanosecond
D) 1 meter/nanosecond
Correct Answer: B
Q14. Why is building ever-faster serial computers limited by miniaturization?
A) Processors are becoming too large to fit in cases
B) There is a limit on how small components can be, even at the atomic level
C) Smaller components consume too much power
D) Software cannot run on small chips

Correct Answer: B

Q15. Why is using multiple moderately fast commodity processors often preferred over a
single super-fast processor?
A) It is less expensive (economic limitations)
B) Commodity processors are easier to find
C) Super-fast processors do not exist
D) It saves electricity

Correct Answer: A

Q16. According to the "Future" slide, what clearly shows that parallelism is the future of
computing?
A) The return to mainframe architectures
B) The slowing down of network speeds
C) Trends in multi-processor architectures and distributed systems
D) The decreasing popularity of desktop PCs

Correct Answer: C

Q17. The future of parallel computing will likely be:

A) Exclusively based on general-purpose PCs
B) Multi-forms, mixing general purpose and specialized solutions
C) Restricted to specialized hardware like IBM Cells
D) Abandoned for serial computing

Correct Answer: B
Q18. Which organization provides statistics on parallel computing users?
A) [Link]
B) [Link]
C) [Link]
D) [Link]

Correct Answer: B

Q19. Based on the Top500 statistics sample, which application sector is the largest?
A) Finance
B) Aerospace
C) Not Specified
D) Gaming

Correct Answer: C

Q20. The Von Neumann architecture is based on which concept?

A) The stored-program concept
B) The read-only memory concept
C) The parallel-execution concept
D) The distributed-network concept

Correct Answer: A

Q21. In a Von Neumann computer, the CPU performs operations on memory in what
manner?
A) Simultaneously
B) Sequentially
C) Randomly
D) Asynchronously

Correct Answer: B
Q22. In the basic design of a Von Neumann computer, what is memory used for?
A) To store data only
B) To store program instructions only
C) To store both program instructions and data
D) To cool the CPU

Correct Answer: C

Q23. Flynn's Taxonomy classifies parallel computers along which two independent
dimensions?
A) Memory and Network
B) Speed and Cost
C) Instruction and Data
D) Hardware and Software

Correct Answer: C

Q24. In Flynn's Taxonomy, the dimension of "Data" can have which two possible states?
A) Fast or Slow
B) Single or Multiple
C) Shared or Distributed
D) Local or Global

Correct Answer: B

Q25. Which acronym stands for "Single Instruction, Single Data"?

A) SIMD
B) MIMD
C) SISD
D) MISD
Correct Answer: C

Q26. Which type of computer is considered a serial (non-parallel) computer?

A) SISD
B) SIMD
C) MISD
D) MIMD
Correct Answer: A
Q27. Which of the following is an example of an SISD computer?
A) Cray C90
B) Most PCs and single CPU workstations
C) Connection Machine CM-2
D) IBM SP2

Correct Answer: B

Q28. In SIMD architecture, what happens during any given clock cycle?
A) Different processors execute different instructions
B) All processing units execute the same instruction
C) Data is transferred between all units
D) One unit executes while others wait

Correct Answer: B

Q29. SIMD is best suited for which type of problem?

A) Problems with high regularity, such as image processing
B) Problems with complex branching and logic
C) Problems requiring frequent user interaction
D) Database management problems

Correct Answer: A

Q30. Which of the following is a variety of SIMD architecture?

A) Symmetric Multiprocessor
B) Vector Pipelines
C) Beowulf Cluster
D) SMP

Correct Answer: B

Q31. Which category of Flynn's taxonomy has few actual examples?

A) SISD
B) SIMD
C) MIMD
D) MISD
Correct Answer: D
Q32. In MISD, what operates on a single data stream?
A) A single processing unit
B) Multiple processing units using independent instruction streams
C) Multiple processing units using the same instruction stream
D) A vector pipeline

Correct Answer: B

Q33. Which is currently the most common type of parallel computer?

A) SISD
B) SIMD
C) MISD
D) MIMD

Correct Answer: D

Q34. In MIMD architecture, execution can be:

A) Only synchronous
B) Only deterministic
C) Synchronous or asynchronous
D) Only serial

Correct Answer: C

Q35. Networked parallel computer "grids" are examples of which architecture?

A) SISD
B) SIMD
C) MISD
D) MIMD

Correct Answer: D

Q36. What is a "Task" in parallel terminology?

A) The physical processor performing work
B) A logically discrete section of computational work
C) The network cable connecting computers
D) The operating system scheduler
Correct Answer: B
Q37. What defines a "Parallel Task"?
A) A task that must be run on a GPU
B) A task that yields correct results only when run serially
C) A task that can be executed by multiple processors safely
D) A task that controls the memory bus

Correct Answer: C

Q38. What is "Serial Execution"?

A) Execution of a program sequentially, one statement at a time
B) Execution of a program where all statements run at once
C) Execution of a program across a network
D) Execution of a program without any data

Correct Answer: A

Q39. What distinguishes "Parallel Execution" from serial execution?

A) It uses a single processor
B) It executes a program by more than one task simultaneously
C) It runs faster but with more errors
D) It requires no memory

Correct Answer: B

Q40. From a hardware view, what characterizes "Shared Memory"?

A) All processors have direct access to common physical memory
B) Each processor has only local memory
C) Memory is stored on a hard drive
D) Memory is accessed via the Internet only

Correct Answer: A

Q41. In "Distributed Memory," how do tasks access memory on other machines?

A) Direct addressing
B) Common physical bus
C) Communications
D) They cannot access it at all
Correct Answer: C
Q42. What is "Synchronization" in parallel computing?
A) The coordination of parallel tasks in real time
B) The speed at which data travels
C) The clock speed of the CPU
D) The amount of memory used by a task

Correct Answer: A

Q43. Synchronization typically involves what action by at least one task?

A) Writing to disk
B) Waiting
C) Calculating
D) Terminating

Correct Answer: B

Q44. What is "Granularity" in parallel computing?

A) The size of the physical RAM
B) The number of processors in the system
C) The ratio of computation to communication
D) The speed of the network connection
Correct Answer: C

Q45. "Coarse" granularity means:

A) Relatively small amounts of work are done between communication events
B) Relatively large amounts of work are done between communication events
C) No work is done between communication events
D) The communication overhead is higher than computation

Correct Answer: B

Q46. How is "Observed Speedup" defined?

A) Wall-clock time of parallel execution / Wall-clock time of serial execution
B) Wall-clock time of serial execution / Wall-clock time of parallel execution
C) Number of processors / Time taken
D) Serial execution time + Parallel execution time
Correct Answer: B
Q47. Which of the following is a factor in "Parallel Overhead"?
A) Task start-up time
B) CPU clock speed
C) Hard drive capacity
D) Monitor refresh rate

Correct Answer: A

Q48. "Massively Parallel" refers to systems having:

A) One very large processor
B) Many processors (pushing 6 digits)
C) A massive amount of storage
D) A large physical footprint

Correct Answer: B

Q49. "Scalability" refers to a parallel system's ability to:

A) Fit into a smaller room
B) Run on different operating systems
C) Demonstrate a proportionate increase in speedup with the addition of more processors
D) Cost less as it gets older

Correct Answer: C

Q50. Which factor contributes to scalability?

A) Monitor resolution
B) Application algorithm
C) Keyboard layout
D) Mouse sensitivity

Correct Answer: B

Q51. What is a key characteristic of Shared Memory machines?

A) Memory is physically distributed across a network
B) Multiple processors share the same memory resources
C) Changes by one processor are invisible to others
D) There is no concept of global address space
Correct Answer: B
Q52. What does UMA stand for?
A) Unified Memory Access
B) Uniform Memory Access
C) Unique Memory Access
D) Universal Memory Access

Correct Answer: B

Q53. In a UMA system, processors are:

A) Identical
B) Different speeds
C) Located in different buildings
D) Assigned different memory privileges

Correct Answer: A

Q54. What does "Cache Coherent" mean in CC-UMA?

A) If one processor updates shared memory, all others know about the update
B) The cache is cleared every second
C) The cache is larger than the main memory
D) Processors do not use cache

Correct Answer: A

Q55. NUMA systems are often made by physically linking:

A) Two or more single-core PCs
B) Two or more SMPs
C) A CPU and a GPU
D) A hard drive and a tape drive

Correct Answer: B

Q56. In NUMA, memory access across the link is:

A) Faster than local access
B) The same speed as local access
C) Slower than local access
D) Instantaneous
Correct Answer: C
Q57. A primary disadvantage of Shared Memory is:
A) Lack of scalability between memory and CPUs
B) Difficulty in sharing data between tasks
C) Non-global address space
D) Slow data sharing due to network latency

Correct Answer: A

Q58. Distributed memory systems require what to connect inter-processor memory?

A) A shared bus
B) A communication network
C) A global address space
D) A single large hard drive

Correct Answer: B

Q59. In Distributed Memory systems, does the concept of cache coherency apply?
A) Yes, it is critical
B) Only for the master node
C) No, because changes to local memory have no effect on other processors
D) Yes, but only for read-only data

Correct Answer: C

Q60. Who is usually responsible for defining how data is communicated in Distributed
Memory?
A) The hardware
B) The operating system
C) The programmer
D) The network administrator
Correct Answer: C

Q61. A major advantage of Distributed Memory is:

A) Global address space
B) Memory is scalable with the number of processors
C) Automatic cache coherency
D) User-friendly programming perspective
Correct Answer: B
Q62. What is a disadvantage of Distributed Memory?
A) It is expensive to use commodity processors
B) Non-uniform memory access (NUMA) times
C) Programmer is responsible for data communication details
D) Access to local memory is slow

Correct Answer: C

Q63. Hybrid Distributed-Shared Memory systems usually employ:

A) A network of single-processor computers
B) A single massive shared memory machine
C) A network of SMPs
D) Only distributed memory components

Correct Answer: C

Q64. In a Hybrid architecture, the shared memory component is usually:

A) A cache coherent SMP machine
B) A distributed cluster
C) A single-core workstation
D) A vector pipeline

Correct Answer: A

Q65. Which programming model involves tasks sharing a common address space?
A) Message Passing
B) Data Parallel
C) Shared Memory
D) Hybrid

Correct Answer: C

Q66. In the Shared Memory Model, how do tasks read and write?
A) Synchronously
B) Asynchronously
C) Sequentially
D) Predictably
Correct Answer: B
Q67. What mechanisms are used to control access to shared memory?
A) Routers and Switches
B) Compilers and Linkers
C) Locks / Semaphores
D) TCP / IP

Correct Answer: C

Q68. The Threads Model is commonly associated with which architecture?

A) Distributed Memory
B) Shared Memory
C) Vector Processing
D) Single Processor

Correct Answer: B

Q69. A thread's work is best described as analogous to:

A) A separate program entirely
B) A subroutine within the main program
C) A hardware interrupt
D) A network packet

Correct Answer: B

Q70. Which standard resulted in Pthreads?

A) IEEE POSIX 1003.1c
B) ISO 9001
C) ANSI C
D) IEEE 802.11

Correct Answer: A

Q71. OpenMP is based on:

A) Library calls only
B) Compiler directives
C) Assembly language
D) Java bytecode
Correct Answer: B
Q72. OpenMP provides for:
A) Automatic vectorization
B) Hardware upgrades
C) Incremental parallelism
D) Database management

Correct Answer: C

Q73. In the Message Passing Model, how do tasks exchange data?

A) By reading global variables
B) By sending and receiving messages
C) By writing to a shared hard drive
D) By using hardware locks

Correct Answer: B

Q74. Which is the "de facto" industry standard for message passing?
A) PVM
B) MPI
C) HPF
D) OpenMP

Correct Answer: B

Q75. In the Data Parallel Model, tasks usually perform:

A) Different operations on the same data
B) The same operation on different partitions of data
C) Different operations on different data
D) No operations on data

Correct Answer: B

Q76. High Performance Fortran (HPF) is an implementation of which model?

A) Shared Memory
B) Message Passing
C) Data Parallel
D) Threads
Correct Answer: C
Q77. SPMD stands for:
A) Single Process Multiple Data
B) Single Program Multiple Data
C) Shared Program Many Data
D) Synchronous Program Multiple Devices

Correct Answer: B

Q78. In SPMD, all tasks execute:

A) Different programs
B) The same program simultaneously
C) The same instruction at the same time
D) Programs on the same processor

Correct Answer: B

Q79. Which model has multiple executable object files (programs)?

A) SPMD
B) SIMD
C) MPMD
D) SISD

Correct Answer: C

Q80. What is the first step in designing parallel software?

A) Writing the code
B) Buying the hardware
C) Understanding the problem
D) Testing the network

Correct Answer: C

Q81. The calculation of the Fibonacci series is an example of:

A) A parallelizable problem
B) A non-parallelizable problem
C) An embarrassingly parallel problem
D) A data parallel problem
Correct Answer: B
Q82. When identifying hotspots, where should you focus parallelization efforts?
A) Sections that account for little CPU usage
B) The entire program equally
C) Places where most of the real work is being done
D) The user interface

Correct Answer: C

Q83. Which of the following is considered an inhibitor to parallelism?

A) Data independence
B) Data dependence
C) Excess memory
D) Fast networks

Correct Answer: B

Q84. Breaking a problem into "chunks" of work is known as:

A) Compilation
B) Aggregation
C) Decomposition or Partitioning
D) Serialization

Correct Answer: C

Q85. In Domain Decomposition, what is decomposed?

A) The computation work
B) The data associated with the problem
C) The main program code
D) The hardware resources

Correct Answer: B

Q86. In Functional Decomposition, the problem is decomposed according to:

A) The data structure
B) The work that must be done
C) The memory available
D) The number of processors
Correct Answer: B
Q87. Which creates a need for communications between tasks?
A) Embarrassingly parallel problems
B) Tasks acting independently
C) Tasks needing to share data
D) Tasks running on the same processor

Correct Answer: C

Q88. Problems that require very little inter-task communication are called:
A) Massively parallel
B) Embarrassingly parallel
C) Synchronously parallel
D) Strictly parallel

Correct Answer: B

Q89. Which factor is defined as the time it takes to send a minimal (0 byte) message?
A) Bandwidth
B) Throughput
C) Latency
D) Overhead

Correct Answer: C

Q90. Sending many small messages often causes which factor to dominate
communication overhead?
A) Bandwidth
B) Latency
C) CPU speed
D) RAM speed
Correct Answer: B

Q91. Synchronous communications are often referred to as:

A) Non-blocking
B) Blocking
C) Instant
D) Parallel
Correct Answer: B
Q92. Asynchronous communications allow tasks to:
A) Wait for data to arrive
B) Transfer data independently and do other work
C) Block until transfer is complete
D) Share memory addresses directly

Correct Answer: B

Q93. Which type of communication involves two tasks (one sender, one receiver)?
A) Collective
B) Broadcast
C) Point-to-point
D) Scatter

Correct Answer: C

Q94. Which type of communication involves data sharing between more than two tasks in a
group?
A) Point-to-point
B) Blocking
C) Collective
D) Non-blocking

Correct Answer: C

Q95. Which type of tool analyzes source code to identify opportunities for parallelism?
A) Debugger
B) Parallelizing compiler
C) Text editor
D) Network sniffer
Correct Answer: B

Q96. What is a caveat of using automatic parallelization tools?

A) They are too flexible
B) They work on all types of code
C) Performance may actually degrade
D) They never produce wrong results
Correct Answer: C
Q97. Automatic parallelization usually targets which structure?
A) I/O statements
B) Loops (do, for)
C) Variable declarations
D) Comments

Correct Answer: B

Q98. Climate modeling is an example of which type of decomposition?

A) Domain decomposition only
B) Functional decomposition only
C) A combination of domain and functional decomposition
D) No decomposition

Correct Answer: C

Q99. In Signal Processing examples, passing data through distinct filters represents:
A) Domain decomposition
B) Functional decomposition
C) Data replication
D) Serial execution

Correct Answer: B

Q100. Ecosystem modeling often exchanges information with:

A) A central database only
B) Neighbor populations
C) The user interface only
D) The printer

Correct Answer: B
Part 2: Essay Questions
Q1. Define Parallel Computing and explain how it differs from traditional serial
computation. (10 Marks)
Model Answer:
• Parallel computing is the simultaneous use of multiple compute resources to solve
a computational problem.
• Traditionally, serial computation runs on a single computer with a single Central
Processing Unit (CPU).
• In serial computation, a problem is broken into a discrete series of instructions
executed one after another.
• In parallel computing, a problem is broken into discrete parts that can be solved
concurrently.
• Instructions from each part in parallel computing execute simultaneously on
different CPUs.
• Serial computation allows only one instruction to execute at any moment in time.

Q2. List and briefly describe four examples of "real world" parallel events mentioned in the
lecture. (10 Marks)
Model Answer:
• Planetary and galactic orbits: Multiple celestial bodies moving simultaneously
under physical laws.
• Weather and ocean patterns: Complex, interrelated events happening at the same
time.
• Automobile assembly line: Different parts of a car being assembled concurrently at
different stations.
• Rush hour traffic: Multiple independent vehicles moving simultaneously through a
network.
• (Other options include: Daily operations within a business, Building a shopping
mall).
Q3. Explain the physical and practical limitations that constrain the building of ever-faster
serial computers. (10 Marks)
Model Answer:
• Transmission speeds: The speed of a serial computer depends on how fast data
moves through hardware.
• Speed of Light: There is an absolute limit to speed (30 cm/nanosecond).
• Copper Wire limit: Transmission is further limited in copper wire (9
cm/nanosecond).
• Proximity: Increasing speeds requires components to be extremely close together.
• Limits to miniaturization: Even with atomic-level components, there is a limit to how
small transistors can be.
• Economic limitations: It is increasingly expensive to make a single processor faster
compared to using multiple commodity processors.

Q4. Describe the Von Neumann Architecture and its basic design principles. (10 Marks)
Model Answer:
• It uses the stored-program concept where the CPU executes a stored program.
• The program specifies a sequence of read and write operations on the memory.
• Memory is used to store both program instructions and data.
• Program instructions are coded data telling the computer what to do.
• The Central Processing Unit (CPU) gets instructions/data from memory.
• The CPU decodes instructions and performs them sequentially.

Q5. Explain the SISD (Single Instruction, Single Data) architecture from Flynn's Taxonomy.
(10 Marks)
Model Answer:
• SISD represents a serial (non-parallel) computer.
• Single Instruction: Only one instruction stream is acted on by the CPU during any
clock cycle.
• Single Data: Only one data stream is used as input during any clock cycle.
• Execution is deterministic.
• It is the oldest and most prevalent form of computer (e.g., most PCs, single CPU
workstations).
Q6. Describe the SIMD (Single Instruction, Multiple Data) architecture and give an example
of its use. (10 Marks)
Model Answer:
• SIMD is a type of parallel computer where all processing units execute the same
instruction at a given clock cycle.
• Multiple Data means each processing unit operates on a different data element.
• It is best suited for specialized problems with a high degree of regularity, such as
image processing.
• Execution is synchronous (lockstep) and deterministic.
• Varieties include Processor Arrays and Vector Pipelines.
• Examples include the Connection Machine CM-2 or Cray C90.

Q7. Explain the MIMD (Multiple Instruction, Multiple Data) architecture. (10 Marks)
Model Answer:
• MIMD is currently the most common type of parallel computer.
• Multiple Instruction means every processor may execute a different instruction
stream.
• Multiple Data means every processor may work with a different data stream.
• Execution can be synchronous or asynchronous.
• Execution can be deterministic or non-deterministic.
• Examples include current supercomputers, clusters, and multi-processor SMP
computers.

Q8. Define the terms "Shared Memory" and "Distributed Memory" in the context of parallel
computing hardware. (10 Marks)
Model Answer:
• Shared Memory (Hardware): Architecture where all processors have direct access to
common physical memory (usually bus-based).
• Shared Memory (Programming): Parallel tasks share the same "picture" of memory
and address the same logical locations.
• Distributed Memory (Hardware): Refers to network-based memory access where
physical memory is not common to all processors.
• Distributed Memory (Programming): Tasks can only "see" local machine memory
and must use communications to access memory on other machines.
Q9. What is "Granularity" in parallel computing, and what is the difference between Coarse
and Fine granularity? (10 Marks)
Model Answer:
• Granularity is a qualitative measure of the ratio of computation to communication.
• Coarse Granularity: Relatively large amounts of computational work are done
between communication events.
• Fine Granularity: Relatively small amounts of computational work are done between
communication events.

Q10. Explain the concept of "Parallel Overhead" and list factors that contribute to it. (10
Marks)
Model Answer:
• Parallel Overhead is the amount of time required to coordinate parallel tasks, as
opposed to doing useful work.
• Task start-up time contributes to overhead.
• Synchronizations between tasks cause waiting/overhead.
• Data communications consume time.
• Software overhead imposed by parallel compilers, libraries, tools, and operating
systems.
• Task termination time.

Q11. Compare UMA (Uniform Memory Access) and NUMA (Non-Uniform Memory Access)
shared memory architectures. (10 Marks)
Model Answer:
• UMA: Represents Symmetric Multiprocessor (SMP) machines with identical
processors.
• UMA: Processors have equal access and access times to memory.
• UMA: Often Cache Coherent (CC-UMA).
• NUMA: Often made by physically linking two or more SMPs.
• NUMA: Not all processors have equal access time to all memories.
• NUMA: Memory access across the link is slower than local access.
Q12. Discuss the advantages and disadvantages of Shared Memory architectures. (10
Marks)
Model Answer:
• Advantage: Global address space provides a user-friendly programming
perspective.
• Advantage: Data sharing is fast and uniform due to the proximity of memory to
CPUs.
• Disadvantage: Lack of scalability between memory and CPUs (geometric increase
in traffic).
• Disadvantage: Programmer responsibility for synchronization constructs.
• Disadvantage: Increasingly expensive to design and produce with large numbers of
processors.

Q13. Discuss the advantages and disadvantages of Distributed Memory architectures. (10
Marks)
Model Answer:
• Advantage: Memory is scalable with the number of processors.
• Advantage: Each processor accesses local memory without interference or cache
coherency overhead.
• Advantage: Cost-effective (uses commodity processors and networking).
• Disadvantage: Programmer is responsible for data communication details.
• Disadvantage: Difficult to map global memory data structures to distributed
organization.
• Disadvantage: Non-uniform memory access times.

Q14. Describe the Hybrid Distributed-Shared Memory architecture. (10 Marks)

Model Answer:
• It employs both shared and distributed memory architectures.
• The shared memory component is usually a cache-coherent SMP machine.
• Processors on a given SMP address that machine's memory as global.
• The distributed memory component is the networking of multiple SMPs.
• SMPs know only their own memory; network communications are required to move
data between SMPs.
• It is the architecture used by the largest and fastest computers today.
Q15. Explain the Threads Model of parallel programming. (10 Marks)
Model Answer:
• A single process can have multiple, concurrent execution paths (threads).
• Each thread has local data but shares the entire resources of the main program
([Link]).
• This saves overhead associated with replicating program resources.
• Threads communicate through global memory.
• Requires synchronization constructs to prevent conflicting updates to global
memory.
• Threads can come and go, but the main program remains.

Q16. Compare POSIX Threads and OpenMP implementations. (10 Marks)

Model Answer:
• POSIX Threads (Pthreads): Library-based implementation.
• Pthreads: Specified by IEEE standard, C Language only.
• Pthreads: Very explicit parallelism requiring significant programmer attention.
• OpenMP: Compiler directive based.
• OpenMP: Portable and multi-platform (Unix and Windows).
• OpenMP: Available in C/C++ and Fortran.
• OpenMP: Can be easier to use and allows "incremental parallelism".

Q17. Describe the Message Passing Model. (10 Marks)

Model Answer:
• A set of tasks use their own local memory during computation.
• Tasks can reside on the same machine or across arbitrary numbers of machines.
• Tasks exchange data by sending and receiving messages.
• Data transfer usually requires cooperative operations (send must have a matching
receive).
• MPI (Message Passing Interface) is the de facto industry standard.
Q18. Explain the Data Parallel Model. (10 Marks)
Model Answer:
• Parallel work focuses on performing operations on a data set (e.g., array or cube).
• A set of tasks work collectively on the same data structure.
• Each task works on a different partition of the same data structure.
• Tasks perform the same operation on their partition (e.g., "add 4 to every element").
• On distributed memory, data structures are split into "chunks" in local memory.

Q19. Describe the Single Program Multiple Data (SPMD) model. (10 Marks)
Model Answer:
• It is a high-level programming model built upon other models.
• A single program is executed by all tasks simultaneously.
• Tasks can execute the same or different instructions within that program at any
moment.
• Logic is programmed to allow tasks to branch or execute only specific portions of
the program.
• All tasks may use different data.

Q20. What is Automatic Parallelization and what are its caveats? (10 Marks)
Model Answer:
• Automatic Parallelization is when a compiler or pre-processor converts serial code
to parallel code.
• Caveat: Wrong results may be produced.
• Caveat: Performance may actually degrade.
• Caveat: It is much less flexible than manual parallelization.
• Caveat: It is limited to a subset of code (mostly loops).
• Caveat: It may fail if code is too complex.

Q21. Explain Domain Decomposition. (10 Marks)

Model Answer:
• It is a method of partitioning computational work.
• The data associated with the problem is decomposed.
• Each parallel task works on a portion of the data.
• It contrasts with Functional Decomposition.
 • Data can be partitioned in various ways (e.g., 1D, 2D, Block, Cyclic). [67 - Diagram]
Q22. Explain Functional Decomposition and give examples of problems suited for it. (10
Marks)
Model Answer:
• The focus is on the computation to be performed rather than the data.
• The problem is decomposed according to the work that must be done.
• Each task performs a portion of the overall work.
• Examples include Ecosystem Modeling.
• Examples include Signal Processing.
• Examples include Climate Modeling.

Q23. Describe the "Ecosystem Modeling" example of problem decomposition. (10 Marks)
Model Answer:
• The program calculates the population of given groups (Plants, Herbivores,
Carnivores, etc.).
• Each group's growth depends on its neighbors.
• Each process calculates its current state.
• Processes exchange information with neighbor populations.
• All tasks progress to calculate the state at the next time step.
• This represents a functional decomposition where tasks represent different
population types. [68 - Diagram]

Q24. Explain the difference between Latency and Bandwidth in communications. (10
Marks)
Model Answer:
• Latency is the time it takes to send a minimal (0 byte) message from point A to B.
• Latency is commonly expressed in microseconds.
• Bandwidth is the amount of data that can be communicated per unit of time.
• Bandwidth is commonly expressed in megabytes/sec.
• Sending many small messages causes latency to dominate overhead.
• Packaging small messages into larger ones increases effective bandwidth.
Q25. Compare Synchronous and Asynchronous communications. (10 Marks)
Model Answer:
• Synchronous (Blocking): Requires handshaking between tasks.
• Synchronous: Other work must wait until communications complete.
• Asynchronous (Non-Blocking): Allows tasks to transfer data independently.
• Asynchronous: A task can send a message and immediately do other work.
• Asynchronous: Interleaving computation with communication is a major benefit.

Q26. Explain "Point-to-Point" vs "Collective" communications scoping. (10 Marks)

Model Answer:
• Point-to-Point: Involves two tasks.
• Point-to-Point: One task acts as sender/producer, the other as receiver/consumer.
• Collective: Involves data sharing between more than two tasks.
• Collective: Tasks are often specified as members of a common group.
• Examples of collective include broadcast, scatter, gather, and reduction. [Diagram
slide 77]

Q27. How does the "Climate Modeling" example demonstrate hybrid decomposition? (10
Marks)
Model Answer:
• The model components (Atmosphere, Ocean, Hydrology, Land) are separate tasks
(Functional Decomposition).
• Arrows represent data exchange: Atmosphere generates wind data for Ocean;
Ocean generates temperature data for Atmosphere.
• Combining functional decomposition with domain decomposition (likely within
each model) is common.

Q28. Why do some problems NOT need communications? (10 Marks)

Model Answer:
• Some problems can be decomposed and executed with virtually no need to share
data.
• Example: Image processing where every pixel's color is reversed independently.
• The data can be distributed to multiple tasks that act independently.
• These problems are often called "embarrassingly parallel".
Q29. What is the role of the programmer in the Message Passing Model versus the Data
Parallel Model regarding communications? (10 Marks)
Model Answer:
• Message Passing Model: Communications are explicit.
• Message Passing Model: Communications are generally visible and under the
control of the programmer.
• Data Parallel Model: Communications often occur transparently to the programmer.
• Data Parallel Model: On distributed memory, the programmer may not know exactly
how inter-task communications are accomplished.

Q30. List five factors to consider when designing parallel program communications. (10
Marks)
Model Answer:
• Cost of communications (overhead).
• Latency vs. Bandwidth.
• Visibility of communications (explicit vs. transparent).
• Synchronous vs. Asynchronous communications.
• Scope of communications (Point-to-Point vs. Collective).

Q31. Explain the concept of "Scalability" in parallel systems and list four factors that
contribute to it. (10 Marks)
Model Answer:
• Scalability refers to a parallel system's ability to demonstrate a proportionate
increase in parallel speedup with the addition of more processors.
• It applies to both hardware and software aspects of the system.
• One factor is hardware capability, particularly memory-cpu bandwidths and
network communications.
• The application algorithm itself is a contributing factor.
• Parallel overhead related to the specific implementation affects scalability.
• Characteristics of the specific application and its coding also play a role.
Q32. Describe the MISD (Multiple Instruction, Single Data) architecture and provide
examples of its conceivable uses. (10 Marks)
Model Answer:
• In MISD, a single data stream is fed into multiple processing units.
• Each processing unit operates on the data independently via independent
instruction streams.
• There are very few actual examples of this architecture (e.g., Carnegie-Mellon
[Link]).
• A conceivable use is multiple frequency filters operating on a single signal stream.
• Another use is multiple cryptography algorithms attempting to crack a single coded
message.

Q33. Differentiate between "Fully Automatic" and "Programmer Directed" parallelization in

compilers. (10 Marks)
Model Answer:
• Fully Automatic: The compiler analyzes the source code and identifies
opportunities for parallelism on its own.
• It identifies inhibitors to parallelism and may apply a cost weighting to determine
performance benefits.
• Loops (do, for) are the most frequent target for fully automatic parallelization.
• Programmer Directed: The programmer explicitly tells the compiler how to
parallelize the code.
• This is typically done using "compiler directives" or compiler flags.
• Programmer directed methods can be used in conjunction with some degree of
automatic parallelization.

Q34. Explain the Multiple Program Multiple Data (MPMD) programming model. (10 Marks)
Model Answer:
• MPMD is a high-level programming model built upon other models (like Shared
Memory or Message Passing).
• MPMD applications typically have multiple executable object files (programs).
• While the application runs in parallel, each task can execute the same or a different
program from other tasks.
• All tasks may use different data.
 • It differs from SPMD, where all tasks execute the same program.
Q35. Discuss the "Cost of Communications" as a factor in designing parallel programs. (10
Marks)
Model Answer:
• Inter-task communication virtually always implies overhead.
• Machine cycles and resources that could be used for computation are instead used
to package and transmit data.
• Communications frequently require synchronization, which can result in tasks
waiting instead of working.
• Competing communication traffic can saturate available network bandwidth.
• Saturation of bandwidth further aggravates performance problems.

Q36. Describe High Performance Fortran (HPF) as an implementation of the Data Parallel
Model. (10 Marks)
Model Answer:
• HPF consists of extensions to Fortran 90 to support data parallel programming.
• It contains everything included in standard Fortran 90.
• It adds directives that tell the compiler how to distribute data.
• It includes assertions that can improve the optimization of the generated code.
• Data parallel constructs added in HPF are now part of the Fortran 95 standard.
• Implementations are available for most common parallel platforms.

Q37. Explain how Parallel Programming Models are abstractions and not strictly tied to
hardware, providing two examples from the lecture. (10 Marks)
Model Answer:
• Parallel programming models exist as an abstraction above hardware and memory
architectures.
• Any model can theoretically be implemented on any underlying hardware.
• Example 1: The Shared Memory Model can be implemented on a distributed
memory machine (e.g., KSR ALLCACHE approach used "virtual shared memory").
• Example 2: The Message Passing Model (MPI) is frequently implemented on shared
memory machines (e.g., SGI Origin).
• On shared memory machines, MPI usually uses memory copies rather than a
network for performance.
Q38. Compare CC-UMA, CC-NUMA, and Distributed systems in terms of "Scalability" and
"Drawbacks" using the lecture comparison table. (10 Marks)
Model Answer:
• CC-UMA Scalability: Limited to 10s of processors.
• CC-UMA Drawbacks: Memory-CPU bandwidth issues.
• CC-NUMA Scalability: Scalable to 100s of processors.
• CC-NUMA Drawbacks: Non-uniform access times and complex system
administration.
• Distributed Scalability: Scalable to 1000s of processors.
• Distributed Drawbacks: Programming is hard to develop and maintain.

Q39. Besides saving time, what are other significant reasons for using parallel computing?
(10 Marks)
Model Answer:
• Concurrency: To do multiple things at the same time.
• Non-local Resources: To take advantage of resources on a wide area network or the
Internet when local resources are scarce.
• Cost Savings: Using multiple "cheap" commodity computing resources instead of
paying for time on a supercomputer.
• Memory Constraints: Single computers have finite memory; parallel computing
allows using the collective memories of multiple computers for large problems.

Q40. How should a programmer approach identifying "Hotspots" and "Bottlenecks" when
designing a parallel program? (10 Marks)
Model Answer:
• Hotspots: Identify where most of the real work is being done, as scientific programs
usually accomplish most work in a few places.
• Profilers and performance analysis tools should be used to find these hotspots.
• Parallelization efforts should focus on these hotspots and ignore sections with little
CPU usage.
• Bottlenecks: Identify areas that are disproportionately slow or cause work to halt
(e.g., I/O).
• Programmers should look to restructure the program or use different algorithms to
reduce or eliminate these unnecessary slow areas.