Module-2

Introduction to High Performance Computing:

Principles of HPC
• 1. Parallelism
• Definition:
• Parallelism is the practice of performing multiple computations simultaneously to
increase computational speed and efficiency.
• Types:
• Data Parallelism: Distributing data across multiple processors so that each processor
performs the same operation on different pieces of data.
• Task Parallelism: Distributing different tasks or processes across multiple processors,
where each processor performs a different operation.
• Example:
• Processing large datasets by splitting the data into chunks and processing each chunk in
parallel.
• 2. Scalability
• Definition:
• Scalability refers to the ability of an HPC system to maintain performance as more
resources (e.g., processors, memory) are added.
• Types:
• Vertical Scalability: Adding more power (e.g., more CPUs or memory) to a single node.
• Horizontal Scalability: Adding more nodes to a cluster or grid to distribute the
computational load.
• Challenges:
• Ensuring that performance improvements scale linearly with the addition of resources.
• 3. Architecture
• Definition:
• The architecture of an HPC system encompasses the hardware and software
components that work together to perform high-speed computations.
• Key Architectures:
• Clusters: Multiple interconnected computers (nodes) working together as a single
system.
• Supercomputers: Extremely powerful and large-scale systems with thousands of
processors.
• Grid Computing: A distributed network of computers that work together to perform
tasks.
• Cloud Computing: Utilizing cloud-based resources for on-demand computing power.
HPC Architecture
HPC Architectures
• 1. Cluster Architecture
• HPC cluster architecture consists of multiple computer servers networked
together to form a cluster that offers more performance than a single
computer.
• How does HPC architecture differ from traditional computing architecture?
• Although consumer-grade computers can share GPU computing, hardware
acceleration, and many of the same techniques that HPC systems do, there
are major differences between HPC and traditional computing systems.
Especially in areas where HPC is essential to operations, computing and
processing power from HPC systems is exponentially higher than traditional
computers.
• This difference is particularly acute for throughput, scaling speed, and
processing. Thus HPC architecture tends to serve as the foundation for the
cloud. However, HPC architectures can take various forms—most often
cluster, parallel, and grid systems.
• Cluster Computing Architecture: A cluster is a set of multiple
computers that together operates as a single entity, called a node, to
collectively work on the same set of tasks.
• Parallel Computing: With a parallel computing configuration, nodes
are arranged to execute calculations or commands in parallel similar
to GPU processing to scale computation volume and data processing
across a system.
• Grid Computing: Grid computing systems distribute parts of a more
complex problem across multiple nodes.
What is an HPC Cluster?

• HPC meaning: An HPC cluster is a collection of components that enable applications to be

executed. The software typically runs across many nodes and accesses storage for data reads
and writes.
• Typically, HPC cluster components include:
• Head or Login Node: This node validates users and may set up specific software on the
compute nodes.
• Compute Nodes: These perform numerical computations. Their persistent storage may be
minimal, while the DRAM memory will be high.
• Accelerator Nodes: Some nodes may include one or more accelerators, while smaller HPC
clusters, purpose built for a specific use may be set up where all nodes contain an accelerator.
• Storage Nodes or Storage System: An efficient HPC cluster must contain a high performance,
parallel file system (PFS). A PFS allows all nodes to communicate in parallel to the storage
drives. HPC storage allows for the compute nodes to operate with minimal wait times.
• Network Fabric: In HPC clusters, typically low latency and high
bandwidth are required.
• Software: HPC cluster computing requires underlying software to
execute applications and control underlying infrastructure. Software is
essential to the efficient management of the massive amounts of I/O
that are inherent to HPC applications.
• What are the Core/Basic HPC Cluster Components
• High performance computing (HPC) generally processes complex calculations
at high speeds in parallel over multiple servers in groups called clusters.
Although hundreds or even thousands of compute servers may be linked in
an HPC cluster, each component computer is still referred to as a node.
• HPC solutions may be deployed in the cloud or on-premises. Flexibility is key.
A typical HPC solution has 3 main components:
• Compute
• Network
• Storage
• Compute
• Compute hardware includes a dedicated network with servers and
storage and focuses on algorithms, data processing, and problem
solving. In general, a minimum of three servers must be provisioned as
primary, secondary, and client nodes. more available to the cluster if
you virtualize multiple servers.
• The networking infrastructure to support the compute power demands
high-bandwidth TCP/IP network equipment, such as NICs, Gigabit
Ethernet, and switches. Typically, the HPC cluster is managed by the
software layer, including tools for monitoring and provisioning file
systems and executing cluster functions.
• Network
• Reliable, rapid networking is critical to successful HPC architecture,
whether for moving data between computing resources, ingesting
external data, or transferring data between storage resources. Physical
space is needed to house HPC clusters, and sufficient power is necessary
to operate and cool them.
• Storage
• Cloud storage offers high volume and speeds for access and retrieval,
making it integral to the success of an HPC system. Traditionally, external
storage has been the slowest piece of a computer system, but HPC
storage meets the rapid needs of HPC workloads.
• A Glance at HPC Cluster Design
• With these essential elements in mind, let’s move on to HPC Cluster Design and how to build HPC
clusters to specifications. We will approach this piece by piece.
• HPC Cluster Configuration Explained
• An HPC cluster is many separate computers or servers called nodes collected together and
connected via some high-speed means. Different types of nodes may handle different types of
tasks.
• As explained above, typical HPC clusters have:
• A login or headnode node
• A data transfer node
• Compute nodes, some with fat compute nodes with a minimum of 1TB of memory
• Graphical Processing Unit (GPU) nodes for computations
• Connecting switch for nodes
• Most HPC storage solutions are file-based with POSIX support. These
file-based HPC storage solutions can typically be divided into general
purpose and parallel storage solutions.
• While some solutions such as Blob (Binary Large Objects) storage or
object storage can be used directly by some workloads, not all share that
capability.
• General-purpose storage
• General-purpose storage in a HPC cluster is used to store available
application binaries with their libraries to maintain consistency while an
application runs. It is also used for consistent access to user data such as
home directories throughout the HPC cluster.
• Parallel file systems
• A parallel or clustered file system is a shared file system multiple users access at once which serves
the storage resources of multiple servers simultaneously. This direct client access to stored data
avoids abstraction and eliminates overhead, resulting in high performance and low latency.
• Solutions in this space include:
• WEKA
• Ceph via CephFS
• BeeGFS
• Lustre / DDN Lustre
• DAOS
• GPFS / IBM Spectrum Scale
• VAST Data
• Panasas PanNFS
HPC vs Parallel Processing
• 1. High-Performance Computing (HPC):
• Definition: HPC refers to the use of supercomputers and parallel processing techniques to
solve complex computational problems that are too large or time-consuming for standard
computers. HPC systems are designed to deliver high levels of computational power and
speed, often used in scientific research, engineering, simulations, and data analysis.
• Hardware: HPC involves specialized hardware, such as supercomputers, clusters of servers,
and specialized processors (like GPUs).
• Software: HPC applications often require specialized software optimized for performance,
such as scientific simulations, climate modeling, and large-scale data analysis.
• Scale: HPC systems typically involve thousands or even millions of processors working
together to solve large-scale problems.
• Use Cases: Examples include weather forecasting, molecular modeling, physics
simulations, and large-scale financial modeling.
• 2. Parallel Processing:
• Definition: Parallel processing is a computational technique where multiple processors execute
or process an application or computation simultaneously. The main goal is to perform many
calculations or processes at the same time to speed up processing.
• Hardware: Parallel processing can occur on a variety of hardware, from multi-core CPUs in
personal computers to large clusters and supercomputers used in HPC. It can also occur in GPUs,
which are specifically designed for parallel tasks.
• Software: Parallel processing can be implemented at the software level, where tasks are divided
into smaller subtasks that can be executed concurrently. This requires careful programming and
often relies on parallel algorithms and techniques.
• Scale: Parallel processing can range from a few cores in a single CPU to thousands of cores in a
distributed system.
• Use Cases: Examples include image processing, simulations, real-time data processing, and any
task that can be divided into independent sub-tasks.
• Key Differences:
• Scope: HPC is a broader term that includes not just parallel processing, but also the hardware,
software, and network infrastructure required to achieve high performance. Parallel processing
is a specific technique used within HPC and other computing environments.
• Focus: HPC focuses on achieving the highest possible computational performance for complex
tasks, while parallel processing focuses on increasing the speed of computation by dividing
tasks across multiple processors.
• Application: While all HPC systems use parallel processing, not all parallel processing is part of
an HPC system. Parallel processing can be used in everyday computing (like multi-core
processors in desktops) or in specialized environments like HPC.
• In summary, HPC is a broader field that encompasses the use of advanced computing resources
and techniques, including parallel processing, to achieve high performance in computational
tasks. Parallel processing is one of the key techniques used within HPC to accelerate
computations by dividing tasks across multiple processors.
Data partitioning Techniques
Partitioning is used to improve performance, scalability, and
availability
Case Study: Partitioning strategies
for matrix multiplication
• Partitioning strategies for matrix multiplication are crucial for
improving the efficiency of computations, especially when dealing
with large matrices or when performing matrix multiplication in
parallel computing environments. Here’s an overview of the key
strategies and considerations involved:
Domain Decomposition
• Domain decomposition is a strategy used in parallel computing to
break down a large-scale problem into smaller, more manageable
subproblems. Each subproblem can then be processed independently
by different computing units or processors. The goal is to improve
efficiency and scalability. Here’s a detailed overview of the three main
types of domain decomposition: spatial, temporal, and functional.
Spatial Data Partitioning (SDP)
• Description. Using the spatial data partitioning (SDP) strategy, data is
divided among processes according to spatial index. Following this
pattern, processes perform computation on spatially distinct data
with the same temporal index. Typically, each process will perform all
instructions on its assigned data. Additional instructions are usually
added to SDP programs to enable communication and
synchronization. This partitioning strategy is illustrated in Figure
• Applicability. SDP generally increases throughput and decreases
latency, so it is useful when a single processor cannot meet either of
these requirements for a given application. The performance of an
SDP application will be best if the spatial data dimension is large and
has few dependences.
• Forces. The driving force behind the use of the SDP strategy is the
improvement of both throughput and latency for the partitioned
application. However, when applying this strategy one must consider
the opposing forces of communication and load-balancing.
• Communication can oppose the performance gains of the SDP strategy if the time
spent transferring data is large enough to negate the performance benefit of
parallelization. Unfortunately, communication in an SDP implementation is always
application specific and, thus, there are no general guidelines for analyzing
communication using this strategy. However, if a particular application is found to
require a performance-limiting amount of communication, spatial instruction
partitioning may be considered, as SIP can also improve throughput and latency.
• Load-balancing is another force limiting the potential performance gains of SDP.
Since each process is responsible for executing all instructions on its assigned data
the load-balance of an SDP application is determined by the relationship between
the instructions and data. If the same instructions are executed regardless of the
value of data, then an SDP implementation will be easy to load-balance. If the
instructions executed are conditional on the value of data then an SDP
implementation may be very difficult to loadbalance as some data may cause one
process to do more work than others. This difficulty can offset some of the
performance gains of an SDP implementation. Implementation strategy patterns
like the task-queue and the masterworker pattern can help address load-imbalance
• Example in security camera. To implement the SDP strategy in the
security camera example, separate processes work simultaneously on
pixels from the same frame. Each process is responsible for executing
the search, eliminate, and encode functions on its assigned pixels.
Processes communicate with other processes responsible for
neighboring spatial indices.
• Other examples of SDP. This pattern is common in many parallel linear
algebra implementations like ScaLAPACK [1] and PLAPACK [24]. The
High Performance For tran language is designed to express and
manipulate SDP at a high-level [8]. Single-instruction, multiple-data
(SIMD) processor architectures are designed to exploit SDP
• Related Strategy Patterns. DataDecomposition from PPP and Data
Parallelism from OPL are both used to take a data-centric approach to
partitioning an application into concurrent units. In order to perform this
decomposition an index set is defined over the program’s data. The
spatial data partitioning strategy is a sub-strategy of DataDecomposition
and Data Parallelism in which the index set is restricted to spatial indices.
• The TaskDecomposition (PPP) and Task Parallelism (OPL) patterns both
take a task-centric approach to application decomposition. These
patterns allow individual tasks to be composed of the same set of
instructions. The case where each task is the same function applied to
different spatial indices is equivalent to the SDP strategy.
Temporal Data Partitioning (TDP)
• Description. Using the temporal data partitioning (TDP) strategy, data
are divided among processes according to temporal index. Following
this pattern, each process performs computation on all spatial indices
associated with its assigned temporal index as illustrated in Figure
3(b). In a typical TDP implementation each process executes all
instructions on the data from its assigned temporal index. Often,
communication and synchronization instructions need to be added to
allow processes to handle temporal data dependences.
• Applicability. TDP can increase throughput but will not decrease latency. Thus, this
strategy is useful when a single processor can meet the application’s latency
requirement, but not the throughput requirement. The performance of a TDP
application will be best when the temporal dimension is large and has few dependences.
• Forces. The driving force behind the use of the TDP pattern is the improvement in
throughput. Opposing this force is the possible overhead of communication and
loadbalancing.
• Load-balancing in a TDP application tends to be easy. If the same work is done for each
temporal data index then achieving good load-balance is trivial. If the computational
load varies per temporal data index it can be more difficult to achieve an efficient load-
balance; however, the TDP strategy naturally lends itself to implementation through
patterns such as the work-queue and master-worker that can be used to address a load-
imbalance
• Example in security camera. To implement TDP in the security camera example,
each frame is assigned to a separate process and multiple frames are encoded
simultaneously. A process is responsible for executing the eliminate, encode, and
search functions on its assigned frame. Processes receive data from processes
working on earlier temporal indices and send data to processes working on later
temporal indices.
• Other examples of TDP. Packet processing applications like SNORT are commonly
parallelized using the TDP pattern [10]. This application processes a sequence of
packets from a network, possibly searching them for viruses. As packets arrive
they are assigned to a process which performs the required computation on that
packet. The back-end of the StreamIt compiler for MIT’s Raw processor generates
code parallelized according to the TDP strategy [6]. Related Strategy Patterns.
• The DataDecomposition (PPP) and Data Parallelism (OPL) patterns are both used
to take a data-centric approach to partitioning an application into concurrent
units. In order to perform this decomposition an index set is defined over the
program’s data. The temporal data partitioning strategy is a sub-strategy of
DataDecomposition and Data Parallelism in which the index set is restricted to
temporal indices.
Spatial Domain Decomposition in Image
Processing

Concept:
•A large image is divided into smaller blocks (tiles).
•Each processor (CPU core / GPU thread block) processes one block independently.
•At the boundaries of blocks, processors may need to share data (especially for filters like blurring or edge
detection).

Example: Applying a Filter (e.g., Gaussian Blur)

Suppose we have a 4000 × 4000 pixel image.
•Too large for one processor to handle efficiently.
•Split into 4 blocks of 2000 × 2000 pixels (or more, depending on processors).
Each processor does:
[Link] its assigned block.
[Link] the filter to pixels.
[Link] border pixels with neighbors (to correctly apply convolution filters at edges).
[Link] the processed block.
Finally → Blocks are combined to form the output image.
 Workflow in Image Processing Advantages
[Link] Image → split into blocks. ✅ Handles very large images (gigapixel).
[Link] Processors → each gets one block. ✅ Can run on distributed systems (cluster or GPU grid).
[Link] Computation → apply filter or transform to block. ✅ Blocks can be processed independently → scalable.
[Link] Handling → exchange ghost rows/columns.
[Link] Results → reconstruct final image.

Applications Challenges
•Parallel filtering (blur, sharpening, edge detection) ⚠️Boundary artifacts if ghost cells not exchanged.
•Image compression (block-based DCT in JPEG) ⚠️Load imbalance if image regions vary in
•Computer vision pipelines (object detection in tiles) complexity (e.g., some blocks require more
•Satellite image analysis (huge gigapixel images processed in processing).
chunks) ⚠️Reassembly step needed after processing.
Temporal Domain Decomposition
Functional Domain Decomposition
Types of spatial decomposition
Types of Temporal Decomposition:
Types of Functional
Decomposition:
• Functional Decomposition
• Functional decomposition divides a problem based on different
computational tasks or functions. Each processor is assigned a specific
task, allowing for parallel execution of independent or loosely coupled
tasks.
• Problem: Consider a 2D heat distribution problem on a 6×66 \times
66×6 grid. The goal is to calculate the heat distribution over the grid
after several time steps using spatial decomposition. The grid is
divided among four processors.
Temporal Decomposition Numerical
Example
• Problem: Consider a simulation of a particle moving along a 1D path
over 10 time steps. The position of the particle at each time step is
computed based on a simple equation: x(t+1)=x(t)+, where v is the
velocity of the particle.
Functional Decomposition Numerical
Example
Load Balancing
• Load balancing is the distribution of computational work between available
processors.
• In computing, load balancing is the process of distributing a set of tasks over
a set of resources (computing units), with the aim of making their overall
processing more efficient. Load balancing can optimize response time and
avoid unevenly overloading some compute nodes while other compute nodes
are left idle.
• Load balancing is the subject of research in the field of parallel computers.
Two main approaches exist: static algorithms, which do not take into account
the state of the different machines, and dynamic algorithms, which are usually
more general and more efficient but require exchanges of information between
the different computing units, at the risk of a loss of efficiency.
Static and dynamic algorithms
• Static
• A load balancing algorithm is "static" when it does not take into account the state of the system for
the distribution of tasks. Thereby, the system state includes measures such as the load level (and
sometimes even overload) of certain processors. Instead, assumptions about the overall system are
made beforehand, such as the arrival times and resource requirements of incoming tasks. In addition,
the number of processors, their respective power and communication speeds are known. Therefore,
static load balancing aims to associate a known set of tasks with the available processors in order to
minimize a certain performance function. The trick lies in the concept of this performance function.
• Static load balancing techniques are commonly centralized around a router, or Master, which
distributes the loads and optimizes the performance function. This minimization can take into account
information related to the tasks to be distributed, and derive an expected execution time.
• The advantage of static algorithms is that they are easy to set up and extremely efficient in the case of
fairly regular tasks (such as processing HTTP requests from a website). However, there is still some
statistical variance in the assignment of tasks which can lead to the overloading of some computing
units.
• Dynamic
• Unlike static load distribution algorithms, dynamic algorithms take into account the current
load of each of the computing units (also called nodes) in the system. In this approach, tasks
can be moved dynamically from an overloaded node to an underloaded node in order to
receive faster processing. While these algorithms are much more complicated to design,
they can produce excellent results, in particular, when the execution time varies greatly
from one task to another.
• Dynamic load balancing architecture can be more modular since it is not mandatory to have
a specific node dedicated to the distribution of work. When tasks are uniquely assigned to a
processor according to their state at a given moment, it is a unique assignment. If, on the
other hand, the tasks can be permanently redistributed according to the state of the system
and its evolution, this is called dynamic assignment. Obviously, a load balancing algorithm
that requires too much communication in order to reach its decisions runs the risk of
slowing down the resolution of the overall problem.
Example