Processing Bigdata
Scalability in High Performance Computing
• Scalability has been widely used in practice to describe how system sizes and problem sizes influence the
performance of parallel computers and algorithms.
• High performance computing (HPC) clusters are able to solve big problems using a large number of processors.
• This is also known as parallel computing, where many processors work simultaneously to produce exceptional
computational power and to significantly reduce the total computational time.
• In such scenarios, scalability or scaling is widely used to indicate the ability of hardware and software to deliver
greater computational power when the amount of resources is increased. For HPC clusters, it is important that they
are scalable, in other words that the capacity of the whole system can be proportionally increased by adding more
hardware.
• For software, scalability is sometimes referred to as parallelisation efficiency — the ratio between the actual
speedup and the ideal speedup obtained when using a certain number of processors.
• The speedup in parallel computing can be straightforwardly defined as speedup = t1 / tN ,
where t1 is the
computational time for running the software using one processor, and tN is the computational time running the same
software with N processors.
Scalability in High Performance Computing
• In 1967, Amdahl pointed out that the speedup is limited by the fraction of the serial part of the software that is not
amenable to parallelization.
• Amdahl’s law can be formulated as follows speedup = 1 / (s + p / N), where s is the proportion of execution time
spent on the serial part, p is the proportion of execution time spent on the part that can be parallelized, and N is the
number of processors.
• Amdahl’s law states that, for a fixed problem, the upper limit of speedup is determined by the serial fraction of the
code. This is called strong scaling.
• Weak scaling, or the situation where the overall issue size grows as the number of processors is raised, is a
measurement of how the time to solution changes when additional processors are added to a system with a fixed
problem size per processor.
• This also implies a consistent workload for each CPU. Gustafson’s law often describes weak scaling.
• Gustafson’s Law can be formulated as
S = (1-p) + Np / (1-p) + p
= (1-p) + Np
Where S latency is the theoretical speedup of the execution of the whole task;
N is the speedup of the part of the task that benefits from improved system resources, usually it is the number
of processors;
p is the proportion of execution time that the part benefiting from improved resources originally occupied.
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Computing
Distributed File System
DEFINITIONS:
• A traditional Distributed File System ( DFS ) is simply a classical model of a file system distributed across
multiple machines. The purpose is to promote sharing of dispersed files with good transparency to users.
• The resources on a particular machine are local to itself. Resources on other machines are remote.
• A file system provides a service for clients. The server interface is the normal set of file operations: create,
read, etc. on files.
• Clients, servers, and storage are dispersed across machines. Configuration and implementation may vary -
a) Servers may run on dedicated machines, OR
b) Servers and clients can be on the same machines.
c) The OS itself can be distributed (with the file system a part of that distribution.
d) A distribution layer can be interposed between a conventional OS and the file system.
• Clients should view a DFS the same way they would a centralised FS; the distribution is hidden at a lower level.
Performance is concerned with throughput and response time.
Distributed File System Characteristics
• Naming is the mapping between logical and physical objects.
• In a conventional file system, it's understood where the file actually resides; the system and disk are known.
• In a transparent DFS, the location of a file, somewhere in the network, is hidden.
• Location transparency -
The name of a file does not reveal any hint of the file's physical storage location (machine, disk, or disk blocks).
• Location independence -
• The name of a file doesn't need to be changed when the file's physical storage location changes. Dynamic, one-to-many
mapping.
• Separates the naming hierarchy from the storage devices hierarchy.
• Most DFSs today:
• Support location transparent systems.
File System for Google (GFS)
● A new distributed file system developed to meet the demands
of Google’s application workloads and technological
environment
● Shares many of the same goals as the other distributed file
systems
2/11/2026 29
Technological environment
● Clusters of more than 15,000 commodity class PCs
● Hardware prone to failure
● Fault tolerant software
Source: "Web Search for a Planet: The Google Cluster Architecture"
2/11/2026 30
Application Workloads
● Sequential reads
● Indexer reading the contents of web pages
● Frequent Appends
● Crawler appending new pages
● Files used as Producer – Consumer queues
● Indexer waits for the crawler to retrieve contents
● Files used for multi-way merging
2/11/2026 31
GFS – Motivation
● Component failures – norm rather than exception
● Efficient management of large files
● Optimization of frequently performed operations
● Flexibility of co-designing application and file system
2/11/2026 32
GFS - Goals
● Reliability, availability, scalability…
● Tolerance to hardware failures
● Managing numerous files of large size
● Optimizing commonly performed operations
2/11/2026 33
GFS Architecture
●Singlemaster ●Masterstores metadata
●Multiple chunkservers ●Chunkservers store data
●Multiple clients ●Communication via heartbeat messages
2/11/2026 34
GFS Architecture (contd)
● Files are broken into fixed sized chunks
● Chunks are identified by unique chunk handles
● Chunks are stored in chunkservers
● Fixed chunk size – 64 MB
2/11/2026 35
GFS Operations
● File open, close …
● Data reads
● Data Mutations
● Random writes
● Record appends
● Snapshot
2/11/2026 36
Apache Hadoop Basic Modules
● Hadoop Common
● Hadoop Distributed File System (HDFS)
● Hadoop YARN
Other Modules: Zookeeper,
● Hadoop MapReduce Impala, Oozie, etc.
Spark, Storm,
Hive Tez, etc.
Pig
Non-relational
SQL Like
Scripting
Database
Query
HBase
Others
MapReduce
Distributed
Distributed Processing
Processing
Yarn
Resource Manager
HDFS Distributed File System (Storage)
MapReduce: Simple Programming for Big Data
Based on Google’s MR paper (2004)
● MapReduce is simple programming paradigm for the Hadoop ecosystem
● Traditional parallel programming requires expertise of different computing/systems
concepts
● examples: multithreads, synchronization mechanisms (locks, semaphores, and
monitors )
● incorrect use: can crash your program, get incorrect results, or severely impact
performance
● Usually not fault tolerant to hardware failure
● The MapReduce programming model greatly simplifies running code in parallel
● you don't have to deal with any of above issues
● only need to create, map and reduce functions
Hadoop
• MapReduce is a programming model that uses parallel processing to speed large-scale data processing.
• MapReduce enables massive scalability across hundreds or thousands of servers within a Hadoop
cluster.
• The name "MapReduce" refers to the 2 tasks that the model performs to help “chunk” a large data
processing task into many smaller tasks that can run faster in parallel.
• First is the "map task," which takes one set of data and converts it into another set of data formatted as
key/value pairs, such as cities (keys) and daily high temperatures (values).
• Second is the "reduce task," which takes the outputs from a map task, aggregates all values with the
same key and processes the data to produce a final set of key/value pairs.
• The MapReduce programming model is one of the core processing components at the heart of Apache
Hadoop, an open source software framework supported by the Apache Software Foundation and built on
the Java programming language.
• The MapReduce model can be used outside of Hadoop as well.
Mapreduce-
Execution
Map workers read in contents of corresponding input partition
Perform user-defined map computation to create intermediate
<key,value> pairs
Periodically buffered output pairs written to local disk
Reduce
Reduce workers iterate over ordered intermediate data
Each unique key encountered – values are passed to user's
reduce function
eg. <key, [value1, value2,..., valueN]>
Output of user's reduce function is written to output file on global
file system
When all tasks have completed, master wakes up user program
Map
Reduce
Combiner – combines the O/P of a single map task. Same as reducer,
but it stores the intermediate O/P in a local file wrt final output file
Debugging
We can test the tasks locally using special Map reduce libraries
Offers human readable status info on http server
Handling stragglers
• When we are running the map & reduce operations by splitting the computation load
into multiple nodes, even 1 single slow node can end up slowing the overall
MapReduce job. These are known as straggler nodes.
• There can be any number of reasons that can turn a node into straggler. For
example, the network through which the node is reading or writing data can be
fluctuating or there could be some bookkeeping operations running on the node such
as garbage collection blocking them to invest all the resources on the
computation.
• Accounting for such scenarios, whenever a MapReduce job is close to completion the
master schedules backup processes which perform the same computation as that of
the pending processes.
• The computation is marked as complete when either the original process or the
backup process notifies master about their computation. This way if any pending
process is a straggler, the backup process can finish the same execution and in
turn finish the overall MapReduce job. This way we invest some additional
computation resources but reduce the total time required for MapReduce jobs.
Resilient Distributed Datasets
Resilient Distributed Datasets (RDDs)
• Resilient Distributed Datasets (RDDs) are a fault-tolerant, in-memory abstraction for cluster computing,
designed by Spark to address MapReduce's inefficiency in iterative and interactive applications.
• RDDs allow data to be cached in memory, supporting fast recomputation of lost partitions through
lineage, rather than relying on slow disk-based replication.
• The RDD is the core abstraction used by Spark to work on data
• A RDD is a collection of elements partitioned in every cluster node. Spark operates in parallel on them
Every RDD is created from a file on Hadoop filesystem
• They can be made persistent in memory
Resilient Distributed Datasets (RDDs)
Key Concepts of RDDs: Restricted form of distributed shared memory
• Definition: Read-only, partitioned collections of records that can be operated on in parallel.
• Fault Tolerance (Resilience): Instead of replication, RDDs track "lineage" (the sequence of transformations used to build
a dataset) to recover lost data efficiently.
• In-Memory Performance: RDDs are designed to keep intermediate data in memory, making them 10x-100x faster than
MapReduce for iterative workloads.
• Dependencies:
• Narrow: Each partition of the parent RDD is used by at most one child partition (e.g., map, filter).
• Wide: Multiple child partitions use a single parent partition (e.g., groupByKey).
• Operations:
• Transformations: Lazy operations that define a new RDD (e.g., map, filter, join).
• Actions: Operations that return a result to the driver or write to storage (e.g., count, collect, save
Representing RDDs No need of check
pointing Checkpointing
• Apache Spark works by distributing data processing across a
cluster of computers using an in-memory, master-slave
architecture.
• A driver program creates a Directed Acyclic Graph (DAG) of
lazy transformations, which the cluster manager splits into
stages and tasks, executed in parallel by executors on data
partitions, enabling high-speed processing and fault
tolerance.
Core Architectural Components
•Driver Program: The central coordinator (SparkContext) that converts code into a DAG, schedules
tasks, and interacts with the cluster manager.
•Cluster Manager: Allocates resources across the cluster (e.g., YARN, Mesos, Standalone).
•Executors: Worker nodes that run tasks, store data in memory/disk, and return results to the driver.
•Resilient Distributed Datasets (RDDs): The fundamental data structure; immutable, partitioned
collections that allow parallel processing and fault tolerance via lineage.
How Spark Executes Tasks
[Link] Evaluation: Transformations (e.g., map, filter) are not executed immediately. Spark records them in a DAG,
optimizing the execution plan.
[Link] Trigger: An action (e.g., collect, count) triggers the DAG scheduler to convert the logical graph into physical
execution stages.
[Link] Scheduling: The DAG is split into stages based on shuffle boundaries. The Task Scheduler launches tasks on
executors.
[Link] Execution: Executors process data partitions in parallel, caching data in memory to speed up iterative
algorithms.
[Link] Tolerance: If a node fails, Spark uses the RDD lineage to recompute only the lost partition
Key Performance Factors
•In-Memory Processing: Processes data in RAM rather than reading from disk, making it up to 100x
faster.
•Data Partitioning: Data is split into smaller, manageable chunks (partitions) across nodes to
maximize parallelism.
•Optimizers: The Catalyst Optimiser (for DataFrames/SQL) optimizes query execution plan
•The Apache Spark framework uses a master-slave architecture that consists of a driver, which runs
as a master node, and many executors that run across as worker nodes in the cluster. Apache Spark
can be used for batch processing and real-time processing as well.
The Apache Spark framework uses a master-slave architecture that
consists of a driver, which runs as a master node, and many executors
that run across as worker nodes in the cluster.
Apache Spark can be used for batch processing and real-time
processing as well.
The Spark driver
The driver is the program or process responsible for coordinating the execution of the Spark application. It runs the
main function and creates the SparkContext, which connects to the cluster manager.
The Spark executors
Executors are worker processes responsible for executing tasks in Spark applications. They are launched on
worker nodes and communicate with the driver program and cluster manager. Executors run tasks concurrently
and store data in memory or disk for caching and intermediate storage.
The cluster manager
The cluster manager is responsible for allocating resources and managing the cluster on which the Spark
application runs. Spark supports various cluster managers like Apache Mesos, Hadoop YARN, and standalone
cluster manager.
sparkContext
SparkContext is the entry point for any Spark functionality. It represents the connection to a Spark cluster and can
be used to create RDDs (Resilient Distributed Datasets), accumulators, and broadcast variables. SparkContext
also coordinates the execution of tasks.
Task
A task is the smallest unit of work in Spark, representing a unit of computation that can be performed on a single
Fine-Tuning Shuffle Partitions in Apache Spark for Maximum
Efficiency
• Apache Spark’s shuffle partitions are critical in data processing, especially during operations like
joins and aggregations.
• Properly configuring these partitions is essential for optimizing performance.
Default Shuffle Partition Count
• By default, Spark sets the shuffle partition count to 200. While this may work for small datasets (less
than 20 GB), it is usually inadequate for larger data sizes.
Right-Sizing Shuffle Partitions
• To optimize performance, it’s crucial to determine the appropriate number of shuffle partitions.
Here’s a guideline:
Calculating Partition Count
[Link] the Largest Shuffle Stage: Determine the size of the largest shuffle
stage.
[Link] a Target Partition Size: Partition size should never go over 200 MB in
any case.
3. Calculate the partition Count.
Fine-Tuning Shuffle Partitions in Apache Spark for Maximum
Example 1: Efficiency
•Shuffle Data Size: 210 GB
•Target Partition Size: 128 MB
Calculate the number of partitions using the formula:
Partition Count = Stage Input Data (MB) / Target Partition Size (MB)