📘 Resource Management in Big Data

Processing Systems
Big Data processing systems operate on large-scale distributed clusters consisting of
hundreds or thousands of machines. Efficient resource management is crucial to ensure that
these systems run fast, reliably, fairly, and cost-effectively.

1️⃣ Types of Resource Management

🔹 Meaning of Resource Management
Resource management refers to the planning, allocation, scheduling, monitoring, and
optimization of computational resources such as:

●​ CPU cores​

●​ Main memory (RAM)​

●​ Storage (disk)​

●​ Network bandwidth​

across multiple Big Data jobs, users, and applications in a distributed environment.

🔸 Objectives of Resource Management

●​ Efficient utilization of cluster resources​

●​ Reduced job execution time​

●​ Fair sharing among users​

●​ High system throughput​

●​ Fault tolerance and reliability​

🔹 1. Static Resource Management
Explanation:

In static resource management, resources are allocated before job execution and remain
fixed throughout the job’s lifecycle.

Characteristics:

●​ Resource requirements are predefined​

●​ No changes during runtime​

●​ Simple and predictable execution​

Advantages:

●​ Easy to implement​

●​ Low scheduling overhead​

Limitations:

●​ Poor utilization if resources remain idle​

●​ Not suitable for dynamic workloads​

Example:

In Hadoop MapReduce, a fixed number of map and reduce slots are allocated to jobs.

🔹 2. Dynamic Resource Management

Explanation:
Dynamic resource management allocates resources on demand and can increase or
decrease resources based on workload changes during execution.

Characteristics:

●​ Adaptive and flexible​

●​ Responds to runtime conditions​

Advantages:

●​ Better utilization of resources​

●​ Improved performance for variable workloads​

Limitations:

●​ More complex to implement​

●​ Requires monitoring mechanisms​

Example:

Apache Spark dynamically allocates executors based on job requirements.

🔹 3. Centralized Resource Management

Explanation:

A central controller manages and schedules resources for the entire cluster.

Characteristics:

●​ Single authority for decision-making​

●​ Global view of resources​

Advantages:
 ●​ Simplified management​

●​ Efficient global scheduling​

Limitations:

●​ Single point of failure​

●​ Scalability limitations​

Example:

YARN ResourceManager controls resource allocation in Hadoop clusters.

🔹 4. Distributed Resource Management

Explanation:

Resource management decisions are distributed across multiple nodes, rather than a single
controller.

Characteristics:

●​ No single point of control​

●​ Nodes collaborate for scheduling​

Advantages:

●​ High scalability​

●​ Fault tolerance​

Limitations:

●​ Coordination overhead​
 ●​ Complex implementation​

Example:

Apache Mesos distributes resource management across the cluster.

🔹 5. Fair Scheduling
Explanation:

Fair scheduling ensures that all users and jobs receive an equal share of cluster resources
over time.

Characteristics:

●​ Prevents job starvation​

●​ Balances short and long jobs​

Advantages:

●​ Ensures fairness​

●​ Suitable for multi-user environments​

Limitations:

●​ Critical jobs may experience delays​

Example:

YARN Fair Scheduler in Hadoop.

🔹 6. Capacity-Based Scheduling
Explanation:

Cluster resources are divided into predefined queues, each with guaranteed minimum
capacity.

Characteristics:

●​ Queue-based allocation​

●​ Priority-aware scheduling​

Advantages:

●​ Guarantees resources for important jobs​

●​ Predictable performance​

Limitations:

●​ May underutilize resources if queues are idle​

Example:

YARN Capacity Scheduler.

2️⃣ Big Data Processing Systems and Platforms

Big Data processing systems provide frameworks and platforms that execute large-scale
computations efficiently using distributed resources.

🔹 1. Batch Processing Systems

Explanation:

Batch processing systems process large volumes of data collected over time in batches.
Characteristics:

●​ High latency​

●​ High throughput​

●​ Suitable for historical data​

Advantages:

●​ Efficient for large datasets​

●​ Simple programming model​

Limitations:

●​ Not suitable for real-time analytics​

Examples:

●​ Hadoop MapReduce​

●​ Apache Hive​

●​ Apache Pig​

Use Cases:

●​ Log analysis​

●​ Data warehousing​

●​ Report generation​

🔹 2. Stream Processing Systems

Explanation:
Stream processing systems analyze data continuously as it arrives.

Characteristics:

●​ Low latency​

●​ Event-driven processing​

Advantages:

●​ Real-time insights​

●​ Immediate response​

Limitations:

●​ More complex programming​

●​ Requires continuous resources​

Examples:

●​ Apache Flink​

●​ Apache Spark Streaming​

●​ Apache Storm​

Use Cases:

●​ Fraud detection​

●​ Real-time monitoring​

●​ IoT analytics​

🔹 3. Hybrid Processing Systems

Explanation:

Hybrid systems support both batch and stream processing within the same framework.

Characteristics:

●​ Unified processing model​

●​ Flexible execution​

Advantages:

●​ Reduced system complexity​

●​ Reusable code​

Limitations:

●​ Higher learning curve​

Examples:

●​ Apache Spark​

●​ Apache Flink​

●​ Google Dataflow​

🔹 4. Resource Management Platforms

📌 Apache YARN
●​ Separates resource management from computation​

●​ Supports multiple processing engines​

Components:
 ●​ ResourceManager​

●​ NodeManager​

●​ ApplicationMaster​

📌 Apache Mesos
●​ General-purpose cluster manager​

●​ Supports diverse workloads​

📌 Kubernetes
●​ Container-based orchestration​

●​ Dynamic scaling and resource isolation​

📌 Spark Standalone
●​ Built-in lightweight cluster manager for Spark​

🔥 COMPARISON
Feature Batch Stream
Systems Systems

Data Type Historical Real-time

 Latency High Low

Execution Periodic Continuous

Example Hadoop Flink

📘 Resource Management in the Cloud

Cloud computing provides on-demand, scalable resources such as CPU, memory, storage,
and network over the internet. Efficient resource management is essential to ensure
performance, cost efficiency, fairness, and Service Level Agreement (SLA) compliance.

1️⃣ Single-Resource Management in the Cloud

🔹 Meaning
Single-resource management focuses on the allocation, scheduling, and optimization of
one type of resource at a time, such as:

●​ CPU
●​ Memory
●​ Storage
●​ Network bandwidth

Each resource is managed independently, without considering its interaction with other
resources.

🔹 Why Single-Resource Management is Needed

●​ Simpler cloud architectures initially focused on CPU or memory
●​ Many applications are CPU-bound or memory-bound
●​ Easier monitoring and control
 ●​ Useful for basic cloud services and legacy applications

🔹 Types of Single-Resource Management

1️⃣ CPU Resource Management

Explanation:

CPU management allocates processing power (cores or cycles) to virtual machines (VMs) or
containers.

Techniques:

●​ Time slicing
●​ CPU pinning
●​ Priority-based scheduling

Advantages:

●​ Ensures fair CPU sharing

●​ Prevents CPU starvation

Example:

AWS EC2 assigns virtual CPUs (vCPUs) to instances.

2️⃣ Memory Resource Management

Explanation:

Manages RAM allocation to cloud workloads.

Techniques:

●​ Static memory allocation

●​ Memory ballooning
●​ Memory overcommitment

Advantages:

●​ Improves VM density
 ●​ Prevents memory wastage

Example:

Hypervisors dynamically adjust VM memory usage.

3️⃣ Storage Resource Management

Explanation:

Controls disk space and I/O bandwidth allocation.

Techniques:

●​ Quotas
●​ Tiered storage (hot, warm, cold)
●​ Replication strategies

Example:

Cloud storage services like Amazon S3.

4️⃣ Network Resource Management

Explanation:

Manages bandwidth and latency among cloud applications.

Techniques:

●​ Traffic shaping
●​ Bandwidth throttling
●​ Load balancing

🔹 Advantages of Single-Resource Management

●​ Simple to design and implement
●​ Low computational overhead
●​ Easy monitoring and debugging
🔹 Limitations of Single-Resource Management
●​ Ignores resource interdependencies
●​ Can lead to resource imbalance
●​ Inefficient for modern multi-tier applications

2️⃣ Multi-Resource Management in the Cloud

🔹 Meaning
Multi-resource management involves the joint allocation and optimization of multiple
resources simultaneously, such as:

●​ CPU
●​ Memory
●​ Storage
●​ Network

It treats cloud workloads as multi-dimensional resource consumers.

🔹 Need for Multi-Resource Management

Modern cloud applications:

●​ Are microservice-based
●​ Have varying resource demands
●​ Require coordinated resource allocation

Managing only one resource often causes:

●​ Bottlenecks
●​ SLA violations
●​ Poor utilization

🔹 Key Characteristics
●​ Resource coupling awareness
●​ Dynamic and adaptive allocation
●​ SLA-driven decisions
 ●​ Cost-aware optimization

🔹 Techniques in Multi-Resource Management

1️⃣ Vector-Based Resource Allocation

Explanation:

Resources are represented as vectors (CPU, memory, disk, network).

Advantage:

●​ Balances all resources simultaneously

Example:

Dominant Resource Fairness (DRF)

2️⃣ Dominant Resource Fairness (DRF)

Explanation:

Ensures fairness by allocating resources based on the dominant resource usage of each
application.

Advantage:

●​ Prevents monopolization of any resource

Used In:

Apache Mesos

3️⃣ SLA-Aware Resource Allocation

Explanation:

Allocates resources based on performance guarantees defined in SLAs.

Example:

Autoscaling resources when response time increases.

4️⃣ Predictive Resource Management

Explanation:

Uses historical data and machine learning to predict future resource needs.

Advantage:

●​ Prevents over-provisioning and under-provisioning

🔹 Advantages of Multi-Resource Management

●​ Higher resource utilization
●​ Better application performance
●​ Reduced cost
●​ Improved SLA compliance

🔹 Challenges
●​ High complexity
●​ Increased monitoring overhead
●​ Requires advanced algorithms

3️⃣ Related Work on Resource Management

🔹 Meaning
Related work refers to research, models, frameworks, and systems developed to improve
cloud resource management.

🔹 Key Research Areas

1️⃣ Virtual Machine (VM) Placement

Explanation:

Determines optimal placement of VMs on physical hosts.

Goal:

●​ Minimize energy consumption

●​ Avoid hotspots

2️⃣ Energy-Aware Resource Management

Explanation:

Focuses on reducing power consumption in data centers.

Techniques:

●​ VM consolidation
●​ Dynamic Voltage and Frequency Scaling (DVFS)

3️⃣ Container-Based Resource Management

Explanation:

Manages lightweight containers instead of VMs.

Platforms:

●​ Kubernetes
●​ Docker Swarm

4️⃣ Market-Based Resource Allocation

Explanation:

Resources are allocated using pricing and bidding mechanisms.

Example:

Spot instances in cloud platforms.

5️⃣ AI-Driven Resource Management

Explanation:

Uses machine learning and reinforcement learning for adaptive decisions.

Advantage:

●​ Self-optimizing cloud systems

🔥 KEY DIFFERENCE (EXAM-IMPORTANT)

Aspect Single-Resource Multi-Resource

Focus One resource Multiple resources

Complexity Low High

Efficiency Moderate High

Suitability Simple workloads Modern cloud apps

📘 Local Resource Consumption Shaping

A Case for MapReduce

1️⃣ Background: Why Local Resource Consumption

Shaping?
In Big Data processing frameworks like MapReduce, jobs run on shared cluster nodes and
consume multiple local resources such as:
 ●​ CPU
●​ Memory
●​ Disk I/O
●​ Network bandwidth

Problem:

Traditional MapReduce schedulers focus mainly on:

●​ Job-level scheduling
●​ Cluster-wide resource allocation

They do not effectively control resource usage at the local (node) level, which can lead to:

●​ Resource contention
●​ Disk I/O bottlenecks
●​ CPU starvation
●​ Unpredictable job performance

Solution:

Local Resource Consumption Shaping (LRCS) controls how much each MapReduce task
consumes local resources, ensuring balanced and predictable execution.

2️⃣ Local Resource Consumption Shaping (LRCS)

🔹 Definition
Local Resource Consumption Shaping is a technique that limits, regulates, and shapes the
resource usage of tasks at individual nodes during execution.

It ensures that no single task monopolizes local resources.

3️⃣ A Case for MapReduce

MapReduce is an ideal case for LRCS because:

●​ Tasks run in parallel on shared nodes

●​ Map and Reduce phases have different resource demands
●​ Disk and network I/O are heavily used
 ●​ Multiple jobs may run simultaneously

Typical Resource Usage in MapReduce

Phase Resource
Intensive

Map CPU, Disk I/O

Shuffle Network, Disk I/O

Reduc CPU, Memory

e

Without shaping, these phases may interfere with each other.

4️⃣ Local Resource Shaper

🔹 What is a Local Resource Shaper?
A Local Resource Shaper is a system component deployed on each worker node that:

●​ Monitors resource usage

●​ Enforces resource limits
●​ Dynamically adjusts task behavior

🔹 Functions of Local Resource Shaper

1️⃣ Monitoring

The shaper continuously monitors:

●​ CPU utilization
●​ Disk throughput
●​ Memory usage
●​ Network I/O

This data is collected at fine-grained time intervals.

2️⃣ Enforcement

The shaper applies resource usage caps, such as:

●​ CPU throttling
●​ Disk I/O rate limiting
●​ Network bandwidth control

This prevents resource monopolization.

3️⃣ Dynamic Adjustment

Based on current load:

●​ Resources are increased or decreased

●​ Task execution speed is adjusted
●​ Priority tasks receive more resources

4️⃣ Isolation

Ensures that:

●​ One MapReduce job does not degrade others

●​ Background system tasks are not starved

🔹 Architecture of Local Resource Shaper

Components:

●​ Resource Monitor
●​ Control Policy Engine
●​ Enforcement Module
●​ Feedback Loop

Working:

1.​ Monitor measures resource usage

2.​ Policy engine decides limits
3.​ Enforcement module applies shaping
4.​ Feedback loop adjusts policies
5️⃣ Types of Resource Shaping

1️⃣ CPU Shaping

●​ Limits CPU cycles per task

●​ Prevents CPU-hogging reducers

2️⃣ Disk I/O Shaping

●​ Controls read/write bandwidth

●​ Prevents I/O congestion during shuffle phase

3️⃣ Network Shaping

●​ Regulates shuffle traffic

●​ Avoids network saturation

4️⃣ Memory Shaping

●​ Restricts memory footprint

●​ Reduces garbage collection overhead

6️⃣ Benefits of Local Resource Shaping in MapReduce

●​ Improved performance predictability
●​ Reduced job interference
●​ Better cluster utilization
●​ Fair resource sharing
●​ Increased throughput
7️⃣ Evaluation of Local Resource Consumption Shaping
🔹 Purpose of Evaluation
Evaluation measures:

●​ Effectiveness of resource shaping

●​ Performance improvement
●​ Overhead introduced

🔹 Evaluation Metrics
1️⃣ Job Completion Time

●​ Measures total execution time

●​ Shaping reduces delays caused by contention

2️⃣ Resource Utilization

●​ CPU, disk, and network utilization

●​ Balanced utilization indicates success

3️⃣ Throughput

●​ Number of jobs completed per unit time

●​ Increases due to reduced interference

4️⃣ Fairness

●​ Ensures equal resource access

●​ Prevents starvation

5️⃣ Overhead
 ●​ CPU and memory overhead of shaper
●​ Should be minimal

🔹 Experimental Setup
●​ Cluster running MapReduce jobs
●​ Compare execution with and without LRCS
●​ Multiple concurrent workloads

🔹 Evaluation Results (Typical Findings)

Aspect Without Shaping With Shaping

Job Time Unpredictable Stable

Disk I/O Congested Smooth

Fairness Poor High

Throughput Lower Higher

8️⃣ Challenges and Limitations

●​ Increased system complexity
●​ Requires accurate monitoring
●​ Fine-grained control may add overhead
●​ Policy tuning is difficult

**System Optimization for Big Data

Processing:
Basic Framework of the Hadoop Ecosystem**
1️⃣ Introduction to System Optimization in Big Data
System optimization in big data processing focuses on improving:

●​ Performance
●​ Scalability
●​ Fault tolerance
●​ Resource utilization

The Hadoop Ecosystem provides a distributed framework designed to process large-scale

data efficiently using commodity hardware.

2️⃣ Hadoop Ecosystem: Overview

The Hadoop Ecosystem is a collection of open-source tools and frameworks that support:

●​ Distributed storage
●​ Parallel data processing
●​ Resource management
●​ Data access and analytics

It follows the principle of:

“Move computation to data, not data to computation.”

3️⃣ Basic Framework of the Hadoop Ecosystem

The Hadoop ecosystem can be broadly divided into four layers:

3.1 Storage Layer – HDFS (Hadoop Distributed File

System)
🔹 Purpose
HDFS provides reliable, scalable, and distributed storage for big data.
🔹 Key Components
1️⃣ NameNode

●​ Master node
●​ Stores metadata (file names, block locations)
●​ Manages namespace

2️⃣ DataNode

●​ Worker nodes
●​ Store actual data blocks
●​ Perform read/write operations

🔹 Optimization Features
●​ Data replication (fault tolerance)
●​ Large block size (reduces overhead)
●​ Data locality (improves performance)

3.2 Resource Management Layer – YARN

🔹 Purpose
YARN (Yet Another Resource Negotiator) manages cluster resources and schedules jobs.

🔹 Key Components
1️⃣ ResourceManager

●​ Global resource allocator

●​ Manages cluster-wide resources

2️⃣ NodeManager

●​ Runs on each node

●​ Manages node-level resources

3️⃣ ApplicationMaster
 ●​ Manages application lifecycle
●​ Requests resources from ResourceManager

🔹 Optimization Benefits
●​ Efficient resource utilization
●​ Supports multiple processing models
●​ Dynamic resource allocation

3.3 Processing Layer – MapReduce

🔹 Purpose
MapReduce enables parallel data processing over distributed datasets.

🔹 Phases of MapReduce
1️⃣ Map Phase

●​ Processes input data

●​ Produces key–value pairs

2️⃣ Shuffle and Sort

●​ Transfers data across nodes

●​ Groups data by key

3️⃣ Reduce Phase

●​ Aggregates results
●​ Produces final output

🔹 Optimization Techniques
●​ Combiner functions
●​ Data locality
●​ Parallel execution
 ●​ Fault recovery via re-execution

3.4 Data Access and Analytics Layer

This layer provides high-level tools for data querying, analysis, and workflow management.

🔹 Major Components
1️⃣ Hive

●​ SQL-like querying (HiveQL)

●​ Used for data warehousing

2️⃣ Pig

●​ High-level data flow language (Pig Latin)

●​ Simplifies MapReduce programming

3️⃣ HBase

●​ NoSQL column-oriented database

●​ Provides real-time read/write access

4️⃣ Sqoop

●​ Transfers data between RDBMS and HDFS

5️⃣ Flume

●​ Collects and ingests streaming data

4️⃣ Hadoop Ecosystem Architecture Diagram (Conceptual)

+-------------------------------------+
| Data Access & Analytics Layer |
| Hive | Pig | HBase | Sqoop | Flume |
+-------------------------------------+
| Processing Layer |
| MapReduce |
+-------------------------------------+
| Resource Management Layer |
| YARN |
+-------------------------------------+
| Storage Layer |
| HDFS |
+-------------------------------------+

5️⃣ Role of Hadoop Ecosystem in System Optimization

Aspect Optimization Achieved

Storage Fault tolerance, scalability

Processin Parallelism, speed

g

Resource Efficient utilization

Analytics Ease of use, abstraction

6️⃣ Advantages of Hadoop Ecosystem

●​ Horizontal scalability
●​ Fault tolerance
●​ Cost-effective (commodity hardware)
●​ Handles structured & unstructured data
●​ High throughput

7️⃣ Limitations of Hadoop Ecosystem

●​ High latency (batch processing)
●​ Not suitable for real-time analytics
●​ Complex configuration
●​ Security overhead

1️⃣ Parallel Computation Framework

A parallel computation framework enables the execution of large-scale data processing tasks
by dividing work into smaller tasks and executing them simultaneously across multiple
nodes in a distributed system.

Hadoop provides such a framework using MapReduce with HDFS and YARN.

2️⃣ MapReduce
🔹 Definition
MapReduce is a distributed programming model used to process large datasets in parallel
across a cluster of machines.

🔹 Working Principle
The core idea is:

Split data → Process in parallel → Aggregate results

🔹 Phases of MapReduce
1️⃣ Input Splitting

●​ Input data stored in HDFS is split into blocks

●​ Each block is processed independently

2️⃣ Map Phase

●​ Mapper processes input records

●​ Produces intermediate key-value pairs

Example:​
(word, 1)
3️⃣ Shuffle and Sort

●​ Intermediate data is transferred across nodes

●​ Data is grouped by key

4️⃣ Reduce Phase

●​ Reducer aggregates values for each key

●​ Produces final output

🔹 Features
●​ Automatic parallelism
●​ Fault tolerance
●​ Data locality
●​ Scalability

🔹 Limitations
●​ High latency
●​ Not suitable for iterative algorithms
●​ Disk-based processing

3️⃣ Job Scheduling of Hadoop

🔹 Meaning
Job scheduling determines when and where MapReduce jobs run on a cluster.

Hadoop uses YARN for job scheduling and resource management.

🔹 Goals of Job Scheduling

●​ Maximize cluster utilization
●​ Ensure fairness
 ●​ Meet SLAs
●​ Reduce job completion time

🔹 Types of Hadoop Schedulers

1️⃣ FIFO Scheduler

●​ Jobs executed in submission order

●​ Simple but inefficient

Limitation:​
Small jobs may wait behind large jobs

2️⃣ Fair Scheduler

●​ Resources shared fairly among jobs

●​ Supports job pools

Advantage:​
Improved response time

3️⃣ Capacity Scheduler

●​ Divides cluster into queues

●​ Each queue gets guaranteed capacity

Used in:​
Large multi-tenant clusters

🔹 Scheduling Factors
●​ Resource availability
●​ Job priority
●​ Data locality
●​ Node health
4️⃣ Performance Optimization of HDFS
🔹 HDFS Overview
HDFS is designed for high-throughput access to large files.

🔹 Optimization Techniques
1️⃣ Block Size Tuning

●​ Larger blocks reduce metadata overhead

●​ Improves sequential reads

2️⃣ Replication Factor Adjustment

●​ Default replication = 3
●​ Lower replication improves write speed
●​ Higher replication improves read performance

3️⃣ Data Locality

●​ Moves computation close to data

●​ Reduces network traffic

4️⃣ HDFS Balancer

●​ Distributes data evenly across nodes

●​ Prevents hotspots

5️⃣ Compression
 ●​ Reduces storage and I/O
●​ Improves performance

5️⃣ Performance Optimization of HBase

🔹 HBase Overview
HBase is a NoSQL, column-oriented database built on top of HDFS, supporting real-time
read/write access.

🔹 Optimization Techniques
1️⃣ Region Design

●​ Proper region size improves load distribution

●​ Avoids region server overload

2️⃣ Row Key Design

●​ Avoid sequential keys

●​ Prevents hotspotting

3️⃣ MemStore Tuning

●​ Larger MemStore reduces disk flush frequency

4️⃣ Block Cache Configuration

●​ Improves read performance

●​ Keeps frequently accessed data in memory
5️⃣ Compaction Management

●​ Minor and major compactions

●​ Reduces storage fragmentation

6️⃣ Comparison Summary (Exam-Friendly)

Component Purpose Optimization Focus

MapReduce Parallel processing Task parallelism

Job Resource Fairness & utilization

Scheduling allocation

HDFS Distributed storage Throughput

HBase Real-time database Low latency

7️⃣ Advantages of Hadoop Parallel Framework

●​ Scalable
●​ Fault tolerant
●​ Cost effective
●​ Handles big data efficiently
1️⃣ Diagram Explanation: Hadoop
Architecture (with Components)
📌 Hadoop Architecture Diagram (Conceptual)
+--------------------------------------------------+
| Data Access & Analytics Layer |
| Hive | Pig | HBase | Sqoop | Flume | Oozie |
+--------------------------------------------------+
| Processing Layer |
| MapReduce |
+--------------------------------------------------+
| Resource Management Layer |
| YARN |
| ResourceManager | NodeManager | AppMaster |
+--------------------------------------------------+
| Storage Layer |
| HDFS |
| NameNode | Secondary NN | DataNodes |
+--------------------------------------------------+

🔍 Explanation of Each Layer

1️⃣ Storage Layer – HDFS
HDFS (Hadoop Distributed File System) is responsible for storing large volumes of data
across multiple machines.

Components:

●​ NameNode​
Stores metadata (file names, block locations)
●​ DataNode​
Stores actual data blocks
●​ Secondary NameNode​
Helps in checkpointing metadata

Key Features:
 ●​ Fault tolerance via replication
●​ High throughput access
●​ Designed for large files

2️⃣ Resource Management Layer – YARN

YARN manages cluster resources and job scheduling.

Components:

●​ ResourceManager – allocates resources globally

●​ NodeManager – manages resources on each node
●​ ApplicationMaster – controls job execution

Role:

●​ Efficient resource sharing

●​ Supports multiple processing frameworks

3️⃣ Processing Layer – MapReduce

MapReduce enables parallel processing of large datasets.

Phases:

●​ Map → Shuffle → Reduce

Characteristics:

●​ Batch-oriented processing
●​ Disk-based computation
●​ Highly fault tolerant

4️⃣ Data Access & Analytics Layer

Provides high-level tools for easier data processing.
Tools:

●​ Hive – SQL-like queries

●​ Pig – data flow language
●​ HBase – real-time NoSQL DB
●​ Sqoop / Flume – data ingestion

✅ Diagram Summary (Exam Line)

The Hadoop architecture consists of layered components where HDFS provides
distributed storage, YARN manages resources, MapReduce performs parallel
computation, and higher-level tools support analytics and data access.

2️⃣ Hadoop vs Spark Comparison (Very

Important for Exams)
Feature Hadoop (MapReduce) Apache Spark

Processing Model Batch processing Batch + Real-time

Data Storage Disk-based In-memory

Speed Slower 10–100x faster

Iterative Inefficient Highly efficient

Processing

Fault Tolerance Data replication RDD lineage

Ease of Use Complex Developer-friendly

SQL Support Hive Spark SQL

Streaming Limited Spark Streaming

Machine Learning Mahout (limited) MLlib (rich)

Graph Processing Not efficient GraphX

🔍 Key Differences Explained
1️⃣ Performance

●​ Hadoop writes intermediate data to disk

●​ Spark processes data in memory​
✔ Spark is significantly faster

2️⃣ Processing Capability

●​ Hadoop → Only batch jobs

●​ Spark → Batch + streaming + ML + graph

3️⃣ Iterative Algorithms

●​ Hadoop re-reads data every iteration

●​ Spark caches data in memory

✔ Spark is ideal for ML algorithms

4️⃣ Fault Tolerance

●​ Hadoop → Replication
●​ Spark → Lineage (recomputes lost data)

📝 When to Use What?

Use Hadoop when:

●​ Data size is extremely large

●​ Cost is a concern
●​ Batch processing is sufficient

Use Spark when:

 ●​ Real-time or near real-time processing needed
●​ Iterative algorithms involved
●​ Performance is critical

1️⃣ Introduction: Big Data Replay on Multicore Systems

Modern big data systems generate large execution logs (task traces, I/O patterns, CPU
usage).​
Big Data Replay is the process of re-executing recorded workloads on another system to:

●​ Evaluate system performance

●​ Debug and test systems
●​ Compare hardware and software configurations

Since modern machines are multicore, efficient replay requires parallel execution while
preserving correctness.

2️⃣ Performance Bottlenecks in Big Data Replay

🔹 Meaning
Performance bottlenecks are factors that limit replay speed and scalability when replaying
big data workloads on multicore systems.

🔹 Major Performance Bottlenecks

1️⃣ CPU Bottlenecks

●​ Limited number of cores

●​ Context switching overhead
●​ Imbalanced task execution

Impact:​
Some cores remain idle while others are overloaded.
2️⃣ Disk I/O Bottlenecks

●​ Multiple replay tasks access disk simultaneously

●​ Random disk access patterns
●​ Limited disk bandwidth

Impact:​
Tasks block waiting for I/O completion.

3️⃣ Memory Bottlenecks

●​ Cache contention
●​ Memory bandwidth saturation
●​ Insufficient RAM for parallel replay

Impact:​
Increased latency and cache misses.

4️⃣ Synchronization Bottlenecks

●​ Locks and barriers between tasks

●​ Excessive thread coordination

Impact:​
Reduced parallelism.

5️⃣ Replay Overhead

●​ Log parsing
●​ Timestamp ordering
●​ Task dependency tracking

Impact:​
Replay slower than original execution.
3️⃣ The Replay Method
🔹 Definition
The Replay Method reproduces the original execution behavior of a big data workload using
recorded execution traces.

🔹 Goals of Replay
●​ Preserve task dependencies
●​ Maintain execution order where required
●​ Maximize parallel execution
●​ Achieve performance similarity

🔹 Steps in Replay Method

1️⃣ Trace Collection

●​ Logs collected during original execution

●​ Includes CPU, I/O, memory, and timing information

2️⃣ Dependency Reconstruction

●​ Identify task dependencies

●​ Build execution graph

3️⃣ Task Scheduling

●​ Assign replay tasks to cores

●​ Respect dependencies

4️⃣ Execution Replay

 ●​ Execute tasks on multicore system
●​ Monitor performance

🔹 Types of Replay
●​ Strict Replay: Exact timing and order
●​ Relaxed Replay: Logical correctness with flexibility

4️⃣ Packing Algorithms

🔹 Meaning
Packing Algorithms decide how replay tasks are grouped (packed) and assigned to cores
to maximize performance and resource utilization.

🔹 Why Packing is Needed

●​ Tasks have different resource demands
●​ Poor packing leads to contention
●​ Efficient packing improves throughput

5️⃣ Types of Packing Algorithms

1️⃣ Greedy Packing Algorithm

Explanation:

●​ Assigns tasks to the first available core

●​ Simple and fast

Advantage:

●​ Low scheduling overhead

Limitation:
 ●​ May cause load imbalance

2️⃣ Resource-Aware Packing Algorithm

Explanation:

●​ Considers CPU, memory, and I/O needs

●​ Avoids placing similar heavy tasks together

Advantage:

●​ Reduces contention

3️⃣ Time-Aware Packing Algorithm

Explanation:

●​ Considers task execution time

●​ Packs tasks with complementary durations

Advantage:

●​ Improves core utilization

4️⃣ Dependency-Aware Packing Algorithm

Explanation:

●​ Packs tasks based on dependency graph

●​ Ensures correctness

Advantage:

●​ Preserves execution semantics

5️⃣ Balanced Packing Algorithm

Explanation:
 ●​ Attempts uniform workload distribution
●​ Balances CPU and I/O

6️⃣ Comparison of Packing Algorithms

Algorithm Complexity Performance Awareness

Greedy Low Medium None

Resource-Aware Medium High CPU, I/O

Time-Aware Medium High Execution time

Dependency-Awar High Very High Task graph

e

7️⃣ Benefits of Packing Algorithms

●​ Increased replay throughput
●​ Reduced contention
●​ Better scalability on multicore systems
●​ More accurate performance evaluation

8️⃣ Challenges in Packing Algorithms

●​ High computational overhead
●​ Accurate resource profiling needed
●​ Trade-off between correctness and speed