 Key Features

Queue-based Resource Management

 Multiple Queues: Allows creation of multiple queues, each allocated a portion of total
cluster resources.

 Resource Allocation: Each queue is allocated a specific capacity of memory and CPU. No
single queue can monopolize the cluster.

Guaranteed Capacity

 Guaranteed Resource Allocation: Each queue gets a minimum guaranteed share of

resources.

 Dynamic Allocation: If a queue does not use its capacity, unused resources are given to
other queues.

 [Link]-based Resource Management

 Multiple Queues
“We can create different lines (queues) for different departments or teams. Each line gets
some part of the total resources.”

 Resource Allocation
“Each queue gets a fixed amount of memory and CPU. One queue cannot take all the system
resources.”

 2️.Guaranteed Capacity

 ✅ Guaranteed Resource Allocation

“Every queue is guaranteed a minimum share of resources. So, no one will get zero.”

 ✅ Dynamic Allocation

 “If one queue is not using its resources, other queues can use them temporarily.

 [Link]-based Resource Management

 Multiple Queues
“We can create different lines (queues) for different departments or teams. Each line gets
some part of the total resources.”

 Resource Allocation
“Each queue gets a fixed amount of memory and CPU. One queue cannot take all the system
resources.”

 2️.Guaranteed Capacity

 ✅ Guaranteed Resource Allocation

“Every queue is guaranteed a minimum share of resources. So, no one will get zero.”

 ✅ Dynamic Allocation

 “If one queue is not using its resources, other queues can use them temporarily.

Resource Pools
  Hierarchy and Sub-Queues: Queues can be organized hierarchically with sub-queues.

 Preemption: Lower-priority jobs can be stopped temporarily to give resources to higher-

priority queues.

Fairness and Load Balancing

 Capacity Guarantees: Each queue gets its guaranteed capacity.

 Fair Sharing: Extra resources are distributed based on queue size and needs.

 Resource Pools

 ✅ Hierarchy and Sub-Queues


“Queues can have sub-queues. For example, CSE queue can have AI, DS, and Cybersecurity
sub-queues.”

 ✅ Preemption

 “If a high-priority job needs resources, the system can stop a low-priority job temporarily
and give resources to the important one.”

 4️⃣ Fairness and Load Balancing

 ✅ Capacity Guarantees

 “Each queue will definitely get its assigned percentage of resources.”

 ✅ Fair Sharing

 “If extra resources are available, they are shared based on need.”

 “Capacity Scheduler divides resources into queues, guarantees minimum resources, allows
sharing of unused resources, and ensures fairness among departments.”

 Here is the easy way to explain Configuration in class 👇

 🔹 Configuration of Capacity Scheduler (Simple Explanation)

 ✅ Configured using [Link]

 👉 Explain like this:

“All the settings of Capacity Scheduler are written inside a file called capacity-
[Link].”

 “It is a configuration file where we define how much resource each queue should get.”

 🔹 Queue Configuration Properties

 1️⃣ [Link].<queue-name>.capacity

 👉 Meaning:
This tells how much percentage of total resources a queue will get.

 👉 Explain in class:
“If we write 40%, that queue will get 40% of total memory and CPU.”
  📌 Example:
If cluster has 100GB RAM and queue capacity is 40%,
It gets 40GB RAM.

 2️⃣ [Link].<queue-name>.maximum-capacity

 👉 Meaning:
This tells the maximum percentage a queue can use.

 👉 Explain in class:
“A queue may get extra resources if available, but it cannot cross this maximum limit.”

 📌 Example:
If maximum capacity is 60%,
Even if resources are free, it cannot use more than 60%.

 🎓 Very Simple Classroom Summary

 capacity → Minimum guaranteed percentage.

 maximum-capacity → Maximum limit it can use.

 If you want, I can also give a numerical example problem to solve in class.

 The default queue gets 50% of total cluster resources.

 The default queue can use up to 70% if extra resources are available.

 Marketing queue gets 30% of total resources.

 Marketing queue can use up to 40% if resources are free.

 capacity → Minimum guaranteed percentage.

 maximum-capacity → Maximum limit allowed.

 default and marketing → Queue names.

Advantages of Capacity Scheduler

1. Efficient use of resources

Cluster CPU and memory are used properly without wasting resources.

2. Supports many users

Multiple users or departments can run jobs on the same cluster.

3. Resource guarantee
Each queue gets a fixed minimum share of resources.

4. Flexible queue hierarchy

Queues can be divided into sub-queues for better organization.

5. Resource sharing
If one queue is not using resources, other queues can use them temporarily.

Disadvantages of Capacity Scheduler

 1. Complex configuration
Setting up queues and capacities is difficult for administrators.

2. Needs good planning

If queue capacities are not planned properly, resources may not be used efficiently.

3. Less flexibility
Compared to other schedulers, it is less flexible in dynamically adjusting resources.

4. Possible waiting time

Jobs in small queues may wait longer when the queue capacity is full.

Use Cases

• Multi-Tenant Clusters: Multiple teams sharing a cluster.

• Large Organizations: Departments with different resource needs.

• Balanced Resource Management: Combining guaranteed capacity and fairness.

• 1. Multi-Tenant Clusters

• Many teams or users can share the same Hadoop cluster.

• Example:
In a company, the Data Science team, Testing team, and Development team can run jobs
on the same cluster without disturbing each other.

• 2. Large Organizations

• Big organizations can give different resources to different departments.

• Example:

• Research department → more resources

• Testing department → fewer resources

• This helps each department run their work smoothly.

• 3. Balanced Resource Management

• Resources are shared properly and fairly among all queues.

• Example:
Each queue gets its minimum guaranteed resources, but if another queue is idle, those
resources can be temporarily used by others

• Ensures all running applications get a fair share of resources.

• If only one job is running → it uses all resources.

• If multiple jobs are running → resources are divided equally.

Fair Scheduler gives equal share of cluster resources to all running applications over time.

👉 Every job gets a fair chance to run.

Suppose three students share a computer. Each gets equal time. If one student leaves, the
remaining students get more time. This is exactly how the Fair Scheduler works.”

Fair Scheduler means giving equal chance to all jobs.”

When many jobs run in the cluster, the scheduler tries to share resources fairly among them.

Explain with a simple example

Imagine the cluster has 100 resources (CPU/memory units).

Job 1 running → gets 100 resources

Job 1 and Job 2 running → each gets 50 resources

Job 1, Job 2, Job 3 running → each gets about 33 resources

📌 Idea:
Resources are divided equally among running jobs.

Real-Life Example

 Think of 3 students sharing internet:

 Total speed = 30 Mbps

 3 students → Each gets 10 Mbps

 If 1 student leaves → 2 students get 15 Mbps each

 That is Fair Scheduling

 Resources are shared equally among all running jobs.

 If a job finishes, the remaining jobs get more resources.

 It supports priority, so important jobs can run faster.

 It also supports pools, which means grouping jobs and sharing resources among groups.

 Fair Scheduler in YARN allocates cluster resources equally among running applications. It
ensures that every job gets a fair share of resources over time and improves resource
utilization in multi-user environments.

 Fair Scheduler Advantages and Disadvantages (Simple Points)

Used in Apache Hadoop

 Advantages

 Fair resource sharing – All running jobs get equal resources.

 Better resource utilization – Cluster resources are used efficiently.

 Supports priority – Important jobs can run faster.

 Supports pools – Jobs can be grouped and managed easily.

 Disadvantages
 Configuration is complex – Harder to set up compared to simple schedulers.

 Not strict guarantee – Resources may change when new jobs arrive.

 Fair Scheduler in YARN allocates cluster resources equally among running applications. It
ensures that every job gets a fair share of resources over time and improves resource
utilization in multi-user environments.

 Fair Scheduler Advantages and Disadvantages (Simple Points)

Used in Apache Hadoop

 Advantages

 Fair resource sharing – All running jobs get equal resources.

 Better resource utilization – Cluster resources are used efficiently.

 Supports priority – Important jobs can run faster.

 Supports pools – Jobs can be grouped and managed easily.

 Disadvantages

 Configuration is complex – Harder to set up compared to simple schedulers.

 Not strict guarantee – Resources may change when new jobs arrive.

 [Link]
→ This file is used to configure the Fair Scheduler.

 [Link]
→ This property gives the path of the allocation file where resource sharing rules are
stored.

 🗣️Simple sentence to tell students

 “Fair Scheduler settings are stored in the [Link] file, and the allocation file
path tells Hadoop where the resource sharing rules are defined.”
“Configuration means settings. Fair Scheduler settings are stored in [Link], and
the allocation file decides how cluster resources are shared.”

 Example Configuration

 <property>
<name>[Link]</name>
<value>/etc/hadoop/conf/[Link]</value>
</property>

 🗣️Simple Explanation

 <property> → Defines a configuration setting.

 <name> → Name of the property used by the scheduler.

 <value> → Location of the [Link] file.

 🎓 One-line explanation for class

 “This property tells Hadoop where the [Link] file is located, which contains the
rules for sharing cluster resources.”

 ✅ Even shorter (5-second explanation):

“This configuration gives the path of the Fair Scheduler allocation file.”

 <allocations>

 <!-- Root element that defines resource allocation rules for all pools -->

 <pool name="default">

 <!-- Default pool for normal jobs -->

 <minResources>

 <!-- Minimum resources guaranteed for this pool -->

 <memory-mb>512</memory-mb>

 <!-- Minimum memory allocated = 512 MB -->

 <vcores>1</vcores>

 <!-- Minimum CPU cores allocated = 1 -->

 </minResources>

 <weight>1.0</weight>

 <!-- Weight determines share of resources compared to other pools -->

 </pool>

 <pool name="high-priority">

 <!-- Pool for high priority jobs -->

 <minResources>

 <!-- Minimum resources guaranteed for high priority jobs -->

 <memory-mb>1024</memory-mb>

 <!-- Minimum memory allocated = 1024 MB -->

 <vcores>2</vcores>

 <!-- Minimum CPU cores allocated = 2 -->

 </minResources>

 <weight>2.0</weight>

 <!-- Higher weight means this pool gets more resources -->

 </pool>

 </allocations>

 <!-- End of allocation configuration -->

  🗣️Simple explanation for students

 default pool → used for normal jobs

 high-priority pool → used for important jobs

 minResources → minimum CPU and memory guaranteed

 weight → decides how much resource share a pool gets

 ✅ One-line classroom explanation:

“Pools divide cluster resources, and each pool is given minimum resources and a weight to
control how much it can use.”

Use Cases

• Multi-tenant clusters

• Dynamic workloads

• Environments where fairness is important

Use Cases (Simple)

You can explain these very shortly and simply in class like this 👇

Use Cases of Fair Scheduler

1️⃣ Multi-Tenant Clusters

When many teams use the same cluster, resources are shared so one team cannot use everything.

2️⃣ Dynamic Workloads

When job needs change frequently, resources adjust automatically.

3️⃣ Fair Sharing

Every job gets a fair share of resources over time.

🎓 One-line explanation for students

“Fair Scheduler shares cluster resources fairly among multiple users and adjusts resources when
job needs change.” Delay Scheduling is a method used in distributed systems to improve
performance.

Instead of immediately assigning a task to any available resource,

The system waits for a short time to get a better resource (like a closer or more suitable machine).

Why is it useful?

Better performance: Tasks run faster when placed on the right machine.

Less network usage: It tries to run tasks where the data is already stored (data locality).

Better resource use: Reduces unnecessary load and conflicts.

In simple words:

Delay Scheduling means “Wait a little now, so the task can run better later.”
🔷 Fair Scheduler – Configuration Properties

Fair Scheduler is configured in:

📄 [Link]

🔑 Important Configuration Properties

1️⃣ Enable Fair Scheduler

<property> <name>[Link]</name>
<value>[Link]</value>
</property> 👉 Activates Fair Scheduler in YARN.

2️⃣ Allocation File Location

<property> <name>[Link]</name> <value>/etc/hadoop/fair-

[Link]</value> </property> 👉 Specifies the file where queue/pool configurations are
defined.

3️⃣ Maximum Allocation (Memory)

<property> <name>[Link]-allocation-mb</name> <value>8192</value>

</property> 👉 Maximum memory per container.

4️⃣ Maximum Allocation (vCores)

<property> <name>[Link]-allocation-vcores</name> <value>4</value>

</property> 👉 Maximum CPU cores per container.

5️⃣ Minimum Allocation (Optional)

<property> <name>[Link]-allocation-mb</name> <value>1024</value>

</property> 👉 Minimum memory per container.

📝 In Short (Exam Point)

Fair Scheduler is configured in [Link] using properties like:

[Link]

[Link]

Memory and vCore allocation limits

If you want, I can also give:

[Link] sample file

2-line short answer

5-mark structured answer 😊

Simple Visual Explanation

Without Delay Scheduling:

Task → Immediately assigned to Node B (Data moves over network ❌)
With Delay Scheduling:
Task → Wait → Assigned to Node A (Data local ✅)

Scheduler waits for a short time.

It tries to find a node where the required data is available.

If found → assigns task there.

FIFO

First job runs first.

Others must wait.

Very simple.

🔹 Capacity

Cluster divided into queues with fixed %.

Each department gets guaranteed resources.

🔹 Fair

All running jobs get equal share.

Small jobs finish faster.

Uses delay scheduling for better performance.

🎯 One-Line Difference

FIFO → Order based

Capacity → Percentage based

Fair → Equal sharing based

Key Concepts of Delay Scheduling (Simple & Short)

1. Fairness vs. Data Locality

Data Locality: Tasks run faster if they are executed on the machine where the data is stored.

Problem: Strict fairness may assign a task to any free machine, even if data is not there.

Solution: Delay scheduling waits for a short time to get a better machine with data.

Result: Balance between fairness and better performance.

2. Task Delay Mechanism

When a task is ready, it is not assigned immediately.

The scheduler waits for a short time to find a better machine.

If no better machine is found, it runs on any available machine.

This avoids long waiting and improves performance.

In simple words:
Wait a little for a better place, but don’t wait too long.

Key Concepts of Delay Scheduling

Fairness vs. Data Locality

 In distributed systems (like Hadoop or Spark), tasks perform better when they are scheduled
on machines where the required data is already stored (data locality).

 Sometimes strict fairness policies force a task to be scheduled on a machine that does not
have the necessary data locally, leading to inefficient resource usage and data transfers.

 Delay scheduling temporarily delays tasks to allow resources with better data locality to
become available, balancing fairness and performance.

Task Delay Mechanism

 When a task is ready to execute, instead of scheduling it immediately on any available

resource, the scheduler waits for a short period (delay) to check if a better resource (e.g., a
machine that holds the data) becomes available.

 If no better resource becomes available after the delay, the task is scheduled on any
available resource to avoid starvation.

Delay Scheduling

Delay Scheduling improves resource allocation by delaying task assignment briefly to achieve better
data locality.

Key Concepts

Fairness vs Data Locality

• Tasks perform better when executed where data is stored.

• Strict fairness may ignore data location.

• Delay scheduling waits briefly for better placement.

Task Delay Mechanism

• Scheduler waits for a short time to find a better node.

• If none found, task runs on any available node.

• You can explain this in short and simple words for students like this 👇Advantages

• Improved Data Locality: By waiting for a short delay, tasks are more likely to be scheduled
on nodes where their required data resides, reducing network traffic and improving
performance.

• Better Resource Utilization: Delay scheduling avoids underutilization of resources by

ensuring tasks are scheduled on optimal nodes when possible.

• Fairness: The system ensures that no task waits indefinitely for the perfect resource; tasks
eventually get assigned to any available node if necessary.
 • Use Cases:

• Big Data Frameworks (Hadoop, Spark): Delay scheduling is frequently used in systems that
need to balance the trade-off between data locality and resource fairness.

• Cloud Environments: In multi-tenant cloud systems, delay scheduling helps in efficiently

allocating shared resources among competing tasks.

• Delay Scheduling

• Delay Scheduling improves resource use by waiting a short time to run a task on the node
where the data is stored.

• Key Concepts

• 1️⃣ Fairness vs Data Locality

• Tasks run faster when they run where the data is located.

• Strict fairness may ignore data location.

• So the scheduler waits a little to find a better node.

• 2️⃣ Task Delay Mechanism

• The scheduler waits for a short time to find the correct node.

• If not found, the task runs on any available node.

• 🎓 One-line explanation for class

• “Delay Scheduling waits briefly so a task can run on the node where its data is stored,
improving performance.”

Advantages

Improved Data Locality: By waiting for a short delay, tasks are more likely to be scheduled on nodes
where their required data resides, reducing network traffic and improving performance.

Better Resource Utilization: Delay scheduling avoids underutilization of resources by ensuring tasks
are scheduled on optimal nodes when possible.

Fairness: The system ensures that no task waits indefinitely for the perfect resource; tasks
eventually get assigned to any available node if necessary.

 Use Cases:

Big Data Frameworks (Hadoop, Spark): Delay scheduling is frequently used in systems that need to
balance the trade-off between data locality and resource fairness.

Cloud Environments: In multi-tenant cloud systems, delay scheduling helps in efficiently allocating
shared resources among competing tasks.

 Improved Data Locality: By waiting briefly, tasks are more likely to run where their required
data resides, reducing network traffic and improving performance.

 Better Resource Utilization: Avoids underutilization by scheduling tasks on optimal nodes

when possible.
  Fairness: Ensures that no task waits indefinitely; tasks are eventually assigned to available
nodes.

Usecase:

o Big Data frameworks like Hadoop and Spark use delay scheduling to balance data
locality and fairness.

o In cloud environments, it helps efficiently allocate shared resources among

competing tasks.

You can explain these points in short and simple words for students like this 👇

Advantages of Delay Scheduling

1️⃣ Improved Data Locality

Tasks run on the node where the data is stored, so network traffic reduces and performance
improves.

2️⃣ Better Resource Utilization

Resources are used more efficiently by assigning tasks to suitable nodes.

3️⃣ Fairness
Tasks do not wait forever; if the correct node is not found, they run on any available node.

Use Cases

1️⃣ Big Data Frameworks

Used in systems like Hadoop and Spark to balance data locality and fairness.

2️⃣ Cloud Environments

In cloud systems with many users, it helps share resources efficiently.

🎓 One-line classroom explanation

“Delay Scheduling waits briefly to run tasks on the node where data exists, improving
performance while still maintaining fairness.”

You can explain it very short and simple in class like this 👇

Advantages

Improved Data Locality → Reduces network traffic.

Better Resource Utilization → Uses the best nodes.

Fairness → Tasks do not wait too long.

Use Cases

Big Data frameworks like Hadoop and Spark

Cloud environments

Systems needing both fairness and good performance

🎓 One-line explanation
“Delay Scheduling improves performance by running tasks on the best node while still keeping
fairness.”

You can explain this example in very simple and short words like this 👇

Example (Delay Scheduling)

A task needs data from Machine A.

But Machine A is busy.

The scheduler waits for a short time.

👉 If Machine A becomes free, the task runs there for better performance.
👉 If not, the task runs on another available machine.

🎓 One-line explanation for class

“Delay Scheduling waits briefly for the correct node; if it’s still busy, the task runs on another
node.”

Dominant Resource Fairness (DRF)

 DRF is a method to share resources fairly in systems with multiple resources (CPU, memory,
disk, network).

 It makes sure no user or job gets more than its fair share of the most used resource
(dominant resource).

 It works well when different jobs need different types of resources.

 It is used in big data systems like Hadoop, Spark, and Mesos.

 It helps multiple users share the same cluster fairly.

 You can explain Dominant Resource Fairness (DRF) to students in short and simple words
like this 👇

 Dominant Resource Fairness (DRF)

 DRF is a method to share cluster resources fairly when there are multiple resources like
CPU, memory, disk, and network.

 It ensures no user or job uses more than its fair share of the most used resource (called the
dominant resource).

 It works well when different jobs need different resources.

 Used in big data systems like Apache Hadoop, Apache Spark, and Apache Mesos.

 Helps multiple users share the same cluster fairly.

 🎓 One-line explanation for class

 “DRF shares CPU, memory, and other resources fairly so that no job uses more than its fair
share.”

Key Concepts of Dominant Resource Fairness

Multi-Resource Environments

 In big data analytics, tasks require combinations of CPU, memory, storage, and network
bandwidth.

 Traditional fairness approaches (like CPU-centric fairness) do not consider multiple resource
types, which may lead to inefficient or unfair allocation.

Dominant Resource

 The dominant resource of a job is the resource it demands the most relative to the total
available in the system.

 Example: If a job requires 60% of memory but only 10% of CPU, its dominant resource is
memory.

 DRF allocates resources so that each user’s share of their dominant resource is as fair as
possible.

Fairness Definition in DRF

 DRF extends max-min fairness to multiple resources.

 The system attempts to maximize the minimum dominant resource share among all users.

 No user can increase their allocation without decreasing another user's dominant resource
share.

1. Multi-Resource Environment

 In big data systems, jobs need many resources like CPU, memory, disk, and network.

 Old methods look at only one resource (like CPU), which is not fully fair.

 DRF considers all resources together.

2. Dominant Resource

 A job’s dominant resource is the one it uses the most compared to others.

 Example: If a job uses 60% memory and 10% CPU, memory is its dominant resource.

 DRF makes sure this dominant resource is shared fairly among users.

3. Fairness in DRF

 DRF tries to give every user a fair share of their most needed resource.

 It balances resources so no user gets too much compared to others.

 One user cannot increase their share without reducing someone else’s share.

 How DRF Works – Simple Explanation

 Allocation Based on Dominant Resource:

Each job gets resources based on the resource it needs the most (its dominant resource).
 Priority to Lowest Share:
The system gives resources first to the user who is using the least share of their dominant
resource.

 Step-by-Step Allocation:
Resources are given little by little, and the system keeps checking to make sure everyone is
getting a fair share.

 Example of DRF – Simple Explanation

 Suppose a cluster has CPU and Memory resources.

 There are two users: User A and User B.

 User Requirements:

 User A needs more memory (40%) and less CPU (20%) → So, memory is dominant for User
A.

 User B needs more CPU (60%) and less memory (10%) → So, CPU is dominant for User B.

 How DRF Handles This:

 DRF checks the dominant resource of each user.

 It makes sure User A gets a fair share of memory.

 It makes sure User B gets a fair share of CPU.

 No user can take too much of their dominant resource.

 👉 So, both users get fair access, and no one can control the whole cluster.

 Benefits of DRF in Big Data Analytics (Simple & Short)

 1. Better Resource Use

 DRF checks CPU, memory, etc., together.

 It avoids wasting any single resource.

 2. Fairness for All Jobs

 Jobs needing different resources get fair allocation.

 No job is ignored or starved.

 3. Stops Resource Hoarding

 No user can take too much CPU or memory.

 Everyone gets a fair share.

 4. Higher System Performance

 Resources are shared properly.

 More jobs can run at the same time without overload.

 Benefits of DRF in Big Data Analytics:

 Better Resource Utilization:

 By considering multiple resource types, DRF ensures that no single resource (like CPU or
memory) becomes a bottleneck, leading to more efficient utilization of the cluster.

 Fairness Across Multiple Resource Types:

 DRF provides a fairer allocation of resources, especially in environments where jobs have
different resource demands. This avoids scenarios where some jobs starve while others
over-utilize certain resources.

 Prevents Resource Hoarding:

 DRF prevents users or tasks from monopolizing a specific resource, ensuring that all users
have fair access to their most critical resources.

 Maximizes Throughput:

 DRF allows for efficient sharing of cluster resources, which can improve the overall
throughput of the system, allowing more jobs to be processed concurrently without
overloading any single resource.

 Applications in Big Data Frameworks (Simple Explanation)

 1. Apache Mesos

 Uses DRF to share cluster resources fairly among users and applications.

 2. Hadoop YARN

 Capacity Scheduler can use DRF to give fair resource allocation to multiple users.

 3. Apache Spark

 When Spark runs on shared clusters, DRF helps in fair sharing of CPU and memory.

 Challenges (Simple Explanation)

 1. Complex Implementation

 DRF is harder to set up because it must monitor many resources (CPU, memory, etc.)
continuously.

 2. Performance Overhead

 Checking and adjusting fairness regularly can slightly reduce system performance.

 3. Workload Tuning

 Different types of jobs need different settings, so administrators must carefully adjust the
configuration.