Assignment -1
Database Management System
1. Write a short note on short-circuit reading in HDFS. How does it improve the
reading performance?
Short-circuit reading in HDFS allows clients to directly read data from DataNode's
local disk instead of through the DataNode process. This bypasses the network
stack, reducing latency and improving throughput. It enhances performance by
minimizing data transfer overhead and leveraging local disk I/O.
2. How are resources allocated to applications in YARN?
YARN allocates resources to applications using the ResourceManager and
NodeManagers. The ResourceManager has a Scheduler that assigns resources
based on memory, CPU, and priority. Applications request resources through
Application Masters, which manage application lifecycle and interact with the
Scheduler to acquire containers for execution.
3. What is a container in YARN, and what resources does it manage?
A container in YARN is an isolated environment where an application runs. It
manages resources such as CPU, memory, and network bandwidth. Each container
is allocated by the ResourceManager and runs on a NodeManager, which monitors
and reports the resource usage.
4. How does YARN handle failures of Application Masters or Node Managers?
When an Application Master fails, YARN restarts it a configurable number of times. If
the NodeManager hosting the Application Master fails, YARN reallocates the
Application Master to another NodeManager. For NodeManager failures, YARN
reassigns containers to other healthy nodes.
5. Write a short note on Capacity Scheduler.
The Capacity Scheduler in YARN allows multiple tenants to share a cluster while
ensuring capacity guarantees. It provides resource fairness by dividing cluster
resources into queues with configurable capacities. Applications are scheduled
based on available capacity and priority, enabling efficient resource utilization.
6. Explain the process of HDFS block recovery when a DataNode becomes
unavailable.
When a DataNode becomes unavailable, the NameNode marks its blocks as under-
replicated. The NameNode replicates these blocks to other DataNodes to maintain
the replication factor. Once the replication is complete, the system ensures data
integrity and availability.
7. How does HDFS handle file appends, and what are the challenges
associated with appending data to existing files?
HDFS supports file appends by locking the file during writes to maintain consistency.
Challenges include maintaining data integrity, handling concurrent writes, and
ensuring synchronization between the NameNode and DataNodes. Appending can
also increase I/O load and block fragmentation.
8. What is the purpose of the Rebalancer in HDFS, and how does it ensure
balanced data distribution across DataNodes?
The Rebalancer in HDFS redistributes data across DataNodes to maintain balanced
storage utilization. It monitors block distribution and moves blocks from over-utilized
nodes to under-utilized ones without disrupting active operations, ensuring better
cluster performance and fault tolerance.
9. Explain the concept of a write pipeline in HDFS and its importance in data
durability.
The write pipeline in HDFS is the process by which data is written to multiple
DataNodes sequentially. When a client writes data, it is replicated across multiple
DataNodes based on the replication factor. This ensures data durability and fault
tolerance, as data remains available even if a DataNode fails.
10. What happens if a client tries to write to a file in HDFS while another client
is already writing to it?
HDFS does not support multiple concurrent writers to the same file. If a second client
tries to write, it receives an error. This lock mechanism prevents data corruption and
ensures consistency.
11. Explain the architecture of YARN's ResourceManager and its key
components like the Scheduler and Application Manager.
The ResourceManager in YARN is the central authority responsible for managing
cluster resources. It has two main components:
Scheduler: Allocates resources based on policies (e.g., Capacity or Fair
Scheduler).
Application Manager: Manages application lifecycles, accepts job
submissions, and communicates with Application Masters.
12. What are YARN queues, and how can you configure them to implement
resource fairness and priority?
YARN queues are logical divisions of cluster resources that manage application
execution. You can configure queues by setting capacities, priorities, and access
policies in the Capacity Scheduler configuration file. This ensures fair sharing and
prioritization based on business needs.
13. Describe the lifecycle of a YARN application, from submission to
completion.
1. Client submits an application to the ResourceManager.
2. ResourceManager allocates a container for the Application Master.
3. Application Master manages application execution and requests more
containers.
4. NodeManagers execute tasks in containers.
5. Upon task completion, the Application Master notifies the ResourceManager.
6. The application is marked as finished and resources are released.
14. What is the difference between container reuse and container reallocation
in YARN, and how does each impact performance?
Container Reuse: Reuses existing containers for new tasks, reducing
overhead from container creation and improving performance.
Container Reallocation: Allocates new containers for each task, which
increases latency but provides better isolation and fault tolerance.
15. How does YARN's Node Labeling feature work, and when would you use it?
Node labeling allows tagging nodes with labels to control task execution on specific
nodes. It is used to segregate workloads based on hardware capabilities, security
requirements, or performance needs.
16. What happens when the ResourceManager fails in a YARN cluster? How
does YARN ensure high availability?
When the ResourceManager fails, YARN uses an active-standby configuration where
a standby ResourceManager takes over. State information is periodically saved
using ZooKeeper, ensuring minimal downtime and application continuity.
17. What are some advantages of YARN over traditional resource management
models?
Supports multiple frameworks (e.g., MapReduce, Spark).
Better resource utilization through fine-grained allocation.
Scalability and fault tolerance.
Flexible scheduling with policies like Capacity and Fair Scheduler.
18. How does HDFS optimize read performance using data locality, and what
happens if data locality is not possible?
HDFS optimizes read performance by attempting to schedule tasks on nodes where
data resides (data locality). If data locality is not possible, tasks read data remotely,
which increases network traffic and reduces performance.
19. How does the preemption mechanism in YARN work, and in what scenarios
is it used?
Preemption in YARN forcibly reclaims resources from low-priority applications to
meet the demands of high-priority ones. It is used in over-utilized clusters to maintain
service-level agreements (SLAs).
20. How can you optimize YARN cluster utilization by tuning resource
parameters like [Link]-allocation-mb and
[Link]-allocation-vcores?
Optimizing these parameters involves:
Adjusting minimum-allocation-mb to define the smallest memory chunk a
container can request.
Configuring maximum-allocation-vcores to control the maximum CPU cores
a container can use.
Fine-tuning ensures balanced resource allocation and reduces resource
fragmentation, improving overall cluster efficiency.