II [Link] C.

S Introduction to Data Science Unit IV

Unit-IV
Introduction to Hadoop :Hadoop framework – Spark – replacing MapReduce–
NoSQL – ACID – CAP – BASE – types

Introduction to Hadoop
Hadoop is an open-source software framework that is used for storing and
processing large amounts of data in a distributed computing environment.
It is designed to handle big data and is based on the MapReduce
programming model, which allows for the parallel processing of large
datasets.
Its framework is based on Java programming with some native code in C and
shell scripts.
Hadoop is designed to process large volumes of data (Big Data) across many
machines without relying on a single machine.
It is built to be scalable, fault-tolerant and cost-effective.
Instead of relying on expensive high-end hardware, Hadoop works by
connecting many inexpensive computers (called nodes) in a cluster.

Hadoop Architecture

pg. 1
II [Link] C.S Introduction to Data Science Unit IV

Components of Hadoop Architecture

Hadoop Architecture Mainly consists of 4 components:
 MapReduce
 HDFS (Hadoop Distributed File System)
 YARN (Yet Another Resource Negotiator)
 Common Utilities or Hadoop Common

Hadoop has two main components:

 Hadoop Distributed File System (HDFS): HDFS breaks big files into
blocks and spreads them across a cluster of machines. This ensures data is
replicated, fault-tolerant and easily accessible even if some machines fail.
 MapReduce: MapReduce is the computing engine that processes data in a
distributed manner. It splits large tasks into smaller chunks (map) and then
merges the results (reduce), allowing Hadoop to quickly process massive
datasets.
Apart from the above-mentioned two core components, Hadoop
framework also includes the following two modules Hadoop Common which
are Java libraries and utilities required by other Hadoop modules and Hadoop
YARN which is a framework for job scheduling and cluster resource
management.

pg. 2
II [Link] C.S Introduction to Data Science Unit IV

Hadoop Distributed File System (HDFS)

HDFS is the storage layer of Hadoop. It breaks large files into smaller
blocks (usually 128 MB or 256 MB) and stores them across multiple DataNodes.
Each block is replicated (usually 3 times) to ensure fault tolerance so even if a
node fails, the data remains available.
Key features of HDFS:
 Scalability: Easily add more nodes as data grows.
 Reliability: Data is replicated to avoid loss.
 High Throughput: Designed for fast data access and transfer.
MapReduce
MapReduce is a data processing model in Hadoop that runs on YARN. It
enables fast, distributed and parallel processing by dividing tasks into two
phases Map and Reduce making it efficient for handling large-scale data.

MapReduce is the computation layer in Hadoop. It works in two main phases:

1. Map Phase: Input data is divided into chunks and processed in parallel. Each
mapper processes a chunk and produces key-value pairs.
pg. 3
II [Link] C.S Introduction to Data Science Unit IV

2. Reduce Phase: These key-value pairs are then grouped and combined to
generate final results.
This model is simple yet powerful, enabling massive parallelism and
efficiency.

MapReduce Workflow: Workflow begins when input data is split into key-value
pairs by Map() function. These are then grouped by key and processed by
Reduce() function for tasks like sorting or aggregation. The final output is written
to HDFS.

How Does Hadoop Work?

Here’s a overview of how Hadoop operates:
 Data is loaded into HDFS, where it's split into blocks and distributed
across DataNodes.
 MapReduce jobs are submitted to the ResourceManager.
 The job is divided into map tasks, each working on a block of data.
 Map tasks produce intermediate results, which are shuffled and sorted.
 Reduce tasks aggregate results and generate final output.
 The results are stored back in HDFS or passed to other applications.
Advantages and Disadvantages of Hadoop
Advantages:
 Scalability: Easily scale to thousands of machines.
 Cost-effective: Uses low-cost hardware to process big data.
 Fault Tolerance: Automatic recovery from node failures.
 High Availability: Data replication ensures no loss even if nodes fail.
 Flexibility: Can handle structured, semi-structured and unstructured data.
 Open-source and Community-driven: Constant updates and wide
support.

pg. 4
II [Link] C.S Introduction to Data Science Unit IV

Disadvantages:
 Not ideal for real-time processing (better suited for batch processing).
 Complexity in programming with MapReduce.
 High latency for certain types of queries.
 Requires skilled professionals to manage and develop.
Applications
Hadoop is used across a variety of industries:
1. Banking: Fraud detection, risk modeling.
2. Retail: Customer behavior analysis, inventory management.
3. Healthcare: Disease prediction, patient record analysis.
4. Telecom: Network performance monitoring.
5. Social Media: Trend analysis, user recommendation engines.
Map Task Components:
 RecordReader: reads input data and converts it into key-value pairs,

with keys as location info and values as actual data.

 Mapper: processes each pair and outputs zero or more new key-value
pairs.
 Combiner (optional): acts as a mini-reducer to group Mapper output and
reduce data transfer before shuffling.
 Partitioner: assigns key-value pairs to Reducers using [Link]()
% number Of Reducers.
Reduce Task Components:
 Shuffle and Sort: transfers intermediate key-value pairs from Mappers

to Reducers and sorts them by key. Shuffling begins as soon as some

Mappers finish.
 Reducer: processes grouped key-value pairs, performing tasks like
aggregation or filtering based on logic.
 OutputFormat: writes final results to HDFS using a RecordWriter,
typically storing each record as a key-value line.

pg. 5
II [Link] C.S Introduction to Data Science Unit IV

[Link]
HDFS (Hadoop Distributed File System) is Hadoop’s primary storage
system, built for high-throughput access to large datasets. It runs on inexpensive
commodity hardware and stores data in large blocks to optimize performance.
HDFS ensures fault tolerance and high availability across the cluster.
HDFS Architecture Components:
 NameNode (Master Node): The master node in HDFS that stores
metadata (not actual data), manages file operations and directs clients to
nearest DataNode for efficient access.
 DataNode (Slave Node): Stores actual data blocks, serves read/write
requests and reports to NameNode. Supports replication (default 3) for
fault tolerance and scales storage and performance with more nodes.
High Level Architecture Of Hadoop

File Block In HDFS: In HDFS data is always stored in the form of

blocks. By default, each block is 128MB in size, although this value can be
manually configured depending on the use case (commonly increased to 256MB
or more in modern systems).

pg. 6
II [Link] C.S Introduction to Data Science Unit IV

Suppose you upload a file of 400MB to HDFS. Hadoop will divide this file into
blocks as follows:
128MB + 128MB + 128MB + 16MB = 400MB

This creates four blocks three of 128MB and one of 16MB. Hadoop splits
files purely by size, not content, so a single record can span across two blocks.
1. Comparison with Traditional File Systems
 Traditional file systems use small blocks (e.g., 4KB), while HDFS uses
large blocks (128MB or more).
 Larger blocks in HDFS reduce metadata and I/O overhead, improving
scalability and efficiency for big data processing.
2. Replication In HDFS:
HDFS replication ensures data availability and fault tolerance by storing
multiple copies of each block.
 Default Replication Factor: 3 (configurable in [Link])
 If a file is split into 4 blocks, with a replication factor of 3: 4 blocks × 3
replicas = 12 total blocks
Designed for commodity hardware, where failures are common—replication
prevents data loss. While it increases storage usage, reliability is prioritized over
space efficiency.

pg. 7
II [Link] C.S Introduction to Data Science Unit IV

Rack Awareness:
A rack is a group of machines (typically 30–40) in a Hadoop cluster. Large
clusters have many racks. Rack Awareness helps NameNode to:
 Choose the nearest DataNode for faster read/write operations.
 Reduce network traffic by minimizing inter-rack data transfer.
This improves overall performance and efficiency in data access.
HDFS Architecture

[Link] (Yet Another Resource Negotiator)

YARN is resource management layer in Hadoop ecosystem. It allows
multiple data processing engines like MapReduce, Spark and others to run and
share cluster resources efficiently.
It handles two core responsibilities:
 Job Scheduling: Splits large tasks into smaller jobs, assigns them to
nodes and manages priorities, dependencies and execution.

pg. 8
II [Link] C.S Introduction to Data Science Unit IV

 Resource Management: Allocates and monitors cluster resources (CPU,

memory, etc.) needed for job execution.
Components of Yarn:
 ResourceManager: Master node that manages global resource
allocation.
 NodeManager: Slave node that monitors resources on individual nodes.
 ApplicationMaster: Manages lifecycle of each individual
application/job.
Key Features of YARN:
 Multi-Tenancy: Supports multiple users and applications.
 Scalability: Efficiently scales to handle thousands of nodes and jobs.
 Better Cluster Utilization: Maximizes resource usage across the cluster.
 Compatibility: Works with MapReduce and other processing models
like Spark.
4. Hadoop Common (Common Utilities)
Hadoop Common, also known as Common Utilities, includes core Java
libraries and scripts required by all components in a Hadoop ecosystem such as
HDFS, YARN and MapReduce.
These libraries offer core functionalities such as:
 File system and I/O operations
 Configuration and logging
 Security and authentication
 Network communication
Hadoop Common provides shared libraries and utilities that help all Hadoop
components work together. It handles hardware failures automatically and
includes tools like Hadoop Archive, native library support and RPC
mechanisms.

pg. 9
II [Link] C.S Introduction to Data Science Unit IV

Introduction to NoSQL

The NoSQL system or "Not Only SQL" is essentially a database that is

made specifically for unstructured and semi-structured data in very large
quantities. Unlike Conventional Relational Databases, where data are organized
into tables using predefined schemas. NoSQL allows flexible models to be
organized and horizontally scalable.
Key Features of NoSQL Databases
 Dynamic schema: Allow flexible shaping of data to meet new

requirements without the need to migrate or change schemas.

 Horizontal scalability: They scale horizontally for adding more nodes
into the existing ones and acquire enough storage for even bigger datasets
and much higher traffic by distributing the load on multiple servers.
 Document-based: Data are presented in flexible, semi-structured
formats like JSON/BSON (e.g., MongoDB).
 Key-value-based: They possess a simple but fast access pattern (e.g.,
Redis) by storing data as pairs of keys and values.
 Column-based: Data are organized into columns instead of rows (e.g.,
CASSANDRA).
 Distributed and high availability: They are designed to be highly
available and to automatically handle node failures and data replication
across multiple nodes in a database cluster.
 Flexibility: Allow developers to store and retrieve data in a flexible and
dynamic manner, with support for multiple data types and changing data
structures.
 Performance: Perfect for big data and real-time analytics and high
volume applications.

pg. 10
II [Link] C.S Introduction to Data Science Unit IV

Why Use NoSQL?

Unlike relational databases, which use Structured Query
Language, NoSQL databases do not have a universal query language. In fact,
each NoSQL database has its own approach to query languages.
Traditional relational databases will follow ACID principles, assuring a
strong consistency and a structured relationship between the data.
The needs of applications have been changing through time, due to
increased requirements related to big data, real-time analytics and distributed
environments NoSQL emerged to satisfy:
 where scaling can be done horizontally by adding nodes instead of
upgrading the existing machine.
 Flexibility in supporting unstructured or semi-structured data without a
rigid schema.
 Optimized for fast read/write operations with large datasets resulting in
higher performance.
 Distributed Architecture to build highly available and partition-tolerating
system.
Challenges of NoSQL Databases
 Lack of standardization: NoSQL systems can be vastly different from
one another, making it even harder to choose the right one for a specific
use case.
 Lack of ACID compliance: NoSQL databases may not provide
consistency, which is a disadvantage for applications that need strict data
integrity.
 Narrow focus: Great for storage but lack functionalities as transaction
management, in which relational databases are great.
 Absence of Complex Query Support: They are not designed to handle
complex queries, which means that they are not a good fit for applications
that require complex data analysis or reporting.
 Lack of maturity: Being relatively new, NoSQL may not have the
reliability, security and feature set of traditional relational databases.
 Management complexity: For large datasets, maintaining a NoSQL
database could be quite more complicated than managing a relational
database.
 Limited GUI Tools: While some NoSQL databases, like MongoDB offer
GUI tools like MongoDB Compass, not all NoSQL databases provide
flexible or user-friendly GUI tools.

pg. 11
II [Link] C.S Introduction to Data Science Unit IV

ACID
ACID stands for Atomicity, Consistency, Isolation, and Durability.

There are Four Properties of ACID

1. Atomicity
Atomicity means a transaction is all-or-nothing either all its operations succeed,
or none are applied. If any part fails, the entire transaction is rolled back to keep
the database consistent.
 Commit: If the transaction is successful, the changes are permanently
applied.
 Abort/Rollback: If the transaction fails, any changes made during the
transaction are discarded.
Example: Consider the following transaction T consisting of T1 and T2 :
Transfer of $100 from account X to account Y .

pg. 12
II [Link] C.S Introduction to Data Science Unit IV

If the transaction fails after completion of T1 but before completion of T2,

the database would be left in an inconsistent state. With Atomicity, if any part
of the transaction fails, the entire process is rolled back to its original state, and
no partial changes are made.
2. Consistency
Consistency in transactions means that the database must remain in a valid
state before and after a transaction.
 A valid state follows all defined rules, constraints, and relationships (like
primary keys, foreign keys, etc.).
 If a transaction violates any of these rules, it is rolled back to prevent
corrupt or invalid data.
 If a transaction deducts money from one account but doesn't add it to
another (in a transfer), it violates consistency.
Example: Suppose the sum of all balances in a bank system should always be
constant. Before a transfer, the total balance is $700. After the transaction, the
total balance should remain $700. If the transaction fails in the middle (like
updating one account but not the other), the system should maintain its
consistency by rolling back the transaction.

Total beforeT occurs = 500+200=700 .

Total after T occurs = 400 + 300 = 700 .

pg. 13
II [Link] C.S Introduction to Data Science Unit IV

3. Isolation
Isolation ensures that transactions run independently without affecting
each other. Changes made by one transaction are not visible to others until they
are committed.
It ensures that the result of concurrent transactions is the same as if they were
run one after another, preventing issues like:
 Dirty reads: reading uncommitted data
 Non-repeatable reads: data changes between two reads
 Phantom reads: new rows appear during a transaction
Example: Consider two transactions T and T''.
 X = 500, Y = 500

Explanation:

1. Transaction T:
 T wants to transfer $50 from X to Y.
 T reads Y (value: 500), deducts $50 from X (new X = 450), and adds $50
to Y (new Y = 550).
2. Transaction T'':
 T'' starts and reads X (500) and Y (500).
 It calculates the sum: 500 + 500 = 1000.
 Meanwhile, values of X and Y change to 450 and 550 respectively.
 So, the correct sum should be 450 + 550 = 1000.
 Isolation ensures that T'' does not read outdated values while another
transaction (T) is still in progress.

pg. 14
II [Link] C.S Introduction to Data Science Unit IV

 Transactions should be independent, and T'' should access the final values
only after T commits.
 This avoids inconsistent results, like the incorrect sum calculated by T''.
4. Durability:
Durability ensures that once a transaction is committed, its changes are
permanently saved, even if the system fails. The data is stored in non-volatile
memory, so the database can recover to its last committed state without losing
data.
Example: After successfully transferring money from Account A to Account B,
the changes are stored on disk. Even if there is a crash immediately after the
commit, the transfer details will still be intact when the system recovers,
ensuring durability.
How ACID Properties Impact DBMS Design and Operation
The ACID properties, in totality, provide a mechanism to ensure the
correctness and consistency of a database in a way such that each transaction is
a group of operations that acts as a single unit, produces consistent results, acts
in isolation from other operations, and updates that it makes are durably stored.
1. Data Integrity and Consistency
ACID properties safeguard the data integrity of a DBMS by ensuring that
transactions either complete successfully or leave no trace if interrupted. They
prevent partial updates from corrupting the data and ensure that the database
transitions only between valid states.
2. Concurrency Control
ACID properties provide a solid framework for managing concurrent
transactions. Isolation ensures that transactions do not interfere with each other,
preventing data anomalies such as lost updates, temporary inconsistency, and
uncommitted data.
3. Recovery and Fault Tolerance
Durability ensures that even if a system crashes, the database can recover
to a consistent state. Thanks to the Atomicity and Durability properties, if a
transaction fails midway, the database remains in a consistent state.

pg. 15
II [Link] C.S Introduction to Data Science Unit IV

Critical Use Cases for ACID in Databases

In modern applications, ensuring the reliability and consistency of data is
crucial. ACID properties are fundamental in sectors like:

CAP THEROEM
The CAP theorem is a belief from theoretical computer science about
distributed data stores that claims, in the event of a network failure on a
distributed database, it is possible to provide either consistency or availability—
but not both.
What is the CAP Theorem?
The CAP theorem is a fundamental concept in distributed systems theory
that was first proposed by Eric Brewer in 2000 and subsequently shown by Seth
Gilbert and Nancy Lynch in 2002. It asserts that all three of the following
qualities cannot be concurrently guaranteed in any distributed data system:
1. Consistency
Consistency means that all the nodes (databases) inside a network will
have the same copies of a replicated data item visible for various transactions.
It guarantees that every node in a distributed cluster returns the same, most
recent, and successful write.
It refers to every client having the same view of the data. There are
various types of consistency models. Consistency in CAP refers to sequential
consistency, a very strong form of consistency.

pg. 16
II [Link] C.S Introduction to Data Science Unit IV

Note that the concept of Consistency in ACID and CAP are slightly
different since in CAP, it refers to the consistency of the values in different
copies of the same data item in a replicated distributed system.
In ACID, it refers to the fact that a transaction will not violate the integrity
constraints specified on the database schema.
For example, a user checks his account balance and knows that he has 500
rupees. He spends 200 rupees on some products. Hence the amount of 200 must
be deducted changing his account balance to 300 rupees.
This change must be committed and communicated with all other
databases that hold this user's details. Otherwise, there will be inconsistency,
and the other database might show his account balance as 500 rupees which is
not true.

2. Availability
Availability means that each read or write request for a data item will
either be processed successfully or will receive a message that the operation
cannot be completed. Every non-failing node returns a response for all the read
and write requests in a reasonable amount of time.
The key word here is "every". In simple terms, every node (on either side
of a network partition) must be able to respond in a reasonable amount of time.

For example, user A is a content creator having 1000 other users subscribed to
his channel. Another user B who is far away from user A tries to subscribe to
user A's channel. Since the distance between both users are huge, they are
connected to different database node of the social media network.

pg. 17
II [Link] C.S Introduction to Data Science Unit IV

If the distributed system follows the principle of availability, user B must

be able to subscribe to user A's channel.

3. Partition Tolerance
Partition tolerance means that the system can continue operating even if
the network connecting the nodes has a fault that results in two or more
partitions, where the nodes in each partition can only communicate among each
other. That means, the system continues to function and upholds its consistency
guarantees in spite of network partitions.
Network partitions are a fact of life. Distributed systems guaranteeing
partition tolerance can gracefully recover from partitions once the partition
heals.
For example, take the example of the same social media network where
two users are trying to find the subscriber count of a particular channel. Due to
some technical fault, there occurs a network outage, the second database
connected by user B losses its connection with first database.
Hence the subscriber count is shown to the user B with the help of replica
of data which was previously stored in database 1 backed up prior to network
outage. Hence the distributed system is partition tolerant.

pg. 18
II [Link] C.S Introduction to Data Science Unit IV

The CAP theorem states that distributed databases can have at most two
of the three properties: consistency, availability, and partition tolerance. As
a result, database systems prioritize only two properties at a time.

The Trade-Offs in the CAP Theorem

The CAP theorem implies that a distributed system can only provide two
out of three properties:
1. CA (Consistency and Availability)

These types of system always accept the request to view or modify the
data sent by the user and they are always responded with data which is
consistent among all the database nodes of a big, distributed network.

pg. 19
II [Link] C.S Introduction to Data Science Unit IV

However, such type of distributed systems is not realizable in real world

because when network failure occurs, there are two options: Either send old data
which was replicated moments ago before network failure or do not allow user
to access the already moments old data. If we choose first option, our system
will become Available and if we choose second option our system will become
Consistent.
The combination of consistency and availability is not possible in
distributed systems and for achieving CA, the system has to be monolithic such
that when a user updates the state of the system, all other users accessing it are
also notified about the new changes which means that the consistency is
maintained. And since it follows monolithic architecture, all users are connected
to single system which means it is also available.

These types of systems are generally not preferred due to a requirement

of distributed computing which can be only done when consistency or
availability is sacrificed for partition tolerance.

Example databases: MySQL, PostgreSQL

pg. 20
II [Link] C.S Introduction to Data Science Unit IV

[Link] (Availability and Partition Tolerance)

These types of system are distributed in nature, ensuring that the

request sent by the user to view or modify the data present in the
database nodes are not dropped and are processed in presence of a
network partition.

The system prioritizes availability over consistency and can respond with
possibly stale data which was replicated from other nodes before the partition
was created due to some technical failure. Such design choices are generally
used while building social media websites such as Facebook, Instagram, Reddit,
etc. and online content websites like YouTube, blog, news, etc.
where consistency is usually not required, and a bigger problem arises if
the service is unavailable causing corporations to lose money since the users
may shift to new platform.
The system can be distributed across multiple nodes and is designed to
operate reliably even in the face of network partitions.

Example databases: Amazon DynamoDB, Google Cloud Spanner.

3. CP (Consistency and Partition Tolerance)

These types of system are distributed in nature, ensuring that the

request sent by the user to view or modify the data present in the
database nodes are dropped instead of responding with inconsistent
data in presence of a network partition.

The system prioritizes consistency over availability and does not allow
users to read crucial data from the stored replica which was backed up prior to
the occurrence of network partition. Consistency is chosen over availability for
critical applications where latest data plays an important role such as stock
market application, ticket booking application, banking, etc. where problem will
arise due to old data present to users of application.
For example, in a train ticket booking application, there is one seat which
can be booked. A replica of the database is created, and it is sent to other nodes
of the distributed system. A network outage occurs which causes the user
connected to the partitioned node to fetch details from this replica.

pg. 21
 II [Link] C.S Introduction to Data Science Unit IV

Some user connected to the unpartitioned part of distributed network and

already booked the last remaining seat. However, the user connected to
partitioned node will still one seat which makes the available data inconsistent.
It would have been better if the user was shown error and make the system
unavailable for the user and maintain consistency. Hence consistency is chosen
in such scenarios.

Example databases: Apache HBase, MongoDB, Redis.

BASE Databases

What Is BASE?
Most commonly refers to a baseline model, which is a simple,
unoptimized model serving as a benchmark to evaluate the performance of more
complex models.
The BASE stands for Basically Available, Soft State, and Eventual
Consistency.

 Basically Available: The system is guaranteed to be available in event of failure.

 Soft State: The state of the data could change without application interactions
due to eventual consistency.
 Eventual Consistency: The system will be eventually consistent after the
application input. The data will be replicated to different nodes and will
eventually reach a consistent state. But the consistency is not guaranteed at a
transaction level.

Base Types of Databases:

This refers to the fundamental architectural models used to store and

organize data in a database management system. These include:
 Relational Databases (RDBMS): Organize data into tables with predefined
schemas, using rows and columns. They enforce strict data integrity rules
(ACID properties). Examples include MySQL, PostgreSQL, Oracle.
 NoSQL Databases: Offer more flexible, schema-less structures for handling
large volumes of unstructured or semi-structured data. They are categorized into
sub-types:

pg. 22
 II [Link] C.S Introduction to Data Science Unit IV

 Document Databases: Store data as flexible, self-describing documents (e.g.,

MongoDB).
 Key-Value Stores: Store simple key-value pairs for fast lookups (e.g., Redis).
 Columnar Databases: Store data by columns, optimized for analytical queries
(e.g., Apache Cassandra).
 Graph Databases: Store data as nodes and edges to represent relationships
(e.g., Neo4j).
 Hierarchical Databases: Organize data in a tree-like structure with parent-
child relationships.
 Network Databases: Use a graph-like structure to represent many-to-many
relationships.
 Object-Oriented Databases: Store data as objects, aligning with object-
oriented programming paradigms.

Uses of BASE Databases

BASE databases are used in modern, highly-available, and scalable

systems that handle large amounts of data.
Examples of such systems include online shopping websites, social media
platforms, and cloud-based services.

Difference between Base Properties and ACID Properties

pg. 23