Module 2
Introduction – What is Hadoop?
1. Hadoop is an open-source framework developed by Apache.
2. It is designed to store and process huge volumes of data (Big Data).
3. It follows a distributed computing model → data is stored across multiple machines, and computation also
happens in parallel.
4. It is fault-tolerant (still works even if some nodes fail).
5. Hadoop works on a cluster of commodity hardware (low-cost machines, not supercomputers).
Core Components of Hadoop
Hadoop mainly has four core components:
1. HDFS (Hadoop Distributed File System) – Storage
• It is the storage layer of Hadoop.
• Stores large files by splitting them into blocks (default 128 MB / 256 MB).
• These blocks are replicated on multiple nodes for fault tolerance.
• Hadoop uses a Master/Slave Architecture:
o NameNode →It is also known as Master node,
It keeps metadata (info about files, blocks, locations).
Does not store actual data, only the directory tree of the file system.
Namenode has the job tracker which keep track of files distributed to Datanodes.
Single Name node per cluster.
o DataNode → It is also known Worker nodes or Slave,
It actually store the blocks of data.
Periodically sends heartbeat signals to NameNode → to confirm they are alive.
If a DataNode fails, data is still safe because of replication
To create, move and delete block datanode receives a instruction from the local disk.
• Example: If a 1 GB file is stored, it is split into 8 blocks (128 MB each) and spread across multiple machines.
2. MapReduce – Processing
• It is the processing layer of Hadoop.
• Works on the principle of divide and conquer:
• Hadoop has two important phase for data processing , i.e MapPhase and ReducePhase.
o Map phase → Divides the big task into smaller sub-tasks.
o Reduce phase → Combines results of all sub-tasks.
• Example: To count words in 100 GB of text:
o Map → Each node counts words in its own chunk.
o Reduce → All counts are combined into a final result.
3. Key benefits
• Cost-effective: It uses inexpensive, "commodity" hardware instead of expensive, high-end servers.
• Scalable: It can be scaled to handle extremely large amounts of data by adding more nodes to the cluster.
• Versatile: It can store and process structured, semi-structured, and unstructured data.
• Fast: The parallel processing capabilities allow for fast analysis of large datasets.
4. Limitation
• Hadoop can perform only batch processing and sequential access.
• Sequential access is time consuming.
Introduction – What is NoSQL?
1. NoSQL = "Not Only SQL" databases.
2. They are non-relational databases designed to handle large, unstructured, and semi-structured data.
3. Unlike traditional RDBMS (Relational DBs) which use tables + fixed schema, NoSQL is:
o Schema-free / flexible schema
o Horizontally scalable (can add more servers easily)
o Supports high-speed read/write
4. Common types of NoSQL databases:
o Key-Value stores (e.g., Redis, DynamoDB)
o Document stores (e.g., MongoDB, CouchDB)
o Column-oriented (e.g., Cassandra, HBase)
o Graph databases (e.g., Neo4j)
Business Drivers for NoSQL
(Why companies prefer NoSQL in Big Data Analytics)
1. Scalability
• Businesses generate huge amounts of data (social media, IoT, e-commerce).
• NoSQL provides horizontal scalability → just add more cheap servers instead of costly high-end ones.
2. Handling Big Data (Variety, Velocity, Volume)
• Modern businesses deal with structured + unstructured data (videos, images, logs, clickstreams).
• NoSQL can store and process all types of data, unlike RDBMS which only handles structured data.
3. High Performance
• NoSQL databases are optimized for fast read/write operations.
• Important for real-time applications like online shopping carts, banking transactions, recommendation
engines.
4. Flexible Schema
• RDBMS needs a fixed schema before storing data.
• In dynamic businesses (startups, e-commerce, social media), schema may change frequently.
• NoSQL supports schema-less or flexible schema, so businesses can adapt quickly.
5. Cost-Effectiveness
• Runs on commodity hardware (low-cost servers).
• Saves money compared to buying expensive enterprise database servers.
6. Cloud & Distributed Computing Support
• Many businesses move to cloud platforms (AWS, Azure, GCP).
• NoSQL databases integrate easily with cloud services for distributed storage and global access.
7. Real-Time Analytics
• Businesses need instant insights (fraud detection, stock trading, personalized ads).
• NoSQL supports real-time data processing better than traditional RDBMS.
8. Availability & Fault Tolerance
• NoSQL databases have built-in replication and failover mechanisms.
• Ensures high availability → apps keep running even if some servers fail.
9. Support for Modern Applications
• Social networks, IoT devices, recommendation systems, and gaming apps need:
o High scalability
o Flexible schema
o Real-time processing
• NoSQL is the best fit for these use cases.
CAP THEOREM
The CAP Theorem explains that a distributed database system can provide only two of the following three properties
at the same time:
• C – Consistency:
1. Every node shows the same data at the same time.
2. When a user reads data, they always get the latest updated value.
3. Example: If you update your profile picture, everyone should see the new one instantly.
• A – Availability:
1. The system is always available to respond to user requests.
2. Even if some parts fail, users should still be able to access the data.
• P – Partition Tolerance:
1. The system continues to work even if some network connections fail
2. Even if one server goes down, your app should still work.
A system cannot achieve all three together — it must choose any two.
Example:
• CP System: Focuses on Consistency + Partition Tolerance (e.g., HBase).
• AP System: Focuses on Availability + Partition Tolerance (e.g., Cassandra).
Combination Description Example
CA (Consistency + Works only if there’s no network failure. Hard in real Traditional RDBMS (Single
Availability) distributed systems. server)
CP (Consistency + Partition System stays consistent but may become temporarily HBase, MongoDB (in some
Tolerance) unavailable during failures. modes)
AP (Availability + Partition System stays available but may return stale data Cassandra, CouchDB,
Tolerance) (eventually consistent). DynamoDB
Difference between RDMS VS Hadoop
R Programming
1. Definition:
R is a programming language and software environment used mainly for statistical computing, data
analysis, and visualization.
2. Developed by:
It was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand.
3. Type of Language:
It is an open-source, interpreted, and high-level language.
4. Main Use:
R is widely used for data analysis, data visualization, machine learning, and statistical modeling.
5. Packages:
R has thousands of packages (libraries) like ggplot2, dplyr, and tidyverse that make data handling easy.
6. Data Handling:
R provides powerful tools to manage, clean, and process large datasets efficiently.
7. Visualization:
It can create beautiful and detailed graphs, charts, and plots for data presentation.
8. Integration:
R can be integrated with Hadoop, Python, and SQL, making it useful in Big Data Analytics.
9. Community Support:
R has a large and active community, providing free tutorials, forums, and updates.
10. Example:
x <- c(1, 2, 3, 4, 5)
mean(x)
→ Output: 3
In R programming, we often need to combine multiple datasets to analyze or process data together.
R provides various built-in functions to merge, join, and bind data frames, vectors, or matrices easily.
2. Functions to Combine Different Sets of Data
(i) c() – Combine Elements into a Vector
• Used to combine values or variables into a single vector.
• Example:
• x <- c(1, 2, 3)
• y <- c(4, 5)
• z <- c(x, y)
• print(z) # Output: 1 2 3 4 5
(ii) rbind() – Combine Rows
• Used to add rows of two or more data frames or matrices having the same columns.
• Example:
• df1 <- [Link](ID=c(1,2), Name=c("A","B"))
• df2 <- [Link](ID=c(3,4), Name=c("C","D"))
• result <- rbind(df1, df2)
Combines row-wise.
(iii) cbind() – Combine Columns
• Used to add columns of two or more data frames, vectors, or matrices having the same number of rows.
• Example:
• x <- [Link](ID=c(1,2,3))
• y <- [Link](Name=c("A","B","C"))
• result <- cbind(x, y)
Combines column-wise.
(iv) merge() – Merge Data Frames by Common Columns
• Works like SQL join operations (inner, left, right, full).
• Matches data based on common column(s) (keys).
• Example:
• df1 <- [Link](ID=c(1,2,3), Name=c("A","B","C"))
• df2 <- [Link](ID=c(2,3,4), Age=c(22,23,24))
• result <- merge(df1, df2, by="ID")
Combines where ID matches.
(v) append() – Add Elements to a Vector
• Adds elements of one vector to another.
• Example:
• x <- c(1, 2, 3)
• y <- append(x, 4)
Output: 1 2 3 4
(vi) join() (from plyr or dplyr package)
• Used to join two data frames using different types of joins like inner_join, left_join, right_join, full_join.
• Example (dplyr):
• library(dplyr)
inner_join(df1, df2, by="ID")
Introduction
• In R programming, the workspace is the area where all objects such as variables, data frames, vectors, and
functions are stored during a session.
• R provides several functions to view, manage, save, and remove data objects from the workspace.
2. Functions to Handle Data in R Workspace
(i) getwd() – Get Working Directory
• Displays the current working directory where R reads and saves files.
• Example:
• getwd()
Output: "C:/Users/Sahil/Documents"
(ii) setwd() – Set Working Directory
• Used to change the current working directory to another folder.
• Example:
• setwd("C:/R_Projects/Data")
(iii) ls() – List Objects in Workspace
• Displays the names of all objects (data, variables, functions) stored in the current workspace.
• Example:
• ls()
Output: "x" "y" "df1"
(iv) rm() – Remove Objects
• Removes one or more objects from the workspace.
• Example:
• rm(x) # removes object x
• rm(list=ls()) # removes all objects
(v) save() – Save Objects to a File
• Saves one or more R objects to a file (usually with .RData extension).
• Example:
• save(x, y, file="[Link]")
(vi) load() – Load Saved Objects
• Loads the objects saved earlier using the save() function back into the workspace.
• Example:
• load("[Link]")
(vii) [Link]() – Save Entire Workspace
• Saves all objects in the current workspace to a file named .RData.
• Example:
• [Link](file="session_backup.RData")
(viii) history() – Display Command History
• Displays previously executed commands in the current session.
• Example:
• history()
(ix) q() – Quit R Session
• Exits from R and optionally saves the workspace before quitting.
• Example:
q(save="yes")
Data Stream Management System (DSMS):
A Data Stream Management System is a software system designed to store, process, and analyze continuous data
streams in real time, instead of static data stored in databases.
Issues and Challenges in Data Stream Query Processing:
1. Continuous Flow of Data:
o Data keeps coming rapidly, so queries must run continuously in real time.
2. Limited Memory:
o Streams are unbounded, but system memory is limited, making full storage impossible.
3. High Data Arrival Rate:
o Data can arrive faster than it can be processed, causing delays or data loss.
4. Handling Out-of-Order Data:
o Some data may arrive late or out of sequence, making results inaccurate.
5. Approximate Query Processing:
o Exact answers are hard; systems often give approximate or summarized results.
6. Dynamic Query Changes:
o Users may add, remove, or modify queries at any time while data is streaming.
7. Scalability:
o The system must scale to handle large, fast data streams efficiently.
Types of Stream Queries:
1. Standing Query (Continuous Query):
o Runs continuously over the data stream.
o Produces real-time results as new data arrives.
o Example: Continuously monitoring sensor data for temperature > 50°C.
2. Ad-hoc Query:
o Runs once on the current data available in the stream system.
o Used for one-time analysis or quick checks.
o Example: Checking the average temperature of data received in the last 5 minutes.
df <- [Link](
course = c(1, 2, 3, 4, 5, 6),
id = c(11, 12, 13, 14, 15, 16),
class = c(1, 2, 1, 2, 1, 2),
marks = c(56, 75, 48, 69, 84, 53)
)
df
course id class marks
1 11 1 56
2 12 2 75
3 13 1 48
4 14 2 69
5 15 1 84
6 16 2 53
i. Subset of course < 3 using []
df[df$course < 3, ]
course id class marks
1 11 1 56
2 12 2 75
ii. Subset where course < 3 OR class = 2 using subset()
subset(df, course < 3 | class == 2)
Output
course id class marks
1 11 1 56
2 12 2 75
4 14 2 69
6 16 2 53
If it was AND
subset(df, course < 3 & class == 2)
NOSQL Architectural Pattern
Describe th 4 ways by which big data problems are handled by Nosql
1. Introduction
• NoSQL (Not Only SQL) databases are designed to handle Big Data challenges such as:
o Huge volume of data,
o High velocity (speed), and
o Variety (different formats).
• They provide high scalability, flexibility, and fast data access compared to traditional relational databases.
2. Four Types of NoSQL Databases
(i) Key–Value Store Databases
Definition:
• Data is stored as a collection of key–value pairs (like a dictionary).
• Each key is unique and is used to access its value quickly.
Working:
• Similar to a hash table or map in programming.
• The key acts like an index, and the value can be a string, JSON, or object.
Example:
"User101" → { "Name": "Sahil", "Age": 22, "City": "Pune" }
Features:
• Very fast read/write operations.
• Simple data model, ideal for caching and session management.
Examples of Databases:
• Redis, Amazon DynamoDB, Riak
Use Case:
• Storing user sessions, shopping cart data, or configuration settings.
(ii) Column Store Databases
Definition:
• Data is stored in columns instead of rows (like in RDBMS).
• Each column family stores related data together for faster analytical queries.
Working:
• Each column is stored separately, allowing efficient data compression and aggregation.
• Suited for large-scale analytical workloads.
Example:
ID Name Age City
1 Sahil 22 Pune
→ Stored internally as columns like:
Column1: [1]
Column2: [Sahil]
Column3: [22]
Column4: [Pune]
Features:
• High performance for queries on specific columns.
• Scalable for large datasets.
Examples of Databases:
• Apache Cassandra, HBase, Google Bigtable
Use Case:
• Data warehousing, real-time analytics, and recommendation systems.
(iii) Document-Based Databases
Definition:
• Data is stored in documents (usually JSON, BSON, or XML format).
• Each document represents one record and contains key-value pairs, arrays, or nested objects..
Working:
1. Data is stored as documents inside collections (similar to tables in SQL).
2. Each document has a unique ID used to find it quickly.
3. Documents can contain different types of data — text, numbers, arrays, even other documents.
4. You can easily add, remove, or update fields without changing the structure of other documents.
5. Queries are done using field names, not fixed columns.
Example:
{
"StudentID": 101,
"Name": "Sahil",
"Courses": ["BDA", "ML", "AI"],
"City": "Pune"
}
Features:
• Flexible and schema-less
• Fast to read/write large data
• Easy to scale horizontally
• Stores complex data (nested objects)
Examples of Databases:
• MongoDB, CouchDB, RavenDB
Use Case:
• Content management systems, social media data, user profiles.
(iv) Graph Databases
Definition:
• Stores data as nodes (entities) and edges (relationships between nodes).
• Designed to handle highly connected data.
Working:
• Nodes represent objects, and edges represent relationships between them.
• Each relationship can have properties (attributes).
Example:
(Sahil) --[FRIEND_OF]--> (Amit)
(Amit) --[WORKS_AT]--> (TCS)
Features:
• Efficient for relationship-based queries.
• Can quickly answer questions like:
o “Who are Sahil’s friends?”
o “Which company do Sahil’s friends work at?”
Examples of Databases:
• Neo4j, OrientDB, Amazon Neptune
Use Case:
• Social networks, recommendation systems, fraud detection.
Recommendation System
1. A recommendation system suggests items (like products, movies, or songs) to users based on their interests
and behaviour.
2. It helps users discover relevant items from a large collection of data.
3. Uses techniques like data mining, machine learning, and user profiling.
4. Common examples: Netflix movie suggestions, Amazon product recommendations, YouTube video
suggestions.
5. It improves user experience and helps companies increase sales and engagement.
2. Why Recommendation Systems are Needed?
• To reduce information overload.
• To increase user satisfaction.
• To improve sales, engagement, and personalization.
3. Types of Recommendation Systems
(i) Content-Based Filtering
• Recommends items similar to what the user liked in the past.
• Uses item features such as genre, topic, keywords.
(ii) Collaborative Filtering
• Recommends items liked by similar users.
• “People like you also liked…”
(iii) Hybrid Systems
• Combination of content-based + collaborative filtering
• More accurate and powerful.
Explain how node failures are handled in Hadoop.
1. Introduction
• Hadoop is designed to work efficiently even if some computers (nodes) fail.
• It achieves this through fault tolerance — the ability to continue processing despite hardware or network
failures.
2. Types of Nodes in Hadoop
• NameNode → Manages metadata (file info, block locations).
• DataNode → Stores actual data blocks.
• JobTracker / TaskTracker (in older versions) or ResourceManager / NodeManager (in YARN) handle
processing.
3. Node Failures in Hadoop
Failures can occur in:
• DataNode
• NameNode
• Worker Nodes (TaskTracker / NodeManager)
Let’s see how each is handled
(i) DataNode Failure
• Hadoop continuously monitors DataNodes through heartbeats sent to the NameNode.
• If a DataNode stops sending heartbeats (default 10 minutes), it is considered failed.
• The NameNode:
o Marks that DataNode as dead.
o Replicates the lost data blocks from other DataNodes to maintain the required replication factor.
o Updates metadata to reflect new block locations.
Example:
If replication factor = 3 and one DataNode fails, Hadoop copies the missing block to another healthy DataNode.
(ii) NameNode Failure
• Earlier, Hadoop had a Single Point of Failure (SPOF) because only one NameNode existed.
• In new versions (Hadoop 2.x and above):
o Hadoop uses Secondary NameNode / Standby NameNode for high availability.
o When the active NameNode fails, the Standby NameNode takes over using the same metadata
(namespace image + edit logs).
This ensures no data loss and continuous operation.
(iii) TaskTracker / NodeManager Failure
• In MapReduce:
o Each worker node reports to JobTracker (old) or ResourceManager (YARN).
o If a node fails, its running tasks are rescheduled on another available node.
• Input data blocks are already replicated, so the job continues without data loss.
(iv) Block Replication
• Every file in HDFS is split into blocks (usually 128 MB).
• Each block is replicated (default 3 copies) across different DataNodes.
• If one node fails, the block can be read from another copy.
(v) Automatic Recovery
• Hadoop automatically:
o Detects node failure.
o Recreates lost replicas.
o Reassigns pending tasks.
o Ensures reliability and fault tolerance without manual intervention.
Map Reduce Execution Pipeline
1. Introduction
• MapReduce is a programming model used in Hadoop for processing large data sets in a distributed and
parallel manner.
• It works in two main steps —
Map phase → processes data
Reduce phase → aggregates the results
• The entire process is divided into six stages called the MapReduce execution pipeline.
2. Phases of MapReduce Execution
(i) Input Phase
• The process starts with input data stored in HDFS (Hadoop Distributed File System).
• Input data is in the form of large files such as text, CSV, or logs.
• The InputFormat class (like TextInputFormat) divides data into key-value pairs that the mapper can read.
Example:
A file containing lines of text.
• Key = line number
• Value = content of that line
(ii) Splitting Phase
• The input file is divided into small chunks called input splits.
• Each split is usually 128 MB or 256 MB in size (based on HDFS block size).
• Each split is processed by one Mapper in parallel.
Example:
A 512 MB file is divided into 4 splits → 4 mappers will process them.
Purpose: Enables parallel processing on multiple nodes.
(iii) Mapping Phase
• Each Mapper processes one split of data and performs the map() function.
• It converts input key-value pairs into intermediate key-value pairs.
Example (Word Count):
Input: "Hadoop is fast Hadoop is powerful"
Mapper Output:
(Hadoop, 1), (is, 1), (fast, 1), (Hadoop, 1), (is, 1), (powerful, 1)
Purpose: Extract and transform data for grouping.
(iv) Shuffling and Sorting Phase
• This is the intermediate stage between Map and Reduce.
• Hadoop automatically performs two important operations:
Shuffling:
• Transfers mapper output to reducers based on the key.
• Ensures that all values belonging to the same key go to the same reducer.
Sorting:
• Groups all intermediate data by key.
• For each key, all corresponding values are collected and sorted.
Example:
Mapper Output →
(Hadoop, 1), (is, 1), (Hadoop, 1), (is, 1)
After Sorting →
(Hadoop, [1,1]), (is, [1,1])
Purpose: Prepares data for reduction.
(v) Reducing Phase
• Each Reducer processes one group of keys with their list of values.
• The reduce() function aggregates or summarizes the data.
Example (Word Count):
Reducer Input: (Hadoop, [1,1])
Reducer Output: (Hadoop, 2)
Purpose: Combine data and produce the final result.
(vi) Output Phase
• The final output of the reducer is written back to HDFS using the OutputFormat class (like
TextOutputFormat).
• The output is stored as key-value pairs in files (usually in the /output directory).
Example:
Final Output File →
Hadoop 2
is 2
fast 1
powerful 1
Purpose: Store the final processed result in distributed storage.
.
Introduction – What is Big Data?
1. Big Data means extremely large and complex datasets that cannot be handled efficiently by traditional
databases (like RDBMS).
2. It comes from various sources → social media, IoT devices, sensors, banking, healthcare, e-commerce, etc.
3. Big Data is generally described using the “5 Vs” model (sometimes extended to 7Vs).
Five Characteristics of Big Data
1. Volume (Size of Data)
• Refers to the huge amount of data generated every second.
• Example: Facebook generates terabytes of data daily.
• Traditional systems cannot store/process this much data efficiently.
2. Velocity (Speed of Data Generation & Processing)
• Refers to the speed at which data is generated, collected, and processed.
• Example: Stock market trades, online transactions, IoT sensor data arrive in real-time.
• Big Data tools must handle streaming and real-time analytics.
3. Variety (Different Types of Data)
• Data is not only structured (tables) but also:
o Semi-structured (XML, JSON, logs)
o Unstructured (images, videos, social media posts)
• Big Data can handle all types of data sources.
4. Veracity (Trustworthiness of Data)
• Refers to the quality and reliability of data.
• Real-world data may be incomplete, noisy, or inconsistent.
• Big Data analytics uses data cleaning & validation techniques to ensure accuracy.