BIG DATA ANALYTICS

Complete Study Notes

Unit 1 | Unit 2 | Unit 3

Hadoop • MapReduce • Hive • Sqoop • HBase • NoSQL

Prepared for exam revision — All units covered with examples, analogies, and commands
UNIT – 1 : Hadoop Ecosystem & Big Data Fundamentals
Q1. Architecture of the Hadoop Ecosystem

■ What is Hadoop?
Apache Hadoop is an open-source framework designed to store and process enormous amounts of data
across clusters of computers using simple programming models. It is built to scale from a single server to
thousands of machines, each offering local computation and storage. Hadoop is the backbone of the Big
Data world.

■ Core Components
• HDFS (Hadoop Distributed File System): The primary storage unit. Large files are broken into blocks
(default 128 MB) and distributed across multiple DataNodes in the cluster. It ensures fault tolerance by
replicating each block across 3 nodes by default.
• MapReduce: The batch data processing engine. It splits computation into two phases — Map (parallel
processing of data splits) and Reduce (aggregating results) — enabling distributed analysis across
hundreds of machines.
• YARN (Yet Another Resource Negotiator): Acts as the OS of Hadoop. It manages cluster resources
(CPU, RAM) and schedules jobs. It consists of the Resource Manager (global scheduler), Node
Manager (per-machine agent), and Application Master (per-job monitor).
• Hadoop Common: Shared Java libraries and utilities required by all other Hadoop modules to function.

■ Ecosystem Tools
• HBase: A distributed NoSQL column-oriented database that runs on top of HDFS. Ideal for real-time
read/write access on billions of rows.
• Hive: A data warehousing tool that lets users write SQL-like queries (HiveQL) which get converted
internally into MapReduce jobs. Great for analytics on large structured datasets.
• Pig: A high-level scripting platform using Pig Latin language. Developers can write complex data
transformation logic without coding in Java.
• Sqoop: Short for SQL-to-Hadoop. Transfers bulk data bidirectionally between RDBMS (MySQL,
Oracle) and Hadoop (HDFS/Hive/HBase).
• Flume: Collects, aggregates, and moves large amounts of log/event data from various sources into
HDFS in real time.
• Oozie: A workflow scheduler to manage chains of Hadoop jobs (e.g., run Sqoop → then MapReduce
→ then Hive in sequence).
• Zookeeper: Coordinates and synchronizes distributed services, acting as a reliable configuration and
naming registry.
• Mahout: A machine learning library that runs on top of Hadoop for scalable algorithms like clustering
and recommendation.
• Avro: A data serialization framework to efficiently transfer data between Hadoop systems.
 ■ Simple Analogy: Hadoop is like a mega factory. HDFS is the warehouse storing raw materials (data).
YARN is the supervisor assigning tasks to workers. MapReduce is the assembly line that processes
materials in parallel. Hive/Pig/Sqoop are specialist machines for specific jobs.

Q2. The 4 V's of Big Data

Big Data is too large and complex for traditional software to handle. It is described by four key
characteristics known as the '4 V's':

■ 1. Volume
Refers to the massive scale of data generated every second from social media, IoT devices, sensors,
business transactions, and more. Volume is measured in Petabytes (PB), Exabytes (EB), Zettabytes (ZB),
and even Yottabytes (YB). For example, Facebook generates over 500 TB of data daily, and Google
processes over 20 PB per day. Traditional databases cannot store or query data at this scale — that's
where Hadoop/HDFS excels.

■ 2. Velocity
Refers to the incredible speed at which new data is generated and must be processed. In the era of stock
markets, online fraud detection, and social media feeds, data must be captured, processed, and acted
upon in real time or near-real time. Twitter generates 500 million tweets per day. Velocity requires
streaming platforms like Apache Kafka and Spark Streaming.

■ 3. Variety
Refers to the many different types and formats of data:
■ Structured data: Rows and columns in RDBMS tables, Excel spreadsheets — easy to analyze.
■ Semi-structured data: JSON, XML, CSV, emails — partially organized.
■ Unstructured data: Images, videos, audio, PDFs, social media posts — hardest to analyze, but makes
up ~80% of enterprise data.

■ 4. Veracity
Refers to the trustworthiness, accuracy, and quality of the data. Not all data collected is clean — it may
contain noise, inconsistencies, missing values, or duplicates. Veracity ensures data is filtered, cleaned,
and validated before analysis so that business decisions are based on reliable insights rather than
garbage data.

■ Key Point: Some sources also mention a 5th V — Value: the ultimate goal of Big Data —
extracting meaningful, actionable business insights from the data collected.

■ Simple Analogy: Think of Big Data like a river flood. Volume = how much water. Velocity = how fast it
flows. Variety = it carries water, mud, rocks, fish, and branches. Veracity = how clean and safe the water
actually is.

Q3. Characteristics and Applications of Big Data

■ Characteristics
Beyond the 4 V's, Big Data has additional defining characteristics:
• Complexity: Data comes from multiple sources (databases, social media, IoT, logs) and must be
integrated, cross-referenced, and managed carefully. This requires sophisticated data governance.
• Variability: Data flows are inconsistent. Some days see peak loads (e.g., Black Friday online
shopping); systems must scale to handle these spikes dynamically.
• Visualization: Big Data analytics must present results through intuitive charts, graphs, and dashboards
so decision-makers can understand trends without seeing raw numbers.

■ Real-World Applications
• Healthcare: Hospitals use Big Data to analyze patient records, predict disease outbreaks, optimize
treatment plans, and accelerate drug discovery. For example, IBM Watson Health analyzes millions of
medical papers to assist oncologists.
• E-Commerce & Retail: Amazon uses Big Data to power its recommendation engine — analyzing
every click, search, and purchase to suggest products you're most likely to buy next.
• Banking & Finance: Banks process millions of transactions daily. Big Data enables real-time fraud
detection (flagging suspicious transactions instantly), credit scoring, and risk analysis. NYSE
generates ~1 TB of trade data every session.
• Transportation & GPS: Google Maps and Uber analyze millions of GPS signals, traffic patterns, and
historical data to calculate the fastest routes and predict demand surges in real time.
• Social Media: Facebook, Instagram, and YouTube process petabytes of images, videos, and
interactions daily to personalize feeds, target ads, and detect harmful content.
• Manufacturing & IoT: Smart factories use sensor data from machines to predict equipment failures
before they happen (predictive maintenance), reducing downtime and saving millions.
• Education: Learning platforms like Coursera analyze student performance data to personalize course
content and identify at-risk students early.

■ Simple Analogy: Big Data is like a city's CCTV network. Individually, each camera is useless. But
combined and analyzed with AI, you can track traffic flow, detect crimes in real time, and plan better city
infrastructure.
Q4. HDFS Architecture and Commands

■ Overview
HDFS (Hadoop Distributed File System) is the primary storage system of Hadoop. It is designed to store
extremely large files reliably across many machines. HDFS follows a Master-Slave architecture and
splits files into blocks (default size: 128 MB) which are distributed and replicated across the cluster.

■ Key Components
• NameNode (Master): The brain of HDFS. It maintains the filesystem namespace (directory tree) and
tracks where every block of every file is stored. It holds two critical files:
■ FsImage — a complete snapshot of the filesystem at a point in time.
■ EditLogs — a log of all recent changes to the filesystem since the last snapshot.
There is only ONE active NameNode. If it fails, the entire cluster becomes unavailable — this is called the
Single Point of Failure (SPOF) problem, addressed in Hadoop 2.x with High Availability (HA) using a
Standby NameNode.

• DataNode (Slave): Stores the actual data blocks on local disks. A cluster can have hundreds or
thousands of DataNodes. They continuously send heartbeat signals every 3 seconds to the
NameNode to confirm they are alive. If a DataNode misses heartbeats, the NameNode marks it as
dead and re-replicates its blocks elsewhere.

• Secondary NameNode: This is NOT a backup of the NameNode (despite the name). Its job is to
periodically merge the FsImage and EditLogs so that the EditLog doesn't grow too large. This prevents
recovery from taking forever if the NameNode restarts.

■ Replication & Rack Awareness

HDFS replicates each block across 3 DataNodes by default (configurable via [Link]). Rack
Awareness ensures blocks are placed smartly:
■ Replica 1: On the same rack as the writing client.
■ Replica 2: On a different rack.
■ Replica 3: On another node within the second rack.
This ensures data survives even if an entire network rack goes down.

■ Read & Write Workflow

Write: Client contacts NameNode → NameNode assigns DataNodes → Client writes block to first
DataNode → First DataNode pipelines the block to second → second to third → All acknowledge back →
NameNode updates metadata.
Read: Client asks NameNode for block locations → NameNode returns the list of DataNodes → Client
reads directly from the nearest DataNode.

■ Important HDFS Commands

hdfs dfs -ls / # List files in root directory
hdfs dfs -mkdir /mydata # Create a new directory
hdfs dfs -put [Link] /mydata/ # Upload file from local to HDFS
hdfs dfs -get /mydata/[Link] ./ # Download file from HDFS to local
hdfs dfs -cat /mydata/[Link] # Print file content to screen
hdfs dfs -cp /src/[Link] /dest/ # Copy file within HDFS
hdfs dfs -mv /old/path /new/path # Move/rename file in HDFS
hdfs dfs -rm /mydata/[Link] # Delete a file
hdfs dfs -rm -r /mydata/ # Delete directory recursively
hdfs dfs -count /mydata/ # Count dirs, files, and bytes
hdfs dfs -touchz /mydata/[Link] # Create an empty 0-byte file
hdfs dfs -chmod 755 /mydata/[Link] # Change file permissions
hdfs dfs -chown user:group /mydata/ # Change file ownership
hdfs dfs -df -h / # Show disk space usage (human readable)
hdfs dfs -du -s /mydata/ # Show size of directory

■ Simple Analogy: HDFS is like a library chain. The NameNode is the central catalog computer that
knows every book's location in every branch. DataNodes are the branches storing actual books. Each
book is photocopied and kept in 3 different branches (replication) so even if one branch burns down, you
can still find the book.
 UNIT – 2 : MapReduce, Hive & SQL Commands
Q1. MapReduce Architecture with Example

■ What is MapReduce?
MapReduce is Hadoop's core processing framework for distributed data processing. It breaks a large task
into smaller sub-tasks, processes them in parallel across multiple machines, and then combines the
results. It uses two main functions: Map() and Reduce(). The key principle is: move the computation to
where the data is, rather than moving data to computation.

■ Key Daemons
• JobTracker (Master): Accepts job requests from clients, splits jobs into tasks, assigns tasks to
TaskTrackers, monitors progress, and handles failures.
• TaskTracker (Slave): Runs on each DataNode. Executes Map and Reduce tasks assigned by
JobTracker and reports progress back.

■ Execution Phases (in order)

• 1. Input Splitting: HDFS data is divided into logical InputSplits. Each split is typically one HDFS block
(128 MB). One Mapper is spawned per split.
• 2. RecordReader: Converts each split into key-value pairs. For text files, the key is the byte offset and
the value is the line of text.
• 3. Mapper: The user-defined Map function processes each key-value pair and outputs intermediate
key-value pairs stored on local disk (NOT HDFS).
• 4. Combiner (Optional): A mini-reducer that runs locally on each Mapper's output. It performs partial
aggregation to reduce the volume of data sent across the network. This improves performance
significantly.
• 5. Partitioner: Decides which Reducer receives which intermediate key. Default: hash(key) %
number_of_reducers. Ensures all values for the same key go to the same Reducer.
• 6. Shuffle & Sort: Data is transferred from Mappers to Reducers (Shuffle) over the network and
automatically sorted by key (Sort), so each Reducer receives all values for a given key in sorted order.
• 7. Reducer: The user-defined Reduce function receives (key, list_of_values) and outputs final
aggregated results.
• 8. OutputFormat: Writes the Reducer's final output to HDFS using RecordWriter.

■ Word Count Example

Input text: "Hello is hello is Sunny is Sunny is Sunny"

MAPPER OUTPUT:
(Hello, 1), (is, 1), (hello, 1), (is, 1), (Sunny, 1), (is, 1), (Sunny, 1), (is, 1),
(Sunny, 1)

AFTER SHUFFLE & SORT (grouped by key):

 (Hello, [1,1]) (is, [1,1,1,1]) (Sunny, [1,1,1])

REDUCER OUTPUT:
(Hello, 2) (is, 4) (Sunny, 3)

■ Fault Tolerance in MapReduce

If a TaskTracker fails, the JobTracker detects the missed heartbeats and re-schedules the failed tasks on
another available TaskTracker. If the JobTracker itself fails, the entire job must be restarted (in YARN, the
Application Master handles per-job recovery more gracefully).

■ Simple Analogy: MapReduce is like splitting an encyclopedia word-count task among 100 friends.
Each friend counts words on their assigned pages (Mapper). A team leader then collects all counts,
sorts them alphabetically, and sums up the totals (Reducer). You get the final answer without any one
person reading the whole book.

Q2. Hive Architecture

■ What is Apache Hive?

Apache Hive is a data warehouse infrastructure built on top of Hadoop. It enables reading, writing, and
managing large datasets stored in HDFS using a SQL-like query language called HiveQL (HQL). Hive
internally converts HQL queries into MapReduce jobs, making it accessible to analysts who know SQL
without needing to write Java code. Hive is optimized for batch processing and analytical queries (OLAP),
not real-time transactional queries (OLTP).

■ Hive Architecture Components

• 1. Hive Clients: Users and applications connect to Hive through:
■ Hive CLI (Command Line Interface) — for direct terminal interaction.
■ Hive Web UI — a browser-based interface.
■ JDBC/ODBC Drivers — for connecting BI tools like Tableau, PowerBI.
■ Thrift Client — for programmatic access from Java, Python, C++ apps.

• 2. Hive Services:
■ Hive Server 2 (HiveServer2): Accepts client connections, supports multi-client concurrency and
authentication. Replaces the older Hive Thrift Server.
■ MetaStore: The heart of Hive. Stores schema metadata — database names, table names, column
names, data types, partitions, and HDFS file locations. By default uses Derby database but is typically
configured with MySQL or PostgreSQL in production.
■ Hive Driver: Receives HQL queries from clients and manages the lifecycle of a query.
■ Compiler: Parses the query, validates against MetaStore metadata, performs semantic analysis,
optimizes the logical plan, and generates a Physical Plan (DAG of MapReduce/Tez jobs).
■ Execution Engine: Executes the Physical Plan. By default uses MapReduce, but can use Apache Tez
(faster DAG-based execution) or Apache Spark.

• 3. Hive Storage: All table data lives in HDFS under the Hive warehouse directory
(/user/hive/warehouse/ by default). Hive supports multiple file formats: TextFile, SequenceFile, ORC
 (Optimized Row Columnar — best for analytics), and Parquet.

■ Hive Tables: Internal vs External

Internal (Managed) Tables: Hive owns the data. When you DROP the table, both metadata and data in
HDFS are deleted.
External Tables: Hive only manages metadata. The data lives in a user-specified HDFS location. When
you DROP the table, only metadata is removed — data in HDFS remains safe. This is the preferred
approach in production.

■ Hive Partitioning & Bucketing

Partitioning: Divides a table into sub-directories based on a column value (e.g., year, country). Only
relevant partitions are scanned during queries, dramatically improving performance.
Bucketing: Further divides each partition into a fixed number of files (buckets) based on the hash of a
column. Useful for joins and sampling.

■ Simple Analogy: Hive is Google Translate for Hadoop. Analysts type in familiar SQL (source
language). Hive translates it into complex MapReduce code (target language) that Hadoop understands,
runs the job, and returns a human-readable result — all without the analyst needing to know Java.

Q3. DDL, DCL, and DML Commands in Hive

■ DDL – Data Definition Language

DDL commands define and manage the structure of database objects. They affect schemas, tables, and
partitions — not the data inside them. DDL operations are auto-committed.

-- Create a new database

CREATE DATABASE IF NOT EXISTS university
COMMENT 'University database'
LOCATION '/user/hive/university';

-- List all databases

SHOW DATABASES;

-- Describe database details

DESCRIBE DATABASE university;

-- Use a database
USE university;

-- Create a table
CREATE TABLE student (
Student_ID INT,
Student_Name STRING,
Department STRING,
CGPA FLOAT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
 STORED AS TEXTFILE;

-- Create External table

CREATE EXTERNAL TABLE ext_student (
Student_ID INT, Student_Name STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/user/data/students/';

-- Alter table (add a column)

ALTER TABLE student ADD COLUMNS (Email STRING);

-- Truncate table (removes all data, keeps structure)

TRUNCATE TABLE student;

-- Drop table
DROP TABLE IF EXISTS student;

-- Drop database
DROP DATABASE IF EXISTS university CASCADE;

■ DML – Data Manipulation Language

DML commands manipulate the data inside existing table structures — inserting, reading, updating, and
deleting records.

-- Load data from local file into table

LOAD DATA LOCAL INPATH '/home/user/[Link]'
INTO TABLE student;

-- Load from HDFS (moves the file)

LOAD DATA INPATH '/hdfs/path/[Link]'
OVERWRITE INTO TABLE student;

-- Insert single row

INSERT INTO TABLE student VALUES (101, 'Alice', 'CSE', 9.2);

-- Insert result of a SELECT

INSERT INTO TABLE top_students
SELECT * FROM student WHERE CGPA > 8.5;

-- Select with conditions

SELECT Student_Name, CGPA
FROM student
WHERE Department = 'CSE'
ORDER BY CGPA DESC
LIMIT 10;

-- Update a record (requires transactional table - ORC format)

UPDATE student SET CGPA = 9.5 WHERE Student_ID = 101;

-- Delete a record
 DELETE FROM student WHERE Student_ID = 101;

■ DCL – Data Control Language

DCL commands control access permissions to database objects. They determine who can do what with
data.

-- Grant SELECT and INSERT permissions to a user

GRANT SELECT, INSERT ON TABLE student TO USER analyst1;

-- Grant all privileges on a database

GRANT ALL ON DATABASE university TO USER admin1;

-- Revoke INSERT permission

REVOKE INSERT ON TABLE student FROM USER analyst1;

-- Show grants on a table

SHOW GRANT USER analyst1 ON TABLE student;

■ Simple Analogy: DDL = Architect (draws blueprints, builds rooms). DML = Resident (moves furniture
in, rearranges it, throws things away). DCL = Security Guard (decides who gets a key to which room).
UNIT – 3 : Sqoop, SQL vs NoSQL, HBase & CRUD
Q1. Apache Sqoop – SQL to Hadoop Data Transfer

■ What is Sqoop?
Apache Sqoop (SQL + Hadoop) is a command-line tool designed for bulk data transfer between Hadoop
and structured datastores such as relational databases (MySQL, Oracle, PostgreSQL, SQL Server). It
uses JDBC to connect to databases and parallelizes data movement using MapReduce (map-only jobs —
no reducer needed).

■ Sqoop Import — RDBMS → Hadoop

Sqoop Import pulls data from an RDBMS table and stores it in HDFS, Hive, or HBase. It automatically
infers the table schema, generates Java code (a 'record class'), divides data into partitions, and launches
parallel mappers to fetch data simultaneously via JDBC.
# Basic import from MySQL to HDFS
sqoop import \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--table employees \
--target-dir /user/hadoop/employees \
--num-mappers 4

# Import with specific columns and WHERE filter

sqoop import \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--table employees \
--columns "emp_id,name,salary,dept" \
--where "salary > 50000" \
--target-dir /user/hadoop/high_earners

# Import directly into Hive table

sqoop import \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--table employees \
--hive-import \
--hive-table hive_employees

# Incremental import (only new rows since last run)

sqoop import \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--table employees \
--incremental append \
--check-column emp_id \
--last-value 5000
■ Sqoop Export — Hadoop → RDBMS
Sqoop Export reads data from HDFS and pushes it into an RDBMS table. Useful after performing analytics
in Hadoop and needing to write results back to a production database for applications to consume.
# Export from HDFS to MySQL
sqoop export \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--table result_table \
--export-dir /user/hadoop/analysis_results \
--input-fields-terminated-by ','

■ Sqoop Import-All-Tables
# Import every table from a database in one command
sqoop import-all-tables \
--connect jdbc:mysql://localhost:3306/company_db \
--username root --password secret \
--warehouse-dir /user/hadoop/company_data

■ Key Point: Sqoop uses parallel mappers (--num-mappers / -m) to speed up transfers. Setting
-m 1 forces a single mapper, which is slower but avoids primary key conflicts during parallel
splits.

■ Simple Analogy: Sqoop is like a fleet of cargo trucks. Each truck (mapper) picks up a portion of
goods (rows) from the supplier's warehouse (RDBMS) and delivers them to your mega-warehouse
(HDFS). All trucks work simultaneously, making the transfer lightning-fast.

Q2. Difference Between SQL and NoSQL

■ Overview
SQL (Structured Query Language) databases are traditional relational databases that store data in tables
with fixed schemas. NoSQL (Not Only SQL) databases are modern, non-relational databases designed to
handle massive scale, flexible data formats, and high-velocity workloads that SQL databases struggle
with.

Feature SQL (RDBMS) NoSQL

Data Model Tables (rows & columns) Key-Value, Document, Column, Graph

Schema Fixed, predefined schema Dynamic / Schema-free

Query Language Standardized SQL Varies by database (no standard)

Vertical (Scale-Up) Horizontal (Scale-Out)

Scalability
Bigger machine More machines

ACID Compliance Fully ACID compliant BASE model (Eventually Consistent)

Joins Complex multi-table joins Avoids joins (denormalized)

Best For Structured data, complex queries Big Data, real-time, unstructured data
Examples MySQL, Oracle, PostgreSQL, MSSQL HBase, MongoDB, Cassandra, Redis, Neo4J

Relationships Foreign keys, referential integrity Embedded documents or references

Performance Slower at petabyte scale High performance at massive scale

■ 4 Types of NoSQL Databases

• Key-Value Store: Simplest model. Data stored as key → value pairs (like a dictionary). Extremely fast
for lookups. Examples: Redis, DynamoDB, Riak.
• Column-Oriented (Wide Column): Data stored in columns rather than rows, grouped into column
families. Ideal for sparse data and analytical queries. Examples: Apache HBase, Apache Cassandra.
• Document-Oriented: Stores semi-structured data as documents (JSON/BSON/XML). Each document
can have a different structure. Examples: MongoDB, CouchDB.
• Graph-Based: Models data as nodes (entities) and edges (relationships). Ideal for social networks,
recommendation engines, fraud detection. Examples: Neo4J, Amazon Neptune.

■ Simple Analogy: SQL is like a rigid Excel spreadsheet — every row must fit the exact same columns.
NoSQL is like a flexible folder where you can drop any type of document without caring about format.
When the folder gets too full, you just add more filing cabinets (horizontal scaling) instead of buying a
bigger one.

Q3. HBase Architecture

■ What is HBase?
Apache HBase is an open-source, distributed, column-oriented NoSQL database modeled after Google's
Bigtable. It runs on top of HDFS and provides real-time random read/write access to petabytes of data
— something HDFS alone cannot offer. HBase is ideal when you need to look up specific rows instantly
(like a user profile lookup) in a massive dataset.

■ HBase Data Model

HBase organizes data differently from relational databases:
■ Table: A collection of rows.
■ Row Key: Unique identifier for each row. Rows are sorted lexicographically by row key — choosing a
good row key is critical for performance.
■ Column Family: A group of related columns declared at table creation time. Example: 'personal_info',
'academic_info'. Column families are stored together on disk.
■ Column Qualifier: Specific columns within a family. Example: 'personal_info:name',
'personal_info:age'.
■ Cell: The intersection of row key, column family, and column qualifier. Each cell can store multiple
versions of data with timestamps.

■ Architecture Components
• HMaster (Master Server): Manages the cluster. Handles DDL operations (CREATE/DROP table),
assigns regions to Region Servers, manages load balancing, and coordinates failover. There can be
multiple HMasters in an HA setup (active + standby).
 • Region Servers (Slave Nodes): The workhorses of HBase. Each Region Server manages one or
more Regions. They handle all client read/write requests. Inside a Region Server:
■ Region: A contiguous range of rows stored together. As a region grows too large, it automatically splits
into two smaller regions (auto-sharding).
■ MemStore: An in-memory write buffer for each column family. All new writes go to MemStore first (very
fast). When MemStore fills up, data is flushed to disk as an HFile.
■ HFile (StoreFile): Immutable, sorted data files stored on HDFS. Multiple HFiles are periodically merged
through Compaction to improve read performance.
■ BlockCache: An in-memory read cache for frequently accessed data blocks. Reduces disk reads for
hot data.
■ WAL (Write-Ahead Log): Before any write is committed to MemStore, it's recorded in the WAL on
HDFS. This ensures data recovery if a Region Server crashes before the MemStore is flushed.

• Zookeeper: Acts as the coordination service. Maintains the active HMaster's address, stores ROOT
and META table locations (which tell clients which Region Server handles which row key range),
monitors cluster health, and distributes cluster state information. Clients communicate with HBase by
first contacting Zookeeper.

• HDFS: The underlying storage layer where all HFiles (actual data) and WALs are permanently stored.
HBase relies entirely on HDFS for durability and fault tolerance.

■ HBase Shell Commands

# Start HBase shell
hbase shell

# Create a table with column families

create 'student', 'personal', 'academic'

# Insert data (Put operation)

put 'student', 'row1', 'personal:name', 'Alice'
put 'student', 'row1', 'personal:age', '21'
put 'student', 'row1', 'academic:grade', 'A'

# Read a single row

get 'student', 'row1'

# Scan entire table

scan 'student'

# Scan with filter

scan 'student', {LIMIT => 5}

# Delete a specific cell

delete 'student', 'row1', 'personal:age'

# Delete entire row

deleteall 'student', 'row1'

# List all tables

 list

# Describe table structure

describe 'student'

# Drop a table (must disable first)

disable 'student'
drop 'student'

■ Simple Analogy: HBase is like a giant post office. HMaster is the postmaster assigning delivery
zones. Region Servers are postal workers handling their zones. MemStore is the worker's sorting tray
(fast but temporary). HFiles are permanent sealed boxes in the vault (HDFS). Zookeeper is the dispatch
radio keeping everyone coordinated.

Q4. CRUD Operations Explained

■ What is CRUD?
CRUD stands for Create, Read, Update, Delete — the four fundamental operations that any persistent
data storage system must support. Every database interaction, whether SQL, NoSQL, REST API, or file
system, can be mapped to one of these four operations. CRUD provides the complete lifecycle
management of data.

■ C – Create
Create operations add new data to the database. In SQL this involves two steps: first defining the structure
(DDL), then inserting data (DML).
-- SQL: Create structure
CREATE TABLE employees (
emp_id INT PRIMARY KEY AUTO_INCREMENT,
name VARCHAR(100) NOT NULL,
department VARCHAR(50),
salary DECIMAL(10,2),
hire_date DATE
);

-- SQL: Insert single record

INSERT INTO employees (name, department, salary, hire_date)
VALUES ('Alice Smith', 'Engineering', 85000.00, '2023-01-15');

-- SQL: Insert multiple records at once

INSERT INTO employees (name, department, salary, hire_date) VALUES
('Bob Jones', 'Marketing', 65000, '2023-03-01'),
('Carol White', 'HR', 55000, '2023-04-10');

-- HBase Create:
put 'employees', 'emp001', 'info:name', 'Alice Smith'
put 'employees', 'emp001', 'info:dept', 'Engineering'

■ R – Read
Read operations retrieve data from the database. SELECT is the most powerful SQL statement,
supporting filtering, joining, grouping, aggregation, and sorting.
-- Basic SELECT
SELECT * FROM employees;

-- Select specific columns with WHERE filter

SELECT emp_id, name, salary
FROM employees
WHERE department = 'Engineering'
AND salary > 70000;

-- Aggregate functions
SELECT department,
COUNT(*) AS headcount,
AVG(salary) AS avg_salary,
MAX(salary) AS top_salary
FROM employees
GROUP BY department
HAVING COUNT(*) > 2
ORDER BY avg_salary DESC;

-- JOIN two tables

SELECT [Link], d.dept_name, [Link]
FROM employees e
INNER JOIN departments d ON [Link] = d.dept_id
WHERE [Link] > 60000;

-- HBase Read:
get 'employees', 'emp001'
scan 'employees', {COLUMNS => ['info:name', 'info:salary']}

■ U – Update
Update operations modify existing records. Always use a WHERE clause to target specific rows —
omitting WHERE updates ALL rows in the table.
-- Update a specific record
UPDATE employees
SET salary = 90000, department = 'Senior Engineering'
WHERE emp_id = 1;

-- Update with calculation

UPDATE employees
SET salary = salary * 1.10
WHERE department = 'Engineering'
AND hire_date < '2022-01-01';

-- Conditional update
UPDATE employees
SET department = CASE
WHEN salary > 80000 THEN 'Senior'
WHEN salary > 60000 THEN 'Mid'
ELSE 'Junior'
 END;

-- HBase Update (just put with same row key — overwrites):

put 'employees', 'emp001', 'info:salary', '90000'

■ D – Delete
Delete operations remove records. Like UPDATE, always use WHERE in production — a DELETE without
WHERE removes every row in the table permanently.
-- Delete specific records
DELETE FROM employees
WHERE emp_id = 3;

-- Delete with condition

DELETE FROM employees
WHERE department = 'Intern'
AND hire_date < '2020-01-01';

-- Delete all rows but keep structure (faster than DELETE without WHERE)
TRUNCATE TABLE employees;

-- Drop entire table (removes structure AND data)

DROP TABLE employees;

-- HBase Delete:
delete 'employees', 'emp001', 'info:salary' -- delete specific cell
deleteall 'employees', 'emp001' -- delete entire row

■ CRUD in Different Technologies

Operation SQL HBase Shell REST API (HTTP) MongoDB

Create INSERT INTO put POST insertOne()

Read SELECT get / scan GET find()

Update UPDATE SET put (overwrite) PUT / PATCH updateOne()

Delete DELETE FROM delete / deleteall DELETE deleteOne()

■ Simple Analogy: CRUD is the full life of a contact in your phone. Create = adding a new friend. Read
= searching their name. Update = editing their number when they change it. Delete = removing them
when you part ways.

— End of BDA Study Notes — Good Luck on Your Exam! —