BIG DATA ANALYTICS - MODULE I

YASHWANTH S -1AM22CS231
VIJET HEGDE – 1AM22CS226
VEERESH MULIMANI -1AM22CS222
VINAY KUMAR B K -1AM22CS227
DISHANTH G-1AM23CS402

Hadoop Architecture

Hadoop's architecture follows a master-slave topology, designed for distributed storage and
processing of large datasets. The core components are:
1. Hadoop Distributed File System (HDFS):
• NameNode (Master):
Manages the file system namespace, regulates client access to files, and maintains metadata
(e.g., file names, directories, block locations). It does not store actual data.
• DataNode (Slave):
Stores the actual data in blocks and performs read/write operations as instructed by the
NameNode. DataNodes also report their block status to the NameNode periodically through
heartbeats.
 • Secondary NameNode (Helper):
Periodically merges the EditLog (transaction log) with the FsImage (metadata checkpoint) to
prevent the EditLog from growing too large and to provide a recent FsImage for recovery in
case of NameNode failure.
2. Yet Another Resource Negotiator (YARN):
• ResourceManager (Master):
Manages resources across the cluster, allocating resources (containers) to applications. It
consists of a Scheduler (allocates resources based on capacity and queues) and an
ApplicationManager (accepts job submissions and negotiates the first container for the
ApplicationMaster).
• NodeManager (Slave):
Runs on each slave node and manages resources on that specific node, launching and
monitoring containers as instructed by the ResourceManager.
• ApplicationMaster (Per-Application):
Manages the lifecycle of a single application, negotiating resources from the ResourceManager
and working with the NodeManager(s) to execute tasks within its allocated containers.
3. MapReduce (Programming Model):
• JobTracker (Deprecated in YARN):
In older Hadoop versions, the JobTracker was the master for MapReduce, managing job
execution and task tracking.
• TaskTracker (Deprecated in YARN):
In older Hadoop versions, the TaskTracker ran on slave nodes and executed individual map and
reduce tasks.
• With YARN:
MapReduce jobs are executed as applications managed by YARN, with the ApplicationMaster
handling the job execution and task coordination.
Interaction Flow:
• A client submits a job to the Hadoop cluster.
• If it's a MapReduce job, the ApplicationMaster for that job is launched via YARN.
• The ApplicationMaster requests resources from the ResourceManager.
• The ResourceManager allocates containers on various NodeManagers.
• The ApplicationMaster then coordinates with the NodeManagers to execute the map
and reduce tasks.
• HDFS is used for data storage and retrieval during the processing, with the NameNode
directing clients to the appropriate DataNodes.
 DATA WAREHOUSE ARCHITECTURE

A Data Warehouse is a system that combine data from multiple sources, organizes it under a
single architecture, and helps organizations make better decisions. It simplifies data handling,
storage, and reporting, making analysis more efficient. Data Warehouse Architecture uses a
structured framework to manage and store data effectively.
There are two common approaches to constructing a data warehouse:
• Top-Down Approach: This method starts with designing the overall data warehouse
architecture first and then creating individual data marts.
• Bottom-Up Approach: In this method, data marts are built first to meet specific
business needs, and later integrated into a central data warehouse.

1. Operational Systems & Data Sources

• These are the primary data sources.

• Examples: transactional databases, ERP systems, CRM systems, and flat files (like
CSV, Excel, logs).
• They generate raw operational data.
2. ETL (Extraction, Transformation, Loading)

• Extraction: Pulling data from multiple heterogeneous sources (databases, files, APIs).

• Transformation: Cleaning, filtering, aggregating, and converting the data into a

suitable format.

• Loading: Storing the transformed data into the Data Warehouse.

3. Data Warehouse

• A central repository where all integrated data is stored.

• Stores:

o Raw data (detailed records)

o Summary data (aggregated for faster queries)

o Metadata (information about the data, like source, format, meaning).

• Ensures consistency, quality, and availability of data.

4. Data Warehouse Applications

Once the data is stored, it is used for advanced analysis and decision-making:

• OLAP Analysis (Online Analytical Processing): Multidimensional analysis for

complex queries, trends, and patterns (e.g., sales by region, product, and time).

• Data Mining: Discovering hidden patterns, correlations, and predictions using

statistical and machine learning methods.

• Reporting: Generating dashboards, KPIs, and structured reports for business

intelligence.
 CLASSIFICATION OF ANALYTICS

[Link] Analytics

• What happened?

• Focus: Summarizes past data into meaningful insights.

• Examples:
o Sales reports

o Website traffic dashboards

o Monthly revenue trends

• Tools/Techniques: Reporting, dashboards, basic statistics.

2. Diagnostic Analytics

• Why did it happen?

• Focus: Explains the reasons behind past outcomes.

• Examples:

o Root cause analysis of sales drop

o Identifying why customer churn increased

• Tools/Techniques: Drill-down, data discovery, correlation analysis.

3. Predictive Analytics

• What is likely to happen?

• Focus: Uses historical data + statistical models/ML to forecast future trends.
 • Examples:

o Predicting customer churn

o Forecasting product demand

o Stock market prediction

• Tools/Techniques: Regression, machine learning, time-series analysis.

4. Prescriptive Analytics

• What should we do?

• Focus: Suggests optimal actions/decisions using predictions + optimization.

• Examples:

o Recommending personalized products to customers

o Optimizing delivery routes for logistics
o Pricing strategy suggestions

• Tools/Techniques: Optimization algorithms, simulation, AI-based decision-making.

NOSQL
1. What is NoSQL?

• NoSQL (Not Only SQL) refers to a category of database management systems that do
not follow the traditional relational model (RDBMS).

• Unlike relational databases that store data in tables (rows & columns), NoSQL
databases can store data in flexible formats such as key-value pairs, documents,
wide-columns, or graphs.

• They are designed for:

o High scalability

o Flexibility in schema

o Fast performance

o Handling unstructured / semi-structured data

2. Characteristics of NoSQL
1. Schema-less Data Storage
 o No fixed schema (you can add new fields anytime without altering existing
structure).

2. Horizontal Scalability

o Scales out by adding more servers (clusters), unlike RDBMS which often scales
vertically.

3. Distributed Architecture

o Data is spread across multiple nodes for availability and fault tolerance.

4. High Performance

o Optimized for fast read/write operations.

5. Handling Big Data

o Can store massive volumes of structured, semi-structured, or unstructured data.

3. Types of NoSQL Databases

1. Key-Value Stores

o Data stored as key-value pairs.

o Best for caching, session management, and real-time apps.

o Examples: Redis, Amazon DynamoDB.

2. Document Databases

o Data stored in JSON, BSON, or XML format.

o Flexible and great for hierarchical/unstructured data.

o Examples: MongoDB, CouchDB.

3. Column-Family Stores

o Data stored in columns instead of rows.

o Best for analytical queries on large datasets.

o Examples: Apache Cassandra, HBase.

4. Graph Databases

o Data stored as nodes and relationships (edges).

o Great for social networks, fraud detection, recommendation systems.

o Examples: Neo4j, Amazon Neptune.

4. NoSQL vs. RDBMS

Feature RDBMS (SQL) NoSQL

Data Structure Tables (rows & columns) Key-value, Document, Column, Graph

Schema Fixed schema Dynamic / schema-less

Scalability Vertical (scale-up) Horizontal (scale-out)

Best Use Case Structured data, ACID compliance Big data, high scalability, unstructured data

Examples MySQL, PostgreSQL, Oracle MongoDB, Cassandra, HBase, Redis

5. Role of NoSQL in Big Data Analytics

Big Data has 3Vs → Volume, Variety, Velocity.

NoSQL databases are designed to handle exactly these challenges:

1. Volume (Huge Data)

o Distributed storage across clusters handles petabytes of data efficiently.

o Example: Cassandra storing billions of rows across data centers.

2. Variety (Different Data Formats)

o Handles structured (logs), semi-structured (JSON/XML), and unstructured

(images, videos, text) data.

o Example: MongoDB storing JSON documents for analytics.

3. Velocity (High Speed)

o Real-time data ingestion and analytics possible.

o Example: Redis for caching live streaming analytics.

6. Use Cases of NoSQL in Big Data Analytics

• Real-time analytics: Retail (tracking user clicks and recommendations).

• IoT Data: Collecting sensor data from devices at massive scale.

• Social Media Analytics: Graph databases for relationships and trends.

• Fraud Detection: Detecting unusual patterns in financial transactions.

• Content Management: Handling large volumes of multimedia content.
7. Integration with Big Data Ecosystem

• Hadoop & HBase: HBase (a NoSQL column store) works on top of Hadoop HDFS.

• Spark & MongoDB/Cassandra: Spark can directly connect with these NoSQL DBs for
analytics.

• Data Lakes: NoSQL databases often serve as part of data lakes for flexible data storage.