UNIT-3

Distributing data storage and processing with hadoop frameworks:

New big data technologies such as Hadoop and Spark make it much easier to work with and control a cluster
of computers. Hadoop can scale up to thousands of com- puters, creating a cluster with petabytes of storage.
This enables businesses to grasp the value of the massive amount of data available.

Hadoop: aframework for storing and processing largedatasets

Hadoop is a framework that simplifies working with a cluster of [Link] aims to be all of the following things and more:
■ Reliable—By automatically creating multiple copies of the data and redeploying processing logic
in case of failure.
■ Faulttolerant—It detects faults and applies automatic recovery.
■ Scalable—Data and its processing are distributed over clusters of computers (horizontal scaling).
■ Portable—Installable on all kinds of hardware and operatingsystems.

The different components of Hadoop

At the heart of Hadoop we find
■ A distributed file system(HDFS)
■ A method to execute programs on a massive scale(MapReduce)
■ A system to manage the cluster resources(YARN)

MAPREDUCE: HOW HADOOP ACHIEVES PARALLELISM

Hadoop achieves parallelism using a programming model called MapReduce.
 Map phase: The input dataset is split into chunks, and each chunk is processed independently
and in parallel across different nodes.
 Reduce phase: The partial results are then sorted, combined, and aggregated into a final
output.
This parallel processing allows Hadoop to handle very large datasets efficiently.
However, MapReduce has limitations:
 After every computational step, the intermediate data is written to disk.
 This disk I/O makes it slow and expensive, especially for interactive queries or iterative
 UNIT-3

algorithms (like machine learning or graph processing).

1. Mapping phase
 Input data (like documents, logs, or records) is split into key–value pairs.
 For example, in a word count program, each word can be treated as a key, and the value is 1.
 At this stage, many duplicate keys may exist (e.g., the word red might appear in multiple
documents).
2. Reduce phase
 Works similarly to a SQL GROUP BY operation.
 The framework groups together all values associated with the same key.
 A user-defined reduce function then processes these groups to produce a final result.
 Example: If the task is to count colors, the reducer sums all values for each unique color and
returns the total count.
1. Data Ingestion & Exchange
 Sqoop: Moves (imports/exports) data between Hadoop (HDFS) and relational databases (e.g.,
MySQL, Oracle).
 Flume: Collects, aggregates, and transfers large amounts of log data into HDFS.
2. Coordination & Management
 Zookeeper: Provides coordination services — like synchronization, configuration management,
and leader election for distributed applications.
3. Data Storage
 HDFS (Hadoop Distributed File System): The main storage layer that stores large datasets
across multiple nodes in a fault-tolerant way.
4. Resource Management
 YARN (Yet Another Resource Negotiator): Manages cluster resources and job scheduling,
making sure tasks run efficiently across all nodes.
5. Processing Framework
 MapReduce: The core distributed processing framework that splits tasks into map and reduce
functions for parallel computation.
6. Higher-Level Data Processing Tools
 Pig: A high-level scripting language (Pig Latin) for data processing tasks.
 Hive: Provides a SQL-like query engine for Hadoop (good for analysts familiar with SQL).
 Mahout: Machine learning library built on top of Hadoop for clustering, classification, and
recommendations.
 HCatalog: A metadata management service, making schema and table information available for
other tools.
 Oozie: Workflow scheduler to manage Hadoop jobs (MapReduce, Pig, Hive, etc.).
 UNIT-3

 HBase: A NoSQL, column-oriented database that runs on top of HDFS for real-time read/write
access.

7. Monitoring & Security

 Ambari: Tool for provisioning, managing, and monitoring Hadoop clusters (with a web UI).
 Ranger: Provides security features like authentication, authorization, and auditing for Hadoop
components.
Casestudy: Assessing risk when loaning money:
Objective
The goal is to simulate a real-world loan risk assessment scenario using big data technologies. We’ll
see how Hadoop and Spark can process financial data to determine if loan applicants are high risk (likely
to default) or low risk (safe borrowers).
Even though in practice the datasets can be massive (millions of customer records), this case study uses a
smaller dataset for learning purposes, so it can run on a single machine.
Tools Used
1. Hortonworks Sandbox (VM)
o A preconfigured Hadoop ecosystem running inside VirtualBox.
o Provides HDFS, Hive, Pig, Spark, and other tools in a ready-to-use environment.
o Case study uses version 2.3.2.
2. Python Libraries
o Pandas → For data manipulation and cleaning (filtering, grouping, handling missing
values).
o pywebhdfs → For interacting with the Hadoop Distributed File System (HDFS) directly
from Python.
Step 1: Research Goal
The project has two main objectives:
1. Provide our manager with a dashboard – a clear, visual summary of the loan data (e.g., how
many loans are risky, repayment behavior, etc.).
2. Prepare data for others – so analysts can create their own dashboards using the cleaned,
structured dataset.
Step 2: Data Retrieval
 Source: Lending Club website (a peer-to-peer lending company that publishes loan datasets).
 Task: Download the dataset.
 Storage: Upload the raw dataset into HDFS (Hadoop Distributed File System) inside the
Hortonworks Sandbox.
Step 3: Data Preparation
 Use Spark to process and transform the raw data.
o Examples: handle missing values, normalize formats, extract important variables (credit
score, loan amount, repayment history).
 Store the transformed dataset into Hive tables for easy querying with SQL-like syntax.
Steps 4and 6: Exploration and Report Creation
 Use Qlik Sense (a visualization tool) to create interactive dashboards:
o Distribution of loans (by amount, grade, interest rate).
o Default rates across different borrower profiles.
o Risk factors (e.g., low income, high debt-to-income ratio).
 Managers and other teams can then interactively explore the data.

ACID: The Core Principles of Relational Databases

The reliability of relational databases is ensured by the ACID properties, which guarantee correctness
and stability of transactions.
1. Atomicity ("All or Nothing")
 UNIT-3

 A transaction is either fully completed or not executed at all.

 Prevents partial changes.
 Example: If a power failure happens in the middle of inserting a record, you won’t get a half-
written record — it simply won’t be stored.
2. Consistency (Data Integrity)
 Ensures that the database always follows defined rules and constraints.
 No transaction will violate conditions such as:
o Required fields must be filled.
o Data types must be correct (e.g., numeric field can’t contain text).
 Guarantees that only valid data enters the database.
3. Isolation (One Transaction at a Time on the Same Data)
 Transactions do not interfere with each other on the same data at the same moment.
 Ensures serial execution when conflicts might occur.
 Levels of isolation exist (from low to high):
o Low isolation (like Google Docs): Many users can edit simultaneously and see changes
instantly.
o High isolation (like MS Word): Only one user can edit at a time; others see only the
saved version.
Relational databases usually provide high isolation to protect data integrity.
4. Durability (Data Permanence)
 Once a transaction is committed, the data is permanently stored.
 Survives power outages or software crashes.
 Only physical hardware failure (e.g., disk damage) can destroy it.

CAP Theorem: The Problem with Databases on Many Nodes

When a database is distributed across multiple servers (nodes), maintaining ACID principles becomes
harder — especially consistency.
The CAP theorem states that in the presence of a network partition, a database can only guarantee two
out of three properties:
The Three Properties
1. Partition Tolerance (P)
o The system continues to operate even if communication between nodes is interrupted.
o This is a must for distributed systems because network failures are inevitable.
2. Availability (A)
o Every request to a working node gets a response, even if nodes can’t talk to each other.
o You don’t get errors or waiting; the system stays responsive.
3. Consistency (C)
o No matter which node you query, you’ll always see the same, up-to-date data.
o Requires strong synchronization between nodes.
Single-Node Databases
 Always Available (as long as the node runs).
 Always Consistent (no second node to disagree with).
Partitioning only becomes an issue in multi-node systems.
Example: Online Shop (Europe vs. USA servers)
Imagine Fritz (Germany) and Freddy (USA) both want the last available bronze octopus-shaped coffee
table. Suddenly, communication between servers breaks.
 Option 1 → Availability over Consistency (AP)
o Both servers keep selling products.
o Result: Both Fritz and Freddy buy the same last item → inconsistent data.
o Business must fix the conflict later (refund, coupon, etc.).
o Common in e-commerce, where staying online is more important than strict accuracy.
 UNIT-3

 Option 2 → Consistency over Availability (CP)

o Servers stop accepting orders until communication is restored.
o Result: Nobody accidentally double-buys the same item.
o Downside: Customers might leave → lost sales.
o Useful in cases like festival ticket sales (Tomorrowland), where selling too many tickets
would be catastrophic.
Key Takeaways
 Partition Tolerance is non-negotiable in distributed systems.
 So, in practice, systems must choose:
o AP (Available + Partition Tolerant): Prioritize responsiveness, allow temporary
inconsistencies (e.g., Amazon, eBay).
o CP (Consistent + Partition Tolerant): Prioritize correctness, allow temporary
downtime (e.g., banking, ticketing systems).

BASE: The Principles of NoSQL Databases

Unlike relational databases (which enforce ACID for strict consistency), many NoSQL databases use
BASE principles. BASE is more relaxed and is designed for high scalability, fault tolerance, and
availability in distributed systems.
1. Basically Available
 The system guarantees availability in the CAP theorem sense.
 As long as a node is running, it can respond to requests.
 UNIT-3

 If one node fails, others can take over (via replication/sharding).

 Example:
o Elasticsearch divides data into shards (mini-databases).
o Each shard can have replicas on other nodes.
o If one node crashes, replicas ensure the service stays available.
2. Soft State
 The system’s state may change over time, even without new input.
 This happens because nodes may temporarily hold inconsistent versions of data.
 Later, during conflict resolution, the system updates itself to achieve consistency.
 Example:
o Node A says a product status = “Available”,
o Node B says status = “Sold”,
o After synchronization, Node A updates itself to match Node B.
3. Eventual Consistency
 The system doesn’t guarantee immediate consistency, but promises that all nodes will become
consistent eventually.
 Temporary inconsistencies are tolerated.
 Conflict resolution can depend on business rules (first-come-first-served, cheapest shipping, etc.).
 Example:
o Fritz and Freddy both try to buy the last table.
o Both nodes might temporarily confirm the purchase.
o Once nodes reconnect, the system resolves the conflict (e.g., Fritz gets the table, Freddy
is refunded or given a coupon).

NoSQL Database Types

Relational Databases (for comparison)
 Use tables with rows and columns.
 Aim for normalization → store each piece of data only once.
 Handle relationships through linking tables (e.g., many-to-many between people and hobbies).
 Example schema:
o Person table → PersonID, Name, Age
 UNIT-3

o Hobby table → HobbyID, HobbyName

o PersonHobby table → PersonID, HobbyID
This structure works well but becomes complex and slow with huge, unstructured, or highly connected
data. That’s where NoSQL comes in.
1. Key-Value Stores
 Simplest type of NoSQL database.
 Data is stored as a key–value pair (like a Python dictionary or hash map).
 Strength: Fast lookups, great for caching or session storage.
 Limitation: No querying beyond keys.
 Examples: Redis, DynamoDB, Riak.


2. Documents tores
 Store data as documents (usually JSON, BSON, or XML).
 Each document contains flexible fields (schema-less).
 Strength: Good for semi-structured data (profiles, product catalogs).
 Limitation: Relationships between documents are harder to model than in relational DBs.
 Examples: MongoDB, CouchDB, Elasticsea


 UNIT-3

3. Column-Oriented Databases
 Store data in columns instead of rows.
 Designed for big data analytics → very fast aggregation and queries over large datasets.
 Strength: Efficient for queries like “sum of sales per region.”
 Limitation: Not ideal for transactional workloads.
 Examples: Apache Cassandra, HBase.

4. Graph Databases
 Store data as nodes (entities) and edges (relationships).
 Perfect for highly connected data like social networks, fraud detection, or recommendation
systems.
 Strength: Relationships are first-class citizens (no need for complex join tables).
 Examples: Neo4j, OrientDB, Amazon Neptune.

Case Study: What Disease is That?

The Problem
 People often Google symptoms to guess possible diseases.
 General practitioners (GPs) still rely on books/manuals → slow and limited.
 Pharmaceutical companies and advanced hospitals already have access to virtual diagnostic
systems built on protected medical data.
 If doctors worldwide had a simple disease-specific search engine, medical errors could be
reduced.
The Solution Idea
 Build a search engine for diseases using a NoSQL database (Elasticsearch).
 Elasticsearch is perfect because it:
o Stores unstructured text data (like Wikipedia articles).
o Makes it fast and easy to search for terms like “headache + nausea”.
Data Science Process Applied
1. Setting the Research Goal
o Create a searchable database of diseases.
 UNIT-3

o Make it useful for doctors and patients.

2. Data Collection
o Source: Wikipedia articles about diseases (freely available).
o In real-world systems: medical journals, hospital records, pharma databases.
3. Data Preparation
o Wikipedia data isn’t perfect → needs cleaning.
o Apply techniques like:
 Removing stopwords (the, and, is, etc.).
 Tokenizing text (splitting into words).
 Structuring disease info (symptoms, causes, treatments).
4. Data Exploration (Final Product)
o In this case, exploration = making the data searchable.
o Use Elasticsearch to index all diseases so you can query by symptom.
5. Data Modeling (Not Focused Here)
o Normally, advanced systems would use topic modeling or ML models to cluster similar
diseases.
o Here, we only rely on document-term matrices (words vs documents) to enable search.
6. Presenting Results
o End goal: a searchable interface (like a website or app).
o For this study:
 No full UI is built.
 Instead, present results visually as a word cloud, showing important keywords
for each disease.