Why Big Data Frameworks Are Needed
In earlier chapters:
Small data → fit in RAM
Medium data → fit on one machine’s disk
Big data → too large for a single computer
When data becomes terabytes or petabytes, we must:
Store it across many machines
Process it in parallel
Handle machine failures automatically
That is where distributed frameworks come in.
2️⃣ Hadoop: Distributed Storage + Processing
🔹 What is Hadoop?
Apache Hadoop is a framework that:
Stores massive data across many machines
Processes data in parallel
Handles hardware failures automatically
Scales horizontally (add more machines)
🔹 Core Components of Hadoop
Component Purpose
HDFS Distributed storage
MapReduce Distributed computation
YARN Resource management
🔹 HDFS (Hadoop Distributed File System)
Instead of storing:
loan_data.csv
on one computer, HDFS:
Splits file into blocks
Distributes blocks across multiple machines
Replicates blocks (usually 3 copies)
So if one machine crashes → no data loss.
3️⃣ MapReduce: How Hadoop Processes Data
MapReduce has two main steps:
🔹 Example: Counting Toy Colors
Imagine files:
File1:
Green, Blue
Blue, Orange
File2:
Green, Red
Blue, Orange
Step 1 — MAP Phase
Each line becomes key-value pairs:
Green → 1
Blue → 1
Blue → 1
Orange → 1
Green → 1
Red → 1
Blue → 1
Orange → 1
Duplicates are allowed.
Step 2 — SHUFFLE + SORT
Group by key:
Blue → 1,1,1
Green → 1,1
Orange → 1,1
Red → 1
Step 3 — REDUCE Phase
Aggregate:
Blue → 3
Green → 2
Orange → 2
Red → 1
🔹 Problem With MapReduce
After every step, Hadoop:
Writes data to disk
Reads it again
Writes again
For iterative or interactive work → slow
4️⃣ Spark: Faster Distributed Processing
To solve MapReduce limitations, we use:
Apache Spark
🔹 Why Spark is Faster
Instead of writing to disk between steps:
Keeps data in memory
Uses RDDs (Resilient Distributed Datasets)
Supports iterative algorithms
That’s why Spark is 10–100x faster for data science.
🔹 Spark Ecosystem
Component Purpose
Spark Core Processing engine
Spark SQL SQL interface
MLlib Machine learning
GraphX Graph processing
Component Purpose
Spark Streaming Real-time data
5️⃣ Case Study: Lending Risk Analysis
We work with loan data from:
LendingClub
Goal:
Create a dashboard to analyze:
Risk grades
Interest rates
Loan amounts
Recoveries
6️⃣ Step-by-Step Architecture
Here is the full pipeline:
Download data
↓
Store in HDFS (Hadoop)
↓
Clean using Spark
↓
Store in Hive
↓
Visualize in Qlik
7️⃣ Step 1: Store Data in Hadoop (HDFS)
Create directory:
hadoop fs -mkdir /chapter5
Upload file:
hadoop fs -put [Link] /chapter5
Now data is distributed across cluster.
8️⃣ Step 2: Clean Data Using Spark
Load file
data = [Link]("/chapter5/[Link]")
Split CSV
parts = [Link](lambda r: [Link](','))
Remove header
firstline = [Link]()
datalines = [Link](lambda x: x != firstline)
Example Cleaning Function
Suppose column 7 is interest rate like:
"10.4%"
We convert it to:
0.104
def cleans(row):
row[7] = str(float(row[7][:-1]) / 100)
return [[Link]() for s in row]
Apply to all rows:
datalines = [Link](lambda x: cleans(x))
Spark distributes this across all machines automatically.
9️⃣ Step 3: Store Cleaned Data in Hive
Hive allows SQL-style querying on Hadoop.
We create schema:
from [Link] import *
fields = [StructField(field_name, StringType(), True) for field_name in firstline]
schema = StructType(fields)
schemaLoans = [Link](datalines, schema)
[Link]("loans")
Create Summary Table
[Link]("""
CREATE TABLE LoansByTitle
STORED AS PARQUET AS
SELECT title, count(1) as number
FROM loans
GROUP BY title
ORDER BY number DESC
""")
Now data is stored in Hive.
🔟 Step 4: Visualize Using Qlik
We use:
Qlik Sense
Connect via ODBC to Hive.
Load table raw.
Dashboard Contains
KPIs:
Average interest rate
Total loan amount
Average loan amount
Total recoveries
Bar Charts:
Avg interest per grade
Total loan per grade
Recoveries per grade
Pivot Table:
Interest rate by:
Job title
Risk grade
📊 Example Insight
When filtering:
Directors → 11.97% average interest
Artists → 13.32% average interest
Why?
Artists may be considered higher risk → higher interest rate.
🔥 Why This Architecture Matters
Tool Role
Hadoop Distributed storage
Spark Fast distributed cleaning & transformation
Hive SQL interface
Qlik Interactive visualization
This separation allows:
Scalability
Fault tolerance
Performance
Easy reporting