Unit II : Big Data and Big Data Platform Blueprint

Big Data – Complete Detailed Notes

What is Big Data?
Big Data refers to extremely large and complex data sets that cannot be
processed, stored, or analyzed using traditional data processing tools like
relational databases due to their volume, velocity, and variety.
Key Definitions:
1. High-volume, high-velocity, high-variety information assets that require
innovative and cost-effective processing for enhanced insights and
decision-making.
2. Data sets whose size or type exceeds the ability of traditional systems to
capture, store, manage, and analyze with low latency.
3. Large, complex datasets, especially from new data sources, that are too
vast for traditional software to handle efficiently.
Big Data is typically produced by modern technologies such as IoT, social
media, mobile devices, AI systems, and sensors.

Sources of Big Data

Big data originates from varied, high-speed, and often unstructured sources,
including:
• Environmental sensors (weather data, climate monitoring)
• Social media platforms (posts, messages, multimedia)
• Mobile phones (GPS signals, usage logs)
• Digital pictures and videos
• Purchase/transaction records (POS systems)
• IoT devices (smart homes, cars, wearables)
• Web logs, clickstreams
• Network traffic
 • Machine-generated logs
• Audio/video streams
Modern technologies such as AI, Mobile, Social Media, and IoT significantly
increase data complexity and volume.

Types of Big Data

1. Structured Data
• Organized and stored in fixed formats.
• Can be easily stored in RDBMS.
• Examples: Tables, rows, columns.
2. Semi-Structured Data
• Not stored in relational tables but has some organizational properties.
• Examples: XML, JSON, HTML, RDF.
3. Unstructured Data
• No fixed format or organization.
• Complex and difficult to process.
• Examples: Text, audio, video, images, log files.

The 4 V’s of Big Data

Big Data is characterized by its Volume, Variety, Velocity, and Veracity.

1. Volume – Large Quantity of Data

Refers to the enormous amount of data generated every second from various
sources such as:
• Social media platforms
• Sensors
• Financial transactions
 • Stock markets
• Mobile devices
Example:
Facebook produces:
• Over 1 billion messages/day
• 4.5 billion “likes”/day
• 350+ million photo uploads/day
Such massive data requires Big Data technologies for processing and storage.

2. Variety – Different Types of Data

Data comes from multiple sources and in multiple formats:
• Structured: RDBMS
• Semi-structured: JSON, XML, HTML
• Unstructured: videos, texts, images, audio
Big Data must handle all forms of data, not just structured.

3. Velocity – Speed of Data Generation & Processing

Velocity refers to:
• How fast data is generated
• How fast it must be processed
Examples:
• Banking transactions
• IoT devices generating data in milliseconds
• Stock exchange updates
• Fraud detection
• Climate monitoring
For real-time decisions, data must be processed as it arrives.

4. Veracity – Trustworthiness & Quality of Data

Refers to:
• Accuracy
• Reliability
• Consistency
• Completeness
Big Data may contain:
• Noise
• Errors
• Missing values
• Inaccurate information
• Ambiguous sources
Veracity ensures that data used for analysis is trustworthy.

Working on Big Data: Planning is Everything & ETL Challenges

Big Data projects require careful planning because ETL (Extract, Transform,
Load) processes become complicated with large-scale, high-speed, diverse data.
Key Problems With ETL in Big Data

1. Scalability & Performance Issues

Traditional ETL tools struggle with:
• Huge data volumes
• High data velocity
• Complex data formats
Problems:
• Slow performance
• Processing bottlenecks
• ETL job failures
Solutions:
• Plan for scalability
• Use distributed processing systems (Spark, Hadoop)
• Use cloud-based ETL for elastic scaling
• Parallel processing pipelines

2. Data Quality & Consistency Issues

Big Data involves unstructured and noisy data.
Problems:
• Missing values
• Inconsistent formats
• Duplicates
• Errors
Solutions:
• Data profiling
• Data cleansing
• Data validation
• Well-defined transformation rules

3. Handling Complexity & Variety

Integrating data from many sources requires complex ETL logic.
Problems:
 • Complex transformations
• Hard-to-maintain pipelines
• Increased risk of errors
Solutions:
• Clear data model design
• Modular ETL architecture
• Staged transformation approach
• Use schema-on-read (e.g., Spark, Hive)

4. Real-Time Data Processing Challenges

Traditional ETL → batch-based → slow.
Problems:
• Cannot support real-time analytics
• Delayed insights
• Not suitable for fraud detection, IoT, monitoring
Solutions:
• Streaming ETL (Spark Streaming, Kafka Streams)
• ELT (Extract → Load → Transform)
• Event-based processing

5. Cost and Resource Management Challenges

Big Data infrastructure is expensive.
Problems:
• High compute/storage cost
• Inefficient ETL consumes resources
• Cloud usage can become expensive without planning
Solutions:
• Optimize pipelines
• Auto-scaling in cloud
• Use cost-efficient ETL tools
• Resource monitoring and tuning

Why Planning is Important in Big Data Projects

Proper planning helps:

✔ Reduce Risks
• Catch ETL failures early
• Prevent data inconsistencies

✔ Improve Efficiency
• Faster data loading
• Smooth transformations
• Quick insights

✔ Enable Scalability
• Handle growing data volumes
• Better long-term performance

✔ Enhance Data Quality

• Reliable data for analysis
• More accurate insights

✔ Reduce Costs
• Optimized compute/storage usage
• Lower operational expenses

Big Data Architectures

Big Data architecture involves both hardware and software considerations to
handle massive data efficiently.

Hardware Architecture
Traditional systems cannot handle Big Data because:
• Data is too large for one machine
• SMP (Symmetric Multi-Processing) has limits
SMP (Symmetric Multi-Processing) Issues
• Processors share same RAM & I/O bus
• Limited number of disks
• BUS bottlenecks
• Not scalable for big workloads

Scaling Methods
There are two main scaling techniques:

1. Scale Up (Vertical Scaling)

Increase the power of a single machine:
• More CPU
• More RAM
• Faster storage
Pros
• Lower power consumption
• Lower cooling cost
• Easier implementation
• Lower networking complexity
 • Fewer licenses required
Cons
• Very expensive
• Vendor lock-in
• Higher risk (single machine failure)
• Limited upgrade potential

2. Scale Out (Horizontal Scaling)

Scale Out refers to adding more machines/servers (nodes) rather than increasing
the power of a single machine.
Definition
Scale out = Increasing capacity by adding more servers with moderate
CPU/RAM, instead of upgrading one large machine.

✔ Pros of Scale Out

1. Much cheaper than scaling vertically
Adding several low-cost servers is more economical than buying a powerful
single server.
2. New technologies simplify operations
Distributed systems today have built-in:
• Fault-tolerance
• Auto-scaling
• Monitoring
• Load balancing
3. Easy to upgrade
Just add more nodes → capacity increases.
4. Cheaper overall
Commodity hardware reduces cost per unit.
5. Virtually unlimited scalability
Clusters can grow almost infinitely as load increases.

✘ Cons of Scale Out

1. More licensing fees
Software may charge per node or per core.
2. Larger physical footprint
More servers require more space in the data center.
3. Higher power consumption
More electricity + cooling required.
4. More networking equipment
Requires:
• More switches
• More cables
• More routers

Big Data Platform Blueprint

A Big Data platform handles ALL aspects of data lifecycle:
Ingest → Store → Analyse → Display
A modular blueprint ensures each layer is loosely coupled so components can be
replaced or upgraded easily.

1. Ingest Layer
Definition
Getting data from various sources into the system.
Sources
• Tweets, social media posts
• Server logs
• IoT sensors
• Applications
• Transactions
• GPS signals
Process
1. Source systems send data →
2. API services receive it →
3. API pushes to temporary storage or messaging system
Common Tools
• Apache Kafka
• RabbitMQ
• AWS Kinesis
• Redis (specialized caching)
Temporary storage allows downstream systems quick access.

2. Store Layer
Stores large amounts of raw, unprocessed, historical data.
Key Principles
• Follows publish-subscribe model
• Store EVERYTHING (never throw away data)
• Historical data helps future analytics
Tools
• Hadoop HDFS
 • HBase
• Amazon S3
• DynamoDB
This is the central data lake or big data storage layer.

3. Analyse Layer
The most important part of the platform.
Two Types of Analytics
a) Stream Processing (Real-time)
• Data comes directly from ingest
• Fast, real-time results
• Useful for fraud detection, live monitoring
b) Batch Processing
• Analytics done on stored historical data
• Handles very large data sets
• Slower but more accurate
• Used for long-term trends, predictions, ML training

4. Display Layer
Transforms analytics results into a form humans can understand.
Why it is important?
People make data-driven decisions, so data must be clear.
Common Tools
• Dashboards
• BI tools
• Custom user interfaces
If many teams use the platform → allow them to build their own UI.

Lambda Architecture
Lambda Architecture is widely used by big tech companies for processing both
real-time streaming data and historical batch data.
It consists of three layers:
1. Batch Layer (Cold Path)
2. Stream Layer (Hot Path / Speed Layer)
3. Serving Layer

1. Batch Layer (Cold Path)

Function
• Stores all historical data
• Computes accurate batch results
• High latency but high accuracy
Characteristics
• Fault-tolerant
• Handles huge datasets
• Uses distributed storage

2. Stream Layer (Hot Path / Speed Layer)

Function
• Processes data in real-time
• Complements the batch layer
• Performs incremental updates
Characteristics
 • Handles incoming data instantly
• Low latency, but slightly lower accuracy
• Supports time-sensitive applications

3. Serving Layer
Function
• Serves results from both batch & speed layers
• Responds to user queries
Examples of queries
• Dashboards
• Notifications
• Analytics queries

✔ Advantages of Lambda Architecture

1. Balance of speed, reliability & scalability
Batch → reliable
Speed layer → fast
2. Fault-tolerant historical data
Stored in distributed systems (HDFS, S3)
3. Real-time + Offline analytics
Covers almost all business needs.
4. Scalable
Handles large amounts of data easily.

✘ Disadvantages of Lambda Architecture

1. Complex infrastructure
Multiple layers → difficult to manage.
2. Duplicate logic
Batch & speed layers require separate codebases.
3. Maintenance challenges
Two separate pipelines must stay in sync.
4. Performance overhead
Batch jobs run repeatedly → resource heavy.
5. Migration difficulty
Hard to reorganize or model new datasets.

Batch Processing
Definition
Batch processing collects data over time and processes it in fixed chunks or
scheduled intervals.
Examples
• Billing systems
• Report generation
• ETL jobs
• Backups
• Analytics on historical data
Advantages
• High throughput
• Efficient for large datasets
• Optimizes resource usage
• Good for non-urgent insights
Disadvantages
 • High latency
• Not suitable for real-time applications

Stream Processing
Definition
Stream processing handles data in real-time as it arrives.
Examples
• Fraud detection
• Stock market analysis
• IoT device monitoring
• Live dashboards
• Social media analytics
Advantages
• Low latency
• Immediate insights
• Suitable for real-time applications
• Handles high-velocity data
Disadvantages
• Complex infrastructure
• Needs fault tolerance
• Hard to process data that arrives out of order

Kappa Architecture
Kappa Architecture was introduced by Jay Kreps (co-founder of Apache Kafka)
in 2014.
It is a simplified alternative to Lambda Architecture.
Key Idea:
Only ONE processing layer → Stream Layer.
No batch layer.

How Kappa Works

1. Data Ingestion
• Real-time data enters through messaging systems
• Example: Apache Kafka
2. Stream Processing
• A single pipeline handles ALL data
• Uses tools like:
o Kafka Streams
o Apache Flink
o Apache Samza
Processed data is stored in:
• Cassandra
• HBase
3. Serving Layer
• Query engine (Presto, Hive) provides results to users.

✔ Advantages of Kappa Architecture

• Simpler than Lambda
• No duplicate code
• Easy to maintain
• Supports real-time workloads
• Reprocessing = replaying data through the same pipeline
✘ Disadvantages of Kappa Architecture
• Not ideal for heavy batch processing needs
• Requires full data replay for reprocessing
• May be costlier for storing infinite event logs