Unit 4 – Big Data Frameworks, Project Lifecycle, Ethics & Research

Trends
This unit covers Apache Spark, Hadoop ecosystem extensions, Big Data lifecycle, data
governance, cybersecurity, privacy, compliance, and research directions.
Part 1 — Spark Framework Overview
1. Apache Spark Introduction
Apache Spark is an open-source, cluster-computing engine designed for fast, in-memory
data processing.
Why Spark?
 Extremely fast (100× faster than MapReduce)
 Supports in-memory computation
 Supports multiple workloads: batch, streaming, SQL, ML, graph
 Works with HDFS, Cassandra, HBase, AWS S3, local FS
2. Spark Components
Component Purpose
Spark Core Basic execution engine (RDD operations)
Spark SQL SQL queries, DataFrames
Spark Streaming Real-time data processing
MLlib Machine learning
GraphX Graph processing
SparkR / PySpark Language APIs

3. Features of Spark
 In-Memory Processing — faster execution
 Lazy Evaluation — creates DAG before execution
 Fault Tolerance (RDD Lineage) — rebuild lost partitions
 Polyglot API — Scala, Python, Java, R
 Distributed Execution — runs on YARN/Mesos/Kubernetes
4. Spark Architecture
 Driver Program
 Cluster Manager
 Executors
 Workers
 Tasks
4.1 Driver Program
 Runs main()
 Creates SparkContext or SparkSession
 Converts user code into DAG
 Schedules jobs on executors
 4.2 Cluster Manager
 Standalone
 Hadoop YARN
 Apache Mesos
 Kubernetes
4.3 Executors
 Run tasks assigned by driver
 Store RDD partitions
 Return results to driver
4.4 Workers
Nodes in cluster hosting executors.
5. Spark Execution Flow
 User writes code using RDD/DataFrame
 Driver parses & builds DAG
 DAG Scheduler → splits into stages
 Task Scheduler → creates tasks
 Executors run tasks → results to driver
Stages = wide transformations
Tasks = operations on partitions
6. Spark RDD (Resilient Distributed Dataset)
 Immutable
 Partitioned
 Fault tolerant
 Lazy evaluated
6.1 Types of RDD Operations
Transformations (Lazy): map(), flatMap(), filter(), reduceByKey(), join()
Actions: collect(), count(), first(), reduce(), saveAsTextFile()
7. DataFrames & Spark SQL
DataFrame = distributed rows with named columns, optimized by Catalyst.
 Optimized by Catalyst
 Easy SQL queries
 Faster than RDD
Example:
df = [Link]('[Link]', header=True)
[Link]('age','salary').show()
[Link]('emp')
[Link]("SELECT * FROM emp WHERE salary > 50000").show()
 8. Catalyst Optimizer
 Logical plan generation
 Logical optimization
 Physical plan selection
 Code generation
9. Tungsten Execution Engine
 Memory management
 CPU optimization
 Binary processing
 Cache-aware execution
Part 2 — Spark Streaming
10. Spark Streaming Overview
Processes real-time streams using micro-batches.
 Kafka
 Flume
 HDFS
 Socket streams
 AWS Kinesis
10.1 DStreams
DStream = sequence of RDDs over time.
Example:
stream = [Link]("localhost", 9999)
10.2 Structured Streaming
 Continuous processing
 Exactly-once semantics
 SQL support
Part 3 — Big Data Project Lifecycle
11. Big Data Analytics Lifecycle (6 Phases)
 Business Understanding — define problem & KPIs
 Data Collection — logs, DBs, APIs, sensors
 Data Preparation — cleaning, transformation
 Data Storage — HDFS, HBase, NoSQL, Cloud
 Data Analysis — MLlib, statistics, streaming
 Deployment — dashboards, APIs, automation
12. Data Governance
 Accurate
 Secure
 Consistent
 Accessible
 Compliant
 Data quality
 Data lineage
 Metadata management
 Lifecycle management
13. Cybersecurity in Big Data
 Threats: DDoS, leakage, unauthorized access, insider threats
 Techniques: Encryption, RBAC, Tokenization, Secure pipelines, Firewalls
Part 4 — Ethics & Regulations
14. Big Data Ethics
 Privacy
 Bias
 Fairness
 Transparency
 Accountability
15. Privacy & Protection
 Data minimization
 Purpose limitation
 Anonymization
 User consent
 Secure storage
16. Compliance Standards
Regulation Region Purpose
GDPR Europe Data privacy, user rights
HIPAA USA Medical data protection
CCPA California Consumer data protection
PCI-DSS Global Payment card security

Part 5 — Research Directions in Big Data