Apache Spark — Detailed Study Guide (based on Great
Learning Full Course)
Duration: ~7h 43m • Instructor: Raghu Raman • Publisher: Great Learning
1) Introduction (00:00)
Why Spark? Traditional MapReduce writes to disk between stages, which is slow for iterative and interactive
workloads. Spark keeps data in-memory across steps, drastically reducing I/O and enabling near real■time
analytics.
• Use cases: ETL/ELT pipelines, interactive SQL, ML feature engineering, streaming analytics.
• Key goals: speed (in■memory), simplicity (unified APIs), scalability (cluster computing), fault■tolerance
(lineage).
2) Spark Fundamentals (00:01:23)
Spark is a distributed compute engine. A typical application has a Driver (your program) and many Executors
(workers). The Cluster Manager (Standalone, YARN, Kubernetes, Mesos) allocates resources.
• Driver: builds the logical plan/DAG, schedules tasks, hosts SparkContext/SparkSession.
• Executors: run tasks, store cached data, report back results/metrics.
• Lazy evaluation: transformations build a DAG; actions trigger execution.
• Deployment modes: client vs. cluster; local[*] for dev.
• Resilience: lineage recomputes lost partitions; optional checkpointing for long lineage trees.
# PySpark bootstrap (local dev)
from [Link] import SparkSession
spark = [Link] .appName("demo") .master("local[*]") .config("[Link]
print([Link])
3) Spark Ecosystem (00:24:00)
Spark ships multiple libraries so you can use one engine for many workloads.
• Core: RDD execution engine.
• Spark SQL: DataFrames/Datasets + ANSI SQL.
• Structured Streaming: streaming with the same DataFrame API.
• MLlib: scalable ML algorithms/feature tools.
• GraphX: graph processing (Scala-heavy).
4) Spark vs. Hadoop MapReduce (00:51:22)
Spark ≠ storage. Spark often uses HDFS/S3 for data, and can run on YARN/K8s.
Compared to MapReduce, Spark provides higher-level APIs, in-memory processing and optimizations
(Catalyst/Tungsten).
• MR strengths: batch throughput, simplicity; Spark strengths: speed, expressiveness, unification.
• Spark integrates with Hadoop ecosystem: HDFS, Hive metastore, YARN, HBase, etc.
5) RDD Fundamentals (01:08:56)
An RDD is an immutable, partitioned collection. RDDs carry a lineage graph describing how they were derived.
• Narrow vs. wide transformations: narrow (map/filter) suggest pipelineable tasks; wide (groupByKey/join) cause
shuffles.
• Persistence levels: MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, with (de)serialization options.
• Partitioning: hash or range; custom partitioners help co-partition joins.
• When to still use RDDs: fine-grained control, custom partitioners, low-level transformations.
# RDD example
rdd = [Link](range(10), numSlices=4)
evens = [Link](lambda x: x % 2 == 0)
print([Link]())
6) Transformations, Actions & Operations (01:29:22)
Transformations are lazy and return a new RDD/DataFrame; actions trigger the DAG to execute.
• Common transformations: map, filter, flatMap, distinct, sample, union, join, reduceByKey, mapValues.
• Common actions: count, collect (careful!), take, first, save, foreach.
• Performance: avoid groupByKey in favor of reduceByKey/aggregateByKey; minimize shuffles; cache wisely.
from operator import add
rdd = [Link]([("a",1),("b",2),("a",3)])
# Good: reduceByKey aggregates per partition before shuffle
agg = [Link](add)
print([Link]())
7) Jobs, Stages & Tasks (02:36:54)
Spark builds a DAG from your code. Each action creates a Job, which splits into Stages at shuffle boundaries;
Stages contain Tasks that operate on partitions.
• ShuffleMapStage vs ResultStage: shuffles create stage boundaries.
• Task locality matters (PROCESS_LOCAL > NODE_LOCAL > RACK_LOCAL > ANY).
• Speculative execution can mitigate slow tasks (stragglers).
• Use the Spark UI to inspect time spent in shuffles, skewed tasks, GC, and I/O.
8) RDD Creation (03:10:17)
Create RDDs from in-memory collections, files, or existing RDDs. Choose partitions carefully for parallelism.
• Parallelize small local data for demos; use file-based input for production.
• Use coalesce() to reduce partitions (no shuffle), repartition() to increase/redistribute (shuffle).
# File-based RDD
text = [Link]("s3a://bucket/path/*.csv")
words = [Link](lambda line: [Link](","))
print([Link](5))
9) Spark SQL (03:49:15)
DataFrames provide a declarative API and benefit from Catalyst (logical/physical plan optimization) and Tungsten
(memory/codegen).
• Prefer built-in functions over UDFs for optimizer awareness & vectorization.
• Beware of null semantics; enable ANSI mode if desired.
• Explain plans (explain(True)) to see parsed/logical/physical plans.
from [Link] import functions as F
df = [Link]("s3a://bucket/events/")
df = [Link]([Link]("event_type") == "purchase") .groupBy("country").agg([Link]("user_i
[Link]([Link]("buyers")).show(10)
10) DataFrame Basics (04:12:38)
Core operations for analytics.
• Schema: infer vs. provide (better). Use .schema to inspect, .printSchema() to view tree.
• Joins: inner/left/right/full; skew mitigation with salting or broadcast joins.
• Aggregations: groupBy + agg; Window functions for sessionization/rankings.
• Caching: cache/persist for reuse; unpersist when done.
from [Link] import Window, functions as F
w = [Link]("country").orderBy([Link]("revenue"))
df2 = [Link]("rank", F.dense_rank().over(w))
[Link]("rank <= 3").show()
11) Reading Files — CSV/JSON/Parquet/Avro (05:05:30)
Spark supports many formats; columnar (Parquet/ORC) are preferred for analytics.
• CSV/JSON: specify schema, delimiter, header, mode (PERMISSIVE/DROPMALFORMED/FAILFAST).
• Parquet/ORC: column pruning, predicate pushdown, compression (snappy/zstd).
• Partition discovery using directory layout (e.g., country=US/year=2025).
df_csv = [Link]("header", True).option("inferSchema", True).csv("/data/sales/*.csv")
df_parq = [Link]("/data/sales_parquet/")
df_csv.[Link]("overwrite").partitionBy("country","year").parquet("/warehouse/sales/")
12) Spark SQL + Hive (05:46:01)
Spark can use the Hive Metastore for catalogs, external/managed tables, and ACID (with limitations by version).
• External vs Managed: external leaves data where it lives; managed lets Spark manage lifecycle.
• CTAS & partitioned tables; table properties; bucketing (deprecated for many cases in modern Spark).
• Use catalog APIs: [Link](), .listDatabases().
[Link]("CREATE DATABASE IF NOT EXISTS dwh")
[Link]("""
CREATE TABLE IF NOT EXISTS [Link]
(country STRING, year INT, revenue DOUBLE)
USING PARQUET
PARTITIONED BY (country, year)
LOCATION 's3a://warehouse/dwh/sales'
""")
13) Sqoop on Spark (06:04:58)
Sqoop was historically used for bulk transfer between RDBMS and Hadoop. In many Spark shops today, the JDBC
data source with partitioning (column ranges) replaces it for simpler ops and fewer moving parts.
• When to still use Sqoop: legacy clusters, operational teams with established workflows.
• Prefer Spark JDBC for pipelines that already run in Spark.
jdbc_url = "jdbc:postgresql://db/acme"
props = {"user":"svc","password":"***","driver":"[Link]"}
df = [Link](jdbc_url, table="[Link]", properties=props)
[Link]("overwrite").parquet("s3a://lake/raw/orders/")
14) Streaming via Flume → Spark (07:08:07)
The video shows Flume → Spark streaming. In modern stacks, prefer Structured Streaming (direct Kafka
source).
• Micro-batch model with exactly-once sinks via checkpoints + idempotent writes.
• Watermarks for late data; event-time windows; stateful aggregations.
# Structured Streaming (Kafka example)
from [Link] import functions as F
events = ([Link]
.format("kafka")
.option("[Link]","broker:9092")
.option("subscribe","events")
.load())
parsed = [Link]([Link]("value").cast("string").alias("json"))
# ... parse, transform ...
query = ([Link]
.outputMode("append")
.option("checkpointLocation","/chk/events")
.format("parquet")
.start("/streams/events_out"))
# [Link]() # in real apps
15) Performance & Best Practices (Bonus)
Practical tips to keep Spark jobs fast and reliable.
• Prefer DataFrame/SQL APIs over RDDs; avoid Python UDFs when possible (use built-ins/Pandas UDFs).
• Right-size partitions: aim for 128–512MB per partition for large scans; avoid tiny files.
• Use broadcast joins for small dimension tables; set [Link].
• Reduce shuffle volume: use reduceByKey/agg, pre-partition, prune columns early.
• Skew mitigation: salting keys, adaptive query execution (AQE), skew join hints.
• Cache only reused datasets; persist selectively; unpersist when done.
• Monitoring: Spark UI, event logs, metrics to Prometheus/Graphite; watch GC and spill metrics.
# Example: broadcast a small dimension
from [Link] import functions as F
dim = [Link]("/ref/countries") # small
fact = [Link]("/facts/events")
joined = [Link]([Link](dim), "country_id")
16) Handy Commands & Config (from scattered timestamps)
Quick references you can reuse.
• Coalesce vs Repartition: coalesce(n) reduces partitions without shuffle; repartition(n) reshuffles for balance.
• Explain plans: [Link](True); Adaptive Query Execution can change the plan at runtime.
• View config: [Link]().getAll()
print([Link]().getAll())
[Link](True)
[Link](400) # redistribute
small = [Link](50) # collapse without shuffle