■ Module 1: Introduction & Setup (Databricks +
PySpark)
1. What is Apache Spark & PySpark?
• Apache Spark: A distributed computing framework for big data processing. It handles large-scale
data using parallelism across clusters. • PySpark: The Python API for Spark, allowing you to write
Spark jobs in Python. • Databricks: A cloud-based data platform built around Apache Spark,
optimized for collaborative data engineering, machine learning, and analytics.
2. Spark Architecture (Working Methodology)
• Driver Program: Runs your code, creates SparkContext, plans tasks. • Cluster Manager: Allocates
resources (YARN, Kubernetes, Databricks cluster manager). • Executors: Worker nodes executing
tasks in parallel. • Tasks: Smallest unit of execution. Workflow: 1. User writes code (PySpark/SQL).
2. Driver converts into a DAG (Directed Acyclic Graph). 3. Tasks distributed across executors. 4.
Results returned/ persisted.
3. Databricks Environment Overview
• Workspace: UI for notebooks, data, clusters, jobs. • Cluster: A group of VMs (compute resources)
where Spark jobs run. • DBFS (Databricks File System): Cloud object storage (backed by
S3/ADLS/GCS). • Notebook: Interactive coding environment supporting Python, SQL, Scala, R.
4. Getting Started with PySpark on Databricks
1. Create a free Databricks account (community edition). 2. Launch workspace. 3. Create a cluster
(choose small config). 4. Create a notebook in Python. 5. Run PySpark commands.
5. Example Code (Hello World in PySpark)
# Create DataFrame
data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]
df = [Link](data, ["Name", "Age"])
# Show data
[Link]()
# Simple transformation
[Link]([Link] > 30).show()
# SQL way
[Link]("people")
[Link]("SELECT Name FROM people WHERE Age > 30").show()
6. Exercises
1. Create a DataFrame of 5 employees with (Name, Department, Salary).
2. Display employees earning more than 50,000.
3. Register it as a table and run SQL to count employees per department.
4. Save the DataFrame to CSV in DBFS.
7. Interview Questions
1. What is the difference between Spark and Hadoop MapReduce?
2. Explain Spark architecture (Driver, Executors, Cluster Manager).
3. Why is PySpark popular for data engineering?
4. What is Databricks DBFS?
5. What is the difference between transformations and actions in Spark?