SparkBasicSyntax.
py – Full Explanation
This document explains each part of the PySpark RDD example script step by step in simple
English.
Step 1 – Imports and Setup
from pyspark import SparkContext, SparkConf
import sys
SparkConf sets up configuration for your Spark application (like name and resources).
SparkContext creates the connection between your Python program and the Spark engine.
sys is used to handle command-line arguments.
Step 2 – Command Line Arguments
The script expects two arguments when you run it with spark-submit:
1. Input file path
2. Output folder path
It checks like this:
if len([Link]) != 3:
print("Usage: spark-submit rdd_operations.py ")
Step 3 – Start Spark
conf = SparkConf().setAppName("PySpark RDD Operations").setMaster("local[*]")
sc = SparkContext(conf=conf)
- setAppName: name of your Spark job.
- setMaster("local[*]"): runs Spark locally on all CPU cores.
- SparkContext: starts the Spark engine and connects your script to it.
Step 4 – Create RDDs
1. **parallelize()** – turns a Python list into a distributed RDD.
2. **textFile()** – reads a text file, each line becomes an RDD element.
Step 5 – Basic Transformations
Transformations define “what to do” with data but do not run immediately.
- **map()**: applies a function to each element.
- **flatMap()**: splits or expands each element into multiple parts.
- **filter()**: keeps only data matching a condition.
- **union()**, **intersection()**, **subtract()**: set operations.
- **sortBy()**: sorts elements by a custom rule.
Step 6 – Basic Actions
Actions actually trigger Spark to run the computation.
- **collect()**: brings all results to the driver.
- **take(n)**: gets first n elements.
- **top(n)**: gets n largest elements.
- **count()**: counts total records.
- **reduce()**: combines data (like sum).
- **fold()**: like reduce but with a start value.
- **aggregate()**: used for custom sum/count operations.
Step 7 – Key-Value Transformations
These work on (key, value) pairs.
- **reduceByKey()**: adds values with same key.
- **groupByKey()**: groups all values for each key.
- **combineByKey()**: advanced combine (e.g., sum + count).
- **sortByKey()**: sorts by key.
- **subtractByKey()**: removes pairs with same keys from another RDD.
- **mapValues()**, **flatMapValues()**: apply function only to values.
- **keys()**, **values()**: extract only one side.
Step 8 – Key-Value Actions
- **collect()**: returns all pairs.
- **lookup(key)**: returns all values for a specific key.
Step 9 – Save Results
[Link](final_output).saveAsTextFile(output_path)
Writes all results to an output folder as a text file.
Step 10 – Stop Spark
[Link]()
Stops the Spark engine and frees memory.
Summary Table
| Step | Purpose | Example |
|------|----------|----------|
| 1–2 | Setup & Input paths | Import modules, read file args |
| 3 | Start Spark | sc = SparkContext(conf) |
| 4 | Create Data | parallelize(), textFile() |
| 5 | Transformations | map, filter, sortBy |
| 6 | Actions | collect, count, reduce |
| 7 | Key-Value Ops | reduceByKey, groupByKey |
| 8 | Key-Value Actions | lookup |
| 9 | Save | saveAsTextFile |
| 10 | Stop | [Link]() |