Scribd
Upload
12 views1 page

Understanding RDDs and Spark Transformations

Uploaded by

namratapednekar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views1 page

Understanding RDDs and Spark Transformations

Uploaded by

namratapednekar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Module 2: Module 2

Concepts
• RDDs (Resilient Distributed Datasets) – Immutable distributed collections.
• Lazy evaluation with DAG execution.
• Transformations: map, filter, flatMap, union, join.
• Actions: collect, count, reduce, take, saveAsTextFile.

Example
rdd = [Link]([1,2,3,4,5]) squared = [Link](lambda x: x**2)
print([Link]()) # [1,4,9,16,25]

Exercises
• Create RDD of numbers 1–20, filter even numbers, square them.
• Load text file, count word frequency.
• Compare lazy evaluation vs action.

Interview Questions
• Difference between RDD, DataFrame, Dataset.
• Why is Spark lazily evaluated?
• Difference between narrow vs wide transformations.

You might also like