Scribd
Upload
3 views2 pages

2) RDD Operations

The document provides an overview of RDD (Resilient Distributed Dataset) operations in PySpark, including various transformations such as map, filter, distinct, union, intersection, subtract, and cartesian. It also covers actions like collect, count, first, take, reduce, sum, max, min, and countByValue, demonstrating their usage with sample RDDs. The examples illustrate how to manipulate and analyze data using these RDD operations.

Uploaded by

sivatejam2005
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views2 pages

2) RDD Operations

The document provides an overview of RDD (Resilient Distributed Dataset) operations in PySpark, including various transformations such as map, filter, distinct, union, intersection, subtract, and cartesian. It also covers actions like collect, count, first, take, reduce, sum, max, min, and countByValue, demonstrating their usage with sample RDDs. The examples illustrate how to manipulate and analyze data using these RDD operations.

Uploaded by

sivatejam2005
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

8/15/25, 10:30 AM 2) RDD Operations.

ipynb - Colab

keyboard_arrow_down RDD Transformations


from [Link] import SparkSession
# Initialize Spark
spark = [Link]("RDD Operations").getOrCreate()
sc = [Link]
# Sample RDD
rdd = [Link]([1, 2, 3, 4, 5, 6, 2, 3])

# map() - Multiply each element by 2


mapped_rdd = [Link](lambda x: x * 2)
print(type(mapped_rdd))
print(mapped_rdd)
print(mapped_rdd.collect())
print(type(mapped_rdd.collect()))

<class '[Link]'>
PythonRDD[1] at RDD at [Link]
[2, 4, 6, 8, 10, 12, 4, 6]
<class 'list'>

# filter() - Keep only even numbers


filtered_rdd = [Link](lambda x: x % 2 == 0)
print(filtered_rdd.collect())

[2, 4, 6, 2]

# distinct() - Remove duplicates


distinct_rdd = [Link]()
print(distinct_rdd.collect())

[2, 4, 6, 1, 3, 5]

# Create Two RDDs


rdd1 = [Link]([1, 2, 3, 4, 5])
rdd2 = [Link]([4, 5, 6, 7, 8])
# union() - Combines both RDDs (Duplicates are not removed)
union_rdd = [Link](rdd2)
print("Union:", union_rdd.collect())

Union: [1, 2, 3, 4, 5, 4, 5, 6, 7, 8]

# intersection() - Returns only common elements (removes duplicates)


intersection_rdd = [Link](rdd2)
print("Intersection:", intersection_rdd.collect())

Intersection: [4, 5]

# subtract() - Returns elements in rdd1 that are NOT in rdd2


subtract_rdd1 = [Link](rdd2)
subtract_rdd2 = [Link](rdd1)
print("Subtract (rdd1 - rdd2):", subtract_rdd1.collect())
print("Subtract (rdd2 - rdd1):", subtract_rdd2.collect())

Subtract (rdd1 - rdd2): [1, 2, 3]


Subtract (rdd2 - rdd1): [8, 6, 7]

# cartesian() - Returns Cartesian Product (each element of rdd1 pairs with each element of rdd2)
cartesian_rdd = [Link](rdd2)
print("Cartesian Product:", cartesian_rdd.collect())

Cartesian Product: [(1, 4), (1, 5), (2, 4), (2, 5), (1, 6), (1, 7), (2, 6), (2, 7), (1, 8), (2, 8), (3, 4), (3, 5), (4, 4), (4, 5),

keyboard_arrow_down RDD Actions


# collect() - Returns all elements as a list
print("Collect:", [Link]())

Collect: [1, 2, 3, 4, 5, 6, 2, 3]

# count() - Returns the number of elements


print("Count:", [Link]())

[Link] 1/2
8/15/25, 10:30 AM 2) RDD [Link] - Colab

Count: 8

# first() - Returns the first element


print("First:", [Link]())

First: 1

# take(n) - Returns the first n elements


print("Take(3):", [Link](3))

Take(3): [1, 2, 3]

# reduce() - Aggregates elements using a function (sum in this case)


print("Reduce (sum):", [Link](lambda a, b: a * b))

Reduce (sum): 4320

# sum() - Get the sum of all elements


total_sum = [Link]()
print("Sum:", total_sum)

Sum: 26

# max() - Get the maximum value


max_value = [Link]()
print("Max:", max_value)

Max: 6

# min() - Get the minimum value


min_value = [Link]()
print("Min:", min_value)

Min: 1

# countByValue() - Count occurrences of each element


count_values = [Link]()
print("CountByValue:", count_values)

CountByValue: defaultdict(<class 'int'>, {1: 1, 2: 2, 3: 2, 4: 1, 5: 1, 6: 1})

[Link] 2/2

You might also like