Scribd
Upload
15 views2 pages

OOM Code

The document outlines a PySpark script that demonstrates an Out of Memory (OOM) scenario by creating a large DataFrame and performing a groupBy operation with limited memory settings. It includes configurations for Spark session memory and an example of how data skew can affect memory usage. The script also provides instructions for starting a Spark history server and submitting the job.

Uploaded by

umaprashanth
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
15 views2 pages

OOM Code

The document outlines a PySpark script that demonstrates an Out of Memory (OOM) scenario by creating a large DataFrame and performing a groupBy operation with limited memory settings. It includes configurations for Spark session memory and an example of how data skew can affect memory usage. The script also provides instructions for starting a Spark history server and submitting the job.

Uploaded by

umaprashanth
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

from pyspark.

sql import SparkSession


from [Link] import rand, floor

# Spark session with limited memory


spark = [Link] \
.appName("OOM Example") \
.config("[Link]", "512m") \
.config("[Link]", "512m") \
.config("[Link]", "0.1") \
.config("[Link]", "1") \
.config("[Link]", "false") \
.getOrCreate()

df = [Link](0, 10**7).withColumn("col1", floor(rand() * 100)) \


.withColumn("col2", rand())

# Perform an expensive transformation to stress memory


# Example: GroupBy and Aggregate
result = [Link]("col1").sum("col2")
[Link]()

# Stop the Spark session


[Link]()

$SPARK_HOME/sbin/[Link]
spark-submit [Link]

# from [Link] import SparkSession


# from [Link] import rand, floor, col, when

# # Create Spark session with limited memory


# spark = [Link] \
# .appName("OOM Example - Single Task Fail") \
# .config("[Link]", "512m") \
# .config("[Link]", "512m") \
# .config("[Link]", "0.1") \
# .config("[Link]", "1") \
# .config("[Link]", "false") \
# .getOrCreate()

# df = [Link](0, 10**7).withColumn("col1", when(col("id") % 10 == 0,


0).otherwise(floor(rand() * 100))) \
# .withColumn("col2", rand())

# # Explanation of skew:
# # Rows where `id % 10 == 0` will all go to the key `0`, creating a heavily skewed
partition.
# # Other rows are distributed across random keys.

# # Perform a transformation to stress memory


# result = [Link]("col1").sum("col2")

# # Trigger execution
# try:
# [Link]()
# except Exception as e:
# print("Job failed with error:", e)

# # Stop the Spark session


# [Link]()

You might also like