Scribd
Upload
12 views1 page

ETL, UDFs, and Join Strategies in Spark

Uploaded by

namratapednekar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views1 page

ETL, UDFs, and Join Strategies in Spark

Uploaded by

namratapednekar
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Module 5: Module 5

Concepts
• ETL – Extract, Transform, Load.
• Data cleansing (null handling, deduplication).
• Built-in functions vs UDFs vs Pandas UDFs.
• Join strategies (broadcast join, shuffle join).

Example
from [Link] import col, udf from [Link] import StringType upper_udf =
udf(lambda x: [Link](), StringType()) [Link]("NameCaps",
upper_udf(col("Name"))).show()

Exercises
• Load dataset, remove duplicates, fill nulls.
• Write UDF for string manipulation.
• Apply broadcast join for small lookup table.

Interview Questions
• What are UDFs and their performance drawbacks?
• Difference between ETL and ELT.
• How Spark handles skewed data?

You might also like