Essentials
PySpark
functions with
examples
- Ujjwal Sontakke Jain
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
PySpark Essential Functions for Data Engineers
Essential PySpark Functions with Examples ---
PySpark offers a powerful API for big data processing. Here are the
most commonly used functions every data engineer should know —
explained with examples!
--------------------------------------------------------------------------------------------
1. select()
Description – Select specific columns
Example –
[Link]("name", "age").show()
2. withColumn()
Description – Add or replace a column
Example -
from [Link] import col
[Link]("age_plus_1", col("age") + 1).show()
3. filter() / where()
Description – Filter rows
Example –
[Link](col("age") > 25).show()
[Link](col("city") == "Mumbai").show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
4. drop()
Description – Drop columns
Example –
[Link]("unnecessary_column").show()
5. distinct()
Description – Remove duplicate rows
Example -
[Link]("country").distinct().show()
6. dropDuplicates()columns
Description – Drop duplicate rows based on specific
Example –
[Link](["name", "city"]).show()
7. groupBy() + agg()
Description – Aggregate functions
Example –
from [Link] import avg
[Link]("city").agg(avg("salary")).show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
8. orderBy() / sort()
Description – Sort data
Example –
[Link]("age", ascending=False).show()
9. join()
Description – Join DataFrames
Example –
[Link](df2, on="id", how="inner").show()
10. union() / unionByName()
Description – Combine DataFrames
Example –
[Link](df2).show()
11. limit()
Description – Limit number of rows
Example –
[Link](5).show()
12. isNull() / isNotNull()
Description – Handle missing values
Example –
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
[Link](col("email").isNull()).show()
13. fillna() / dropna()
Description – Fill or drop nulls
Example –
[Link]({"city": "Unknown"}).show()
[Link]().show()
14. when() + otherwise()
Description – Conditional column
Example –
from [Link] import when
[Link]("status", when(col("age") > 18,
"Adult").otherwise("Minor")).show()
15. explode()
Description – Flatten arrays
Example –
from [Link] import explode
[Link]("item", explode(col("items"))).show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
16. array_contains()
Description – Check if array column contains a value
Example –
from [Link] import array_contains
[Link](array_contains(col("skills"), "Python")).show()
17. collect_list() / collect_set()
Description – Aggregate values into lists
Example –
from [Link] import collect_list
[Link]("department").agg(collect_list("employee")).show()
18. date_format() / to_date() / current_date()
Description – Date functions
Example –
from [Link] import to_date, current_date
[Link]("dob", to_date("dob", "yyyy-MM-dd")).show()
19. substr() / concat()
Description – String functions
Example –
from [Link] import substr, concat
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
[Link]("initials", concat(substr("first_name", 1, 1),
substr("last_name", 1, 1))).show()
20. cast()
Description – Type conversion
Example –
[Link]("age", col("age").cast("int")).show()
21. repartition() / coalesce()
Description – Control number of partitions
Example –
[Link](4)
[Link](1)
22. cache() / persist()
Description – Persist DataFrames
Example –
[Link]().count()
23. describe() / summary()
Description – Quick stats
Example –
[Link]().show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
[Link]().show()
24. alias()
Description – Rename column temporarily
Example –
[Link](col("age").alias("AgeInYears")).show()
25. createOrReplaceTempView() + [Link]()
Description – SQL queries
Example –
[Link]("people")
[Link]("SELECT * FROM people WHERE age > 30").show()
26. window()
Description – Perform operations over a window
Example –
from [Link] import Window
from [Link] import row_number
windowSpec = [Link]("department").orderBy("salary")
[Link]("rank", row_number().over(windowSpec)).show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
27. lag() / lead()
Description – Access previous/next row in a window
Example –
from [Link] import lag, lead
[Link]("prev_salary", lag("salary",
1).over(windowSpec)).show()
28. ntile()
Description – Distribute rows into buckets
Example –
from [Link] import ntile
[Link]("quartile", ntile(4).over(windowSpec)).show()
29. rank() / dense_rank()
Description – Ranking functions
Example –
from [Link] import rank, dense_rank
[Link]("rank", rank().over(windowSpec)).show()
30. broadcast()
Description – Broadcast small DataFrame to optimize joins
Example –
from [Link] import broadcast
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
[Link](broadcast(df2), "id").show()
31. monotonically_increasing_id()
Description – Add unique ID
Example –
from [Link] import monotonically_increasing_id
[Link]("id", monotonically_increasing_id()).show()
32. input_file_name()
Description – Track source file of each row
Example –
from [Link] import input_file_name
[Link]("source_file", input_file_name()).show()
33. regexp_extract() / regexp_replace()
Description – Regex operations
Example –
from [Link] import regexp_extract, regexp_replace
[Link]("domain", regexp_extract("email", "@(.*)", 1)).show()
34. sha2() / md5()
Description – Hash functions
Example –
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
from [Link] import sha2
[Link]("hashed_id", sha2(col("id"), 256)).show()
35. row_number() / dense_rank() / rank()
Description – Windowed row functions
Example –
from [Link] import row_number
[Link]("row_num",
row_number().over(windowSpec)).show()
36. explode_outer()
Description – Explode arrays while keeping nulls
Example –
from [Link] import explode_outer
[Link]("item", explode_outer(col("items"))).show()
37. posexplode()
Description – Explode with position index
Example –
from [Link] import posexplode
[Link](posexplode(col("items"))).show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
38. map_from_arrays() – Create map column
Description –
Example –
from [Link] import map_from_arrays, array, lit
[Link]("info_map", map_from_arrays(array(lit("name")),
array(col("name")))).show()
39. to_json() / from_json()
Description – Convert to/from JSON
Example –
from [Link] import to_json, from_json,
schema_of_json
[Link]("json_col", to_json(col("struct_col"))).show()
40. size()
Description – Get length of array
Example –
from [Link] import size
[Link]("num_skills", size(col("skills"))).show()
41. array() / struct() / map()
Description – Complex types
Example –
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
from [Link] import array, struct, map
[Link]("full_name", struct("first_name",
"last_name")).show()
42. corr() / covar_pop() / covar_samp()
Description – Statistical analysis
Example –
from [Link] import corr, covar_pop
[Link](corr("age", "salary")).show()
43. bucketBy() + sortBy()
Description – Save optimized Parquet
Example –
[Link](4,
"department").sortBy("salary").saveAsTable("emp_bucketed")
44. foreach() / foreachPartition()
Description – Apply function on each row
Example –
[Link](lambda row: print([Link]))
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]
45. foreachBatch()
Description – For streaming
Example –
query = [Link](lambda df, epoch_id:
[Link]()).start()
46. pivot() / unpivot()
Description – Reshape data
Example –
[Link]("month").pivot("product").sum("revenue").show()
Created By - Ujjwal Sontakke Jain. Follow me on LinkedIn - [Link]