01/01/2025, 01:27 Missing Columns Debugging

Below is a comprehensive list of commonly used PySpark functions categorized by their purpose,
along with examples to help you prepare for your interview:

1. Importing PySpark Functions

python Copy code

from [Link] import SparkSession from [Link] import *

2. Commonly Used Functions

Column Creation and Manipulation

1. withColumn : Add or update a column.

python Copy code

df = [Link]("new_column", col("existing_column") + 10)

2. lit : Add a constant value as a column.

python Copy code

df = [Link]("Country", lit("USA"))

3. alias : Rename a column temporarily.

python Copy code

[Link](col("name").alias("full_name")).show()

4. when and otherwise : Conditional column creation.

python Copy code

df = [Link]("category", when(col("value") > 100, "High").otherwise("Low"))

Aggregations

5. groupBy and agg : Perform group-by operations.

python Copy code

[Link]("category").agg(sum("sales").alias("total_sales")).show()

6. count : Count rows or grouped rows.

python Copy code

[Link]("category").count().show()

7. sum , avg , max , min : Aggregate functions.

python Copy code

[Link](sum("sales").alias("total_sales"),

[Link] 1/13
01/01/2025, 01:27 Missing Columns Debugging
avg("sales").alias("average_sales")).show()

8. distinct : Get distinct rows.

python Copy code

[Link]("column_name").distinct().show()

String Operations

9. concat : Concatenate multiple columns.

python Copy code

df = [Link]("full_name", concat(col("first_name"), lit(" "),

col("last_name")))

10. substr : Extract a substring.

python Copy code

[Link](col("name").substr(1, 5).alias("short_name")).show()

11. lower , upper : Convert to lowercase or uppercase.

python Copy code

df = [Link]("lower_name", lower(col("name")))

12. regexp_extract : Extract values using regex.

python Copy code

df = [Link]("area_code", regexp_extract(col("phone_number"), r"\((\d{3})\)", 1))

13. split : Split a string into an array.

python Copy code

df = [Link]("name_parts", split(col("name"), " "))

Date and Time Functions

14. current_date , current_timestamp : Get the current date and timestamp.

python Copy code

df = [Link]("today", current_date())

15. datediff : Difference between two dates.

python Copy code

df = [Link]("days_diff", datediff(col("end_date"), col("start_date")))

16. date_add , date_sub : Add or subtract days from a date.

[Link] 2/13
01/01/2025, 01:27 Missing Columns Debugging

python Copy code

df = [Link]("next_week", date_add(col("start_date"), 7))

17. year , month , dayofmonth : Extract parts of a date.

python Copy code

df = [Link]("year", year(col("start_date")))

Null Handling

18. fillna : Fill null values with a default value.

python Copy code

df = [Link]({"age": 25, "name": "Unknown"})

19. dropna : Drop rows with null values.

python Copy code

df = [Link](subset=["column_name"])

20. isNull , isNotNull : Check for null values.

python Copy code

[Link](col("name").isNull()).show()

Window Functions

21. row_number : Assign a unique row number within a window.

python Copy code

from [Link] import Window windowSpec =

[Link]("category").orderBy("sales") df = [Link]("rank",
row_number().over(windowSpec))

22. lag , lead : Access previous or next rows in a window.

python Copy code

df = [Link]("previous_value", lag("sales", 1).over(windowSpec))

Filtering

23. filter or where : Filter rows based on conditions.

python Copy code

[Link] 3/13
01/01/2025, 01:27 Missing Columns Debugging

[Link](col("age") > 30).show()

24. between : Check if a column value is in a range.

python Copy code

[Link](col("age").between(20, 30)).show()

Joins

25. join : Combine two DataFrames.

python Copy code

[Link](df2, df1["id"] == df2["id"], "inner").show()

Exploratory Data Analysis

26. describe : Generate summary statistics.

python Copy code

[Link]().show()

27. show : Display rows.

python Copy code

[Link](5)

28. count : Count total rows.

python Copy code

print([Link]())

29. columns : List column names.

python Copy code

print([Link])

30. printSchema : Display DataFrame schema.

python Copy code

[Link]()

Actions

31. collect : Retrieve all rows as a list.

[Link] 4/13
01/01/2025, 01:27 Missing Columns Debugging

python Copy code

rows = [Link]()

32. take : Retrieve the first n rows.

python Copy code

print([Link](5))

33. first : Retrieve the first row.

python Copy code

print([Link]())

34. toPandas : Convert to a Pandas DataFrame.

python Copy code

pdf = [Link]()

Caching and Persistence

35. cache : Cache the DataFrame in memory.

python Copy code

[Link]()

36. persist : Persist the DataFrame with specific storage levels.

python Copy code

from pyspark import StorageLevel [Link](StorageLevel.MEMORY_AND_DISK)

Pro Tips for Interviews

1. Know the Difference Between Transformations and Actions:

Transformations: Lazy operations (e.g., filter , select ).

Actions: Trigger execution (e.g., show , collect ).

2. Explain Partitioning and Optimization:

Use repartition() and coalesce() for tuning partitions.

Talk about Catalyst Optimizer in PySpark.

3. Discuss Real-World Scenarios:

Mention practical use cases like ETL, Data Cleaning, or Joins.

4. Understand Performance Tuning:

Mention caching, broadcasting, and avoiding shuffles.

[Link] 5/13
01/01/2025, 01:27 Missing Columns Debugging

This list covers a broad range of PySpark functions and scenarios. Let me know if you'd like a deeper
dive into any specific area!

check once again if anything missing abov

Here's a refined and complete list of PySpark functions and concepts for your interview preparatio
ensured no key functions or concepts are missed:

Comprehensive PySpark Function List

1. DataFrame Creation

1. createDataFrame : Create a DataFrame from a list or RDD.

python Copy cod

data = [("Alice", 34), ("Bob", 45)] columns = ["Name", "Age"] df =

[Link](data, columns)

2. read : Load data into a DataFrame.

python Copy cod

df = [Link]("file_path", header=True, inferSchema=True)

3. parallelize : Create an RDD and convert it to a DataFrame.

python Copy cod

rdd = [Link](data) df = [Link](columns)

2. Data Exploration

4. show : Display rows.

5. head : Retrieve the first row.

6. columns : List column names.

7. dtypes : Get column data types.

8. printSchema : Display DataFrame schema.

9. explain : Display execution plan.

python Copy cod

[Link](True)

10. summary : Summary statistics (mean, stddev, etc.).

python Copy cod

[Link]().show()

3. Column Manipulation

[Link] 6/13
01/01/2025, 01:27 Missing Columns Debugging

11. select : Select specific columns.

python Copy cod

[Link]("Name", "Age").show()

12. selectExpr : Use SQL-like expressions.

python Copy cod

[Link]("Name as full_name", "Age + 5 as new_age").show()

13. drop : Drop columns.

python Copy cod

df = [Link]("Age")

14. withColumnRenamed : Rename a column.

python Copy cod

df = [Link]("Name", "FullName")

4. Filtering and Conditional Logic

15. filter or where : Filter rows.

16. isin : Filter rows with specific values.

python Copy cod

[Link](col("Name").isin(["Alice", "Bob"])).show()

17. like : Pattern matching.

python Copy cod

[Link](col("Name").like("%A%")).show()

18. rlike : Regex matching.

python Copy cod

[Link](col("Name").rlike("^A.*")).show()

19. case-when : Conditional column logic.

python Copy cod

df = [Link]("category", when(col("Age") > 30, "Senior").otherwise("Junior"))

5. Aggregations

20. pivot : Perform pivoting.

python Copy cod

[Link]("category").pivot("month").sum("sales").show()

21. rollup and cube : Hierarchical aggregations.

python

[Link] 7/13
01/01/2025, 01:27 Missing Columns Debugging

Copy cod
[Link]("category", "sub_category").sum("sales").show()

22. approx_count_distinct : Approximation for large datasets.

python Copy cod

[Link](approx_count_distinct("column_name")).show()

23. collect_list and collect_set : Collect grouped elements.

python Copy cod

[Link]("category").agg(collect_list("values")).show()

6. Data Cleaning

24. replace : Replace values.

python Copy cod

df = [Link]("old_value", "new_value", "column_name")

25. [Link] : Fill null values.

26. [Link] : Replace null or specific values.

27. [Link] : Drop rows with null values.

7. Joins

28. join : Combine two DataFrames.

29. broadcast : Optimize joins by broadcasting small tables.

python Copy cod

from [Link] import broadcast df = [Link](broadcast(df2), "id")

8. String Functions

30. initcap : Capitalize each word.

python Copy cod

df = [Link]("capitalized", initcap(col("name")))

31. translate : Replace characters.

python Copy cod

df = [Link]("new_name", translate(col("name"), "a", "A"))

32. instr : Find substring position.

python Copy cod

df = [Link]("position", instr(col("name"), "substring"))

[Link] 8/13
01/01/2025, 01:27 Missing Columns Debugging
9. Date and Time Functions

33. to_date and to_timestamp : Convert strings to date/time.

python Copy cod

df = [Link]("date", to_date(col("date_string"), "yyyy-MM-dd"))

34. add_months , months_between : Date calculations.

python Copy cod

df = [Link]("next_month", add_months(col("start_date"), 1))

35. trunc : Truncate to the beginning of the month/year.

python Copy cod

df = [Link]("truncated_date", trunc(col("date"), "month"))

10. Performance Tuning

36. repartition : Increase or decrease partitions.

python Copy cod

df = [Link](10)

37. coalesce : Reduce partitions.

python Copy cod

df = [Link](1)

38. checkpoint : Create an execution checkpoint.

python Copy cod

df = [Link]()

11. Miscellaneous

39. orderBy or sort : Sort rows.

python Copy cod

df = [Link](col("Age").desc())

40. sample : Random sampling.

python Copy cod

[Link](fraction=0.1).show()

41. explode : Flatten an array or map column.

python Copy cod

df = [Link]("element", explode(col("array_column")))

42. json_tuple : Parse JSON columns.

python Copy cod

[Link] 9/13
01/01/2025, 01:27 Missing Columns Debugging

df = [Link](json_tuple(col("json_column"), "key1", "key2"))

43. udf (User-Defined Functions): Apply custom Python logic.

python Copy cod

from [Link] import udf from [Link] import StringType def

custom_func(value): return [Link]() udf_func = udf(custom_func, StringType())
df = [Link]("new_column", udf_func(col("existing_column")))

Missing Concepts from Previous List

Pivot, Rollup, and Cube for hierarchical aggregations.

Broadcast Joins for optimization.

Explode and JSON Functions for handling nested data.

UDFs for custom transformations.

Approximate Aggregations like approx_count_distinct .

Pro Tip
1. Understand Catalyst Optimizer: Mention how PySpark optimizes queries using DAGs (Directe
Acyclic Graphs).

2. Mention Use Cases: For example:

Using PySpark for ETL.

Working with big data in distributed environments.

3. Discuss Performance: Mention strategies like:

Broadcast joins.

Repartition vs. Coalesce.

Persist vs. Cache.

Let me know if you'd like additional details or examples for any function!

### **Pro Tip** 1. **Understand Catalyst Optimizer**: Mention how PySpark optimizes queries using DAGs
(Directed Acyclic Graphs). 2. **Mention Use Cases**: For example: * Using PySpark for ETL. * Working with big dat
in distributed environments. 3. **Discuss Performance**: Mention strategies like: * **Broadcast joins**. *…

Give this concepts with exmaple for my interview prospect

Detailed Concepts with Examples for PySpark Interview Preparation

1. Understand Catalyst Optimizer

The Catalyst Optimizer in PySpark is an internal query optimizer for transforming logical query pl
into optimized physical query plans. It uses rule-based and cost-based optimization to improve
query performance.

Key Points:

[Link] 10/13
01/01/2025, 01:27 Missing Columns Debugging

It analyzes and optimizes query execution.

Automatically applies techniques like predicate pushdown, column pruning, and join
optimization.

Converts transformations into a Directed Acyclic Graph (DAG) for efficient execution.

Example:

python Copy cod

from [Link] import SparkSession spark =

[Link]("CatalystOptimizerExample").getOrCreate() data = [("Alice"
34), ("Bob", 45), ("Cathy", 29)] columns = ["Name", "Age"] # Create DataFrame df =
[Link](data, columns) # Logical and physical plans are optimized by
Catalyst Optimizer df_filtered = [Link](df["Age"] > 30) # View the execution plan
df_filtered.explain(True) # Shows the DAG and physical plan

2. Mention Use Cases

PySpark is widely used in real-world applications for big data processing. Be prepared to discuss
scenarios where PySpark can solve problems.

Example Use Cases:

1. ETL with PySpark: PySpark can extract data from sources like S3 or HDFS, transform it, and lo
it into a target database.

python Copy cod

# Extract df = [Link]("s3://bucket/[Link]", header=True,

inferSchema=True) # Transform df_transformed = [Link]("updated_col",
df["value"] * 2) # Load
df_transformed.[Link]("overwrite").csv("s3://bucket/[Link]")

2. Big Data Processing: PySpark handles distributed processing of large datasets, making it idea
for batch processing or real-time streaming.

python Copy cod

# Read a large dataset and perform transformations large_df =

[Link]("s3://bucket/large_dataset.csv", header=True, inferSchema=True)
aggregated_df = large_df.groupBy("category").sum("sales") aggregated_df.show()

3. Discuss Performance

PySpark provides several techniques to optimize the performance of large-scale data processing.

a. Broadcast Joins

Use broadcast joins when one of the datasets is small enough to fit into memory. This reduces the
data shuffle during joins.

python Copy cod

from [Link] import broadcast # Small dataset to broadcast small_df =

[Link]([("A", 1), ("B", 2)], ["key", "value"]) # Large dataset large_df
[Link]([("A", 100), ("B", 200), ("C", 300)], ["key", "amount"]) # Perfor
a broadcast join result = large_df.join(broadcast(small_df), "key") [Link]()

[Link] 11/13
01/01/2025, 01:27 Missing Columns Debugging

b. Repartition vs. Coalesce

repartition : Increases or decreases the number of partitions. It involves a full shuffle of dat

coalesce : Reduces the number of partitions without a full shuffle, which is faster.

python Copy cod

# Repartition to increase partitions repartitioned_df = [Link](10) # Coalesce

to reduce partitions coalesced_df = [Link](1)

c. Persist vs. Cache

cache : Stores the DataFrame in memory only.

persist : Allows storing in memory and disk (or other storage levels).

python Copy cod

from pyspark import StorageLevel # Cache DataFrame [Link]() # Persist DataFrame

[Link](StorageLevel.MEMORY_AND_DISK)

When to Use:

Use cache or persist when a DataFrame is reused multiple times to avoid recomputation.

4. Combine Concepts for Real-World Optimization Example

Here’s a scenario combining the above techniques:

python Copy cod

# Load large data large_df = [Link]("s3://bucket/large_data.csv", header=True,

inferSchema=True) # Broadcast join with a small lookup table small_df =
[Link]("s3://bucket/small_lookup.csv", header=True, inferSchema=True) result_df
= large_df.join(broadcast(small_df), "key") # Transform data transformed_df =
result_df.withColumn("discounted_price", col("price") * 0.9) # Cache the transformed
DataFrame for reuse transformed_df.cache() # Aggregate data aggregated_df =
transformed_df.groupBy("category").agg(sum("discounted_price").alias("total_sales")) #
Write output aggregated_df.[Link]("overwrite").csv("s3://bucket/output/")

Tips for Interview

1. Understand Execution Plans:

Use .explain() to demonstrate knowledge of logical and physical plans.

Discuss DAG optimization and shuffles.

2. Highlight Real-World Applications:

Discuss scenarios where PySpark improved processing time or handled large datasets
efficiently.

3. Performance Tuning Insights:

Focus on reducing shuffles with broadcast joins and partition tuning.

Emphasize using caching and persistence for iterative operations.

[Link] 12/13
01/01/2025, 01:27 Missing Columns Debugging

[Link] 13/13