1)Avoid Data Skewness by using salting technique
2)Use Colase instead of repartion
3)Use reduce by key instead of group by key
4)Always reduce the shuffle operations and minimize the group by key operations
5) Choose the right file format like when we are qowking with coloum base go with the parquet and
row based avro
6)Use broad cast joins
7)Use kyro serailzer for better performance
8) Always give more than 10% of memory to memory overhead in both exeutoe and driver
9)Choose correct level of Storage level
10)Enable Adaptive query execution
11)Use predicate push down
12)Use aprtion pruning
13)Use bucketing and partioining when it is required
14) Always use the Dataframe sinstead of RDD
15)Use optimize that will club all small files into single files
16) use vaccum for every 168 hours
17) when working with analytics projects prefer the parquet instead of avro
1)Apollo Hospitals
2)grasim
[Link]
Interview Question: Why is Parquet faster in PySpark?
Most people say:
👉 “Because it’s a columnar format”
That’s true…
But that’s only part of the story.
Parquet is faster because Spark is designed to optimize around it.
𝙏𝙇;𝘿𝙍:
Parquet improves Spark performance through:
✅ Columnar storage
✅ Predicate pushdown
✅ Compression
✅ Partition pruning
✅ Reduced IO
Here’s the simple mental model I use in interviews (and performance tuning):
1️⃣ Columnar storage
Unlike CSV or JSON,
Parquet stores data column-wise instead of row-wise.
So if your query needs:
select name, salary
👉 Spark reads ONLY those columns
👉 Not the entire dataset
Less data read = faster jobs.
2️⃣ Predicate pushdown
Example:
filter(col("salary") > 100000)
👉 Spark pushes filters down to the Parquet reader
👉 Irrelevant row groups are skipped entirely
This avoids unnecessary scanning at the storage level itself.
3️⃣ Built-in compression
Parquet compresses similar column values extremely well.
Result:
✅ Smaller files
✅ Less disk IO
✅ Faster network transfer
✅ Lower storage cost
And Spark can still read it efficiently without full decompression overhead.
4️⃣ Partition pruning
If data is partitioned like:
year=2025/month=05
And your query asks only for:
month=05
👉 Spark skips all unrelated partitions
👉 Huge reduction in data scanned
This becomes critical at TB/PB scale.
5️⃣ Spark’s optimizer loves Parquet
Catalyst Optimizer + Tungsten execution engine are heavily optimized for columnar formats.
That means:
Better query plans
Vectorized reads
Efficient memory usage
Fewer CPU cycles
CSV works.
Parquet works WITH Spark.
🧠 Interview One liner (gold):
“Parquet is faster in Spark because it minimizes IO using columnar storage, predicate pushdown,
compression, and partition pruning.”
Do you know how to calculate the size of a Spark cluster (Executors, Cores & Memory) based on the
data file size? 🤔💻
Here’s a simple way to answer this common PySpark / Azure Databricks Interview Question 👇
📌 Scenario-👉 “Suppose you have a 10 GB file to process in PySpark. How would you calculate
👉 Input Data Size = 10 GB
Consider Default Spark Partition Size = 128 MB
🔹 Step 1: Calculate Number of Partitions
No. of Partitions = Total Data Size / Partition Size
10×1024÷12810 \times 1024 \div 12810×1024÷128=80 Partitions= 80 \text{ Partitions}=80
Partitions✅ Total Partitions = 80
🔹 Step 2: Calculate Number of Tasks / Cores
👉 In Spark:
1 Partition = 1 Task
Ideally, 1 Core executes 1 Task at a time
So,
✅ 80 Partitions = 80 Tasks = 80 Cores Required
🔹 Step 3: Calculate Number of Executors
👉 Assume:
1 Executor = 8 Cores
So,
No. of Executors = Total Cores / Cores per Executor
80÷8=1080 \div 8 = 1080÷8=10✅ Total Executors Required = 10
🔹 Step 4: Calculate Executor Memory
👉 Total Data Size = 10 GB
Distribute across executors:
10 GB / 10 Executors = 1 GB per Executor
But in real-time Spark execution, extra memory is required because:
✅ Spark uses memory for:
Shuffle Operations 🔄
Caching 📦
Execution Memory ⚡
Overhead Memory 🧠
🔹 Step 5: Apply Memory Factor
👉 Generally, we consider approximately 3.5x memory overhead
Reason:
Around 60% Spark Pool Memory Allocation
Around 50% Executor Memory Usage
Additional buffer for shuffle & spill operations
So,
1 GB × 3.5 = 3.5 GB
✅ Recommended Executor Memory = 3.5 GBemp table
id name sal
1 A 1000
2 B 1000
3 C 2000
4 D 3000
with cte as(
select *,dense_rank() over (order by sal) as rnk from emp)
select * from cte
where rnk=1
Select id,count(*) from emp
group by id
having count(*)>1
id from to
101 hyd banglore
101 banglore chennnai
102 hyd banglore
102 banglore hyd
with CTE as(
Select id,from as places from table
uninon all
select id,to as places from table)
101 hyd
101 banglore
101 banglore
101 chennnai
select *,
df=[Link]("delimiter",',').option("mode","PERMESSIVE").csv("");
l1=[1,2,5,7]
for x in range(0,9):
if x not in l1:
[Link](x)
MERGE INTO TARGET_TBL T USING SOURCE_TBL S ON [Link]=[Link]
WHEN MATCHED THEN UPDATE SET()
WHEN NOT MATCHED THEN INSERT()VALUES()
MERGE TARGET_TBL T USING SOURCE_TBL S ON [Link]=[Link]
WHEN MATCHED THEN UPDATE SET()
WHEN NOT MATCHED THEN INSERT() VALUES ();
WITH OrderedPeriods AS (
SELECT start_date, end_date,
MAX(end_date) OVER(ORDER BY start_date ROWS BETWEEN UNBOUNDED PRECEDING AND 1
PRECEDING) AS max_end_so_far
FROM intervals
),
IslandTriggers AS (
SELECT start_date, end_date,
CASE WHEN max_end_so_far >= start_date THEN 0 ELSE 1 END AS is_new_island
FROM OrderedPeriods
),
IslandIDs AS (
SELECT start_date, end_date,
SUM(is_new_island) OVER(ORDER BY start_date ROWS UNBOUNDED PRECEDING) AS island_id
FROM IslandTriggers
),
MergedIslands AS (
SELECT MIN(start_date) AS island_start, MAX(end_date) AS island_end
FROM IslandIDs
GROUP BY island_id
SELECT SUM(DATEDIFF(day, island_start, island_end) + 1) AS total_overlapping_days
FROM MergedIslands;