Python

1. Data Types and Variables: - Variable is nothing but a container

which can holds some value.
Computer based question’s:
1) CPU Full form?
A) Central processing unit.

2) Brain of the computer is?

A) CPU

3) What is RAM?
A) Random Access Memory.

4) What we called when computer switch on?

A) Booting

5) One byte equal to how many bits?

A) 8 bits

6) What is OS?
A) Operating System

7) What we called Windows, Linux and Mac ?

A) Operating System

8) How many function keys in a Keyboard?

A) 12

9) What is a Printer?
A) Output Device
10) What is MS office, Excel, powerpoint ?
A) Application Software
11) Ctrl + C ?
A ) Copy
12) Ctrl + V?
A) Paste

13) Which software is used for check webpages?

A) Browser
14) What is wifi?
A) Wirelessly
15) Pdf Means?
A) portable document format
16) What is Google sheet?
A) Excel Sheet
17)Which type memory is ROM memory?
A) Permanent memory
18) What is Screenshot Key?
A) PrtSc(print screen)Key
19) What we called computer closing?
A) Shutdown
20) What is spelling check shortcut in MS Word?
A) F7
21) What is Shortcut key for text BOLD in MS Word?
A) ctrl+B
22) start with which symbol in a Excel formula?
A) =
23) Shortcut key for slide changing in powerpoint?
A) ctrl +N
24)Sum function in excel?
A) = sum()
25) How to save document in word?
A) ctrl + s
26) rows insert shortcut in excel?
A) ctrl + shift + +
27)PPT means?
A) Power point presentation
28) Text underline shortcut in word?
A) ctrl + u
29) how to showed column name in excel?
A) A,B,C….
30) Paragraph alignment keys in MS word?
A) ctrl +L|E|R
31)How to insert filter in Excel?
A) ctrl + shift + L
32) page break shortcut in word?
A) ctrl +enter
33) text copy and paste shortcut?
A) ctrl + c and ctrl + v
34) max column count in excel?
A) 16384
35)where we get themes in power point?
A) design
36) for text select in word?
A) ctrl + A
37) average formula in excel?
A) =Average()
38) where Is header and footer tab in word?
A) Insert
39) chart insert in excel shortcut?
A) Alt + F1
40) find shortcut in word?
A) ctrl + F

41) Array
42) list
43)Database
44) Threads
45) Networks
46) Memories in C
47) DBMS
48) SQL Basics
49) Operating Systems
50) Python Functions
Part-1
Data Engineer Interview - Coding
1) SQL MOST ASKED
2) PYTHON CODING
3) PYSPARK CODING
SQL Tutorial:-
1) Database
--1. How to Create Database
CREATE DATABASE RAJESH
--[Link] TO USE DATABASE (GO TO SPECIFIC
DATABASE)
USE DATABASE EMPLOYEE

--[Link] TO DELETE DATABASE

DROP DATABASE EMPLOYEE

--[Link] TO CREATE TABLE

CREATE TABLE
CREATE TABLE EMP(
ID INT,
NAME VARCHAR(30),
 ADDRESS VARCHAR(30),
SALARY INT
)
2) Insert
CREATE TABLE employee(
employee_id INT,
name VARCHAR(30),
salary INT,
loc VARCHAR(30),
)

select *from employee

INSERT INTO employee (employee_id, name, salary, loc)

VALUES (1, 'manish', 10000, 'India'),
(2, 'Neha', 5000, 'India'),
(4, 'Surya', 5000, 'UK')
3) Constraints
SQL constraints are rules applied to columns in a database table to enforce data integrity and
ensure the accuracy and reliability of your data.

SQL constraints are rules applied to columns in a database table. Maintain data integrity and
data will be correct and consistent

In SQL constraints apply rules to columns in a database table. For this data will follows
consistency, unique, accuracy and integrity.

Types of SQL Constraints:

NOT NULL: Prevents NULL values in specified columns.

UNIQUE: Ensures all values in a column are distinct.

PRIMARY KEY: Uniquely identifies each row in a table.

FOREIGN KEY: Enforces relationships between tables for referential integrity.

 CHECK: Validates that data meets specific criteria.

DEFAULT: Sets default values for columns when none are provided.

4) Not null & unique

5) Check & default
6) Primary key
7) Foreign Key

8) Filter & Sort

You have a table and perform some analysis work. In that can Filtering
and Sorting is most important operation.
How to filter and sort data efficiently in SQL using WHERE, ORDER BY, and other essential clauses.

Filtering with WHERE The WHERE clause is used to filter rows based on specific conditions.

Sorting with ORDER BY The ORDER BY clause helps you sort data in ascending (ASC) or descending
(DESC) order.

9) Delete vs Drop vs Truncate

We all use these for deleting purpose.
Delete: - When you delete any specific row or complete data from
table.
Truncate: - Whenever we have to delete complete data from table then
we use truncate.
Drop: - Removes the entire table or database, including all its data and structure also.
10) Update
Update: - The UPDATE command allows you to modify one or more rows in a table based on specific
conditions.
11) Conditional Statement
12) Aggregate Function
13) Group by
The GROUP BY clause groups rows that share the same values in specific columns. It’s often combined
with aggregate functions like SUM(), COUNT(), AVG(), MIN(), and MAX() to summarize data.

14) Like Operator

The LIKE operator is used in SQL to match specific patterns in text fields. It’s helpful when you need to
filter data based on partial matches or wildcards.

The LIKE operator is used in a WHERE clause to search for a specified pattern in a column.

15) Having Clause

The HAVING clause filters the results of aggregate functions (like totals and averages) used with the
GROUP BY clause. It works similarly to the WHERE clause, but HAVING is used to filter aggregated data.

The SQL Having clause is similar to the WHERE clause, are used to filter rows based on specified criteria.

Having clause was added to SQL because the WHERE keyword cannot be used with aggregate functions.

16) Top or Limit

17) Distinct
The DISTINCT keyword ensures that only unique rows are returned, eliminating any duplicate data from
your query result.

18) Coalesce
Whenever deal with any null value in a table we use coalesce function.

The COALESCE function returns the first non-NULL value from a list of arguments. It’s useful when
dealing with missing data or optional fields. Examples of COALESCE Usage Replace NULL with a default
value:

19) Joins
20) Union Vs Union all
The UNION operator combines the result sets of two or more SELECT queries and removes duplicate
rows.

The UNION ALL operator also combines result sets, but it keeps duplicate rows.

21) Alter
The Alter Statement is used to change the structure of a table after it
has been created.
This helps when you need to update the schema without losing existing
data.
22) Windows Function
23) Rank and DenseRank
24) Lead and Lag
25) CTE in Sql
26) Views in Sql
27) Stored Procedures
28) Subquery
29) Triggers in Sql

SQL Questions
--Query 1. Remove duplicate values from employee table
Select distinct *from employee
--Query [Link] a query to find out duplicate values from employee
SELECT emp_id, COUNT(*) AS count
FROM employee
GROUP BY emp_id
HAVING COUNT(*) > 1;
(or)
--display duplicate values
-- use Windows function row_number() with over()
clause and use partition by based
-- based on employee_id i want to partition
it(means grouping emp_id) and user order by
employee_id
-- and i give this as rank from employee table.
-- Try to group employee_id and try to find out
row numbers.
-- from hear we want filter the data then we
want to find out
-- what are the duplicate values present with
the use of(with cte as() function)
with cte as(
select*, row_number() over(partition by
employee_id order by employee_id)as rank from
emp
)
select *from cte where rank=2
--Query [Link] a query to find out highest earning employee based on
each position
Select max(salary), position from employee group
by position
--Query [Link] a query to get top 3 highest earning employee
SELECT TOP 3 Emp_ID, Emp_Name, Salary
FROM Employee
ORDER BY Salary DESC
--Query [Link] a query to get top 3 lowest earning employee
SELECT TOP 3 Emp_ID, Emp_Name, Salary
FROM Employee
ORDER BY Salary ASC;
Video 2
--Query [Link] a query to find out 2nd highest salary employee
SELECT MAX(Salary) AS SecondHighestSalary
FROM Emp
WHERE Salary < (SELECT MAX(Salary) FROM Emp)
--Query [Link] a query to get 2nd lowest earning employee
SELECT MIN(Salary) AS SecondLowestSalary
FROM Emp
WHERE Salary > (SELECT MIN(Salary) FROM Emp)
--Query [Link] a query to get 2nd highest salary based on each
department
with cte as(
Select * , row_number() over(partition by
department order by salary desc) as rn from emp
)
Select * from cte where rn=2
--Query [Link] a query to get 3rd lowest salary based on each location
with cte as(
Select * , row_number() over(partition by
location order by salary asc) as rn from emp
)
Select * from cte where rn=3
--Query [Link] a query to get bottom 2 salary based on each location
with cte as(
Select * ,row_number() over(partition by
location order by salary asc) as rn from emp
)
Select * from cte where rn in (1,2)
--Query [Link] a query to get top 2 salary based on each department
with cte as(
Select * ,row_number() over(partition by
location order by salary desc) rn from emp
)
Select * from cte where rn in (1,2)
Video 3
--Query [Link] many rows will you get when you will perform inner
join
Only matching records from the both tables will
be present.
Select * from table1 join table2 on
[Link]=[Link]
--Query [Link] many rows will you get when you will perform left join
All the records from the left table and only the
matching records from the right table.
Select * from table1 left join table2 on
[Link]=[Link]
--Query [Link] many rows will you get when you will perform right
join
All the records from the right table and only
the matching records from the left table
Select * from table1 right join table2 on
[Link]=[Link]
--Query [Link] many rows will you get when you will perform Full join
All the records you get from both tables weather
its matching or not.
Select * from table1 full join table2 on
[Link]=[Link]
Video 4
--Query [Link] a query to create new table with same schema as
employee table (only schema)
Select * into table3 from emp2 where 1=2
--Query [Link] new table same like employee table (data + schema)
Select *into table4 from emp2 where 1=1
--Query [Link] a query where employee name starts with letter A
Select * from employee where employee_name like
'A%'
--Query [Link] a query where department_id starts letter ends letter
is same
Select * from employee where
left(department_id,1)=right(department_id,1)
--Query [Link] a query to get records in xml format
Select * from employee for xml auto
--Query [Link] to get current date
Select getdate()
(or)
Select CURRENT_TIMESTAMP
--Query [Link] to get current month
select month(getdate())
--Query [Link] to get current year
select year(getdate())
Video 5
--Query 24.D/B Union and Union all
UNION: Combines the results of two queries and
removes duplicate rows.
UNION ALL: Combines the results of two queries
without removing duplicate rows.
--Query 25.D/B Primary key v/s Union
primary key - Uniquely identifies each record
(Or) Used to serve as a unique identifier for
each row in a table.

Cannot accept NULL values.

Only one primary key

UNIQUE KEY- Prevents duplicate values in a

column (or) uniquely determines a row that isn’t
the primary key.
Can accept NULL values.
More than one unique key

emp_id primary key mobile number unique

--Query 26.D/B Rank and Dense Rank

ROW_Number(): unique numbering for each row, no
duplicates.
RANK(): Skips ranks if there are ties.
DENSE_RANK(): Does not skip ranks, even if there
are ties.

Row_Number rank dense rank salary

1 1 1 100000
2 2 2 70000
3 2 2 70000
4 4 3 50000

--Query [Link] vs Drop vs Truncate

DELETE: - Remove selected records.
The SQL DELETE command is a DML (Data
Manipulation Language) command that deletes
existing
Records from the table in the database. It can
delete one or more rows from the table
Depending on the condition given with the WHERE
clause
TRUNCATE: - Quickly clear a table but keep
schema.
The TRUNCATE command helps us delete the
complete records from an existing table

DROP: - Remove table completely

The DROP command drops the existing table from
the database. It only requires the name of the
table to be dropped.

--Query [Link] are different Windows function

Window functions are special functions that
perform calculations across a set of table rows
related to the current row. Examples include:
ROW_NUMBER()->Assigns sequential numbers to
rows. no duplicates.
RANK()->Assigns rank and Skips ranks if there
are ties.
DENSE_RANK()->Assigns rank and Does not skip
ranks, even if there are ties.
AVG() OVER()->Applied over a window of rows.
--Query 29.D/B Where and Having Clause
WHERE: Where clause Filters rows before grouping
(GROUP BY) or aggregation. WHERE works on
individual records.

HAVING: Having clause Filters rows after

aggregation. groups created by GROUP BY. HAVING
works on aggregated results.

--Query [Link] to handle null values in sql

Use functions like:
IS NULL or IS NOT NULL to filter records.
COALESCE(column, default_value) to replace NULL
with a default value.
Video 6
--Query [Link] a query to find out the employees manager details
from employee table
select [Link], [Link] as manager_name ,
[Link] from emp4 a left join emp4 b
on [Link]=[Link]
--Query [Link] a query to find out cumulative sum of salary
In SQL, this means accessing all the previous rows, summing them, and
adding the sum to the current row's value.
select * from employee

select * , sum(salary) over(order by

employee_id) as rn from employee
--Query [Link] Null value with previous values
with cte as(
select * , row_number() over(order by (select
null)) rn ,
case when brand_name is null then 0 else 1 end
rn1
from chocolate_brands
)
,cte1 as(
select * , sum(rn1) over(order by rn) roll_sum
from cte
)
select chocolate_name, brand_name ,
max(brand_name) over(partition by roll_sum) as
new_brand_name from cte1
--Query [Link] new and repeated customer on each date
1) Find how many duplicate records there in a table?
2) Remove duplicate from table?
3) Find second highest salary from table?
4) Find maximum salary based on their post?
5) Find cumulative sum of salary where it should be sum based on id
column?
6) Find previous salary of employee based on name?
7) Find next salary of employee based on name?
8) emp salary greater then 10L are senior , salary 10L are junior and less
then 10L are Intern?
9) Join emp and dept table based on id and find out number of record?
10) How can you create empty tables with the same structure as
another table?
11) Find highest salary based on post?
12) Create table with same schema?
13) Copy data with same table structure and schema?
14) D/B Union and Union all?
15) Drop vs Delete vs Truncate?
16) What is Intersect in SQL?
17) Find ODD rows from table?
18) How to find unique records from table?
--Query 1. Remove duplicate values from employee table
--Query [Link] a query to find out duplicate values from employee
--Query [Link] a query to find out highest earning employee based on
each position
--Query [Link] a query to get top 3 highest earning employee
--Query [Link] a query to get top 3 lowest earning employee

--Query [Link] a query to find out 2nd highest salary employee

--Query [Link] a query to get 2nd lowest earning employee
--Query [Link] a query to get 2nd highest salary based on each
department
--Query [Link] a query to get 3rd lowest salary based on each location
--Query [Link] a query to get bottom 2 salary based on each location
--Query [Link] a query to get top 2 salary based on each department
--Query [Link] many rows will you get when you will perform inner
join
--Query [Link] many rows will you get when you will perform left join
--Query [Link] many rows will you get when you will perform right
join
--Query [Link] many rows will you get when you will perform Full join
--Query [Link] a query to create new table with same schema as
employee table (only schema)
--Query [Link] new table same like employee table (data + schema)
--Query [Link] a query where employee name starts with letter A
--Query [Link] a query where department_id starts letter ends letter
is same
--Query [Link] a query to get records in xml format
--Query [Link] to get current date
--Query [Link] to get current month
--Query [Link] to get current year

--Query 24.D/B Union and Union all

--Query 25.D/B Primary key v/s Union
--Query 26.D/B Rank and Dense Rank
--Query [Link] vs Drop vs Truncate
--Query [Link] are different Windows function
--Query 29.D/B Where and Having Clause
--Query [Link] to handle null values in sql

--Query [Link] a query to find out the employees manager details

from employee table
--Query [Link] a query to find out cumulative sum of salary

--Query [Link] Null value with previous values

--Query [Link] new and repeated customer on each date
Python imp queries
Control Statements
1) Python program to check Prime Number?

def prime(n):
if n==0 or n==1:
print("not prime")
elif n==2:
print("prime")
else:
for i in range(2,n):
if n%i==0:
print("not prime")
break
else:
print("prime")
prime(773)

2) Python program to find the Factorial of a Number? 5*4*3*2*1

def factorial_iterative(n):
def fact(n):
if n==0 or n==1:
return 1
else:
print(n)
return n*fact(n-1)
fact(5)
# Example usage
num = 5
print(f"Factorial of {num} is: {factorial_iterative(num)}") # Output: 120

3) Python program for nth Fibonacci number? 0,1,1,2,3,5,8,13

def fib(n):
if n==1:
return 0
elif n==2:
return 1
else:
return fib(n-1)+fib(n-2)
fib(6)
output: 5 - 0,1,1,2,3,5,8…..
4) Python program for Sum of squares of first n natural numbers?
1+4+9+16+25=55
def natural(n):
sum=0
 for i in range(1,n+1):
sum+=i*i
return sum
natural(5)
# Output: Sum of squares: 55
# (Because 1^2 + 2^2 + 3^2 + 4^2 + 5^2 = 55)

5) Armstrong number is a number that is equal to the sum of its digits,

each raised to the power of the number of digits in the number. For
example, 153 is an Armstrong number because 1^3 + 5^3 + 3^3 = 153.
def armstrong(n):
sum=0
temp=n
while temp>0:
digit=temp%10
sum+=digit**3
temp//=10
if n==sum:
print("armstrong")
else:
print("not armstrong")
armstrong(153)
Armstrong
Arrays
6) Python program to find Sum of Array
arr = [10, 20, 30, 40, 50]
result = sum(arr)
print("Sum of array elements:", result)
# Output: Sum of array elements: 150

7) Python program to find Largest Element in an Array

arr = [10, 24, 45, 90, 3, 67]
largest = max(arr)
print("The largest element is:", largest)
# Output: The largest element is: 90

8) Python program to Array Rotation

def left_rotate(arr, d):
d = d % len(arr)
return arr[d:] + arr[:d]
# Example usage
arr = [1, 2, 3, 4, 5]
print(left_rotate(arr, 2)) # Output: [3, 4, 5, 1, 2]
String
11) Python program to check if a String is Palindrome or Not
string = "madam"
a=''
for i in string:
a=i+a
if a==string:
print("palindrome")
else:
print("not palindrome")

12) Python program to Reverse Words in a Given String in Python

String = "Hello World"
list = [Link]()
list[::-1]
a=' '.join(list[::-1])
a

13) Python program to How to Remove Letters from a String in Python

s = "hello world"
s = [Link]("l", "")
print(s) # Output: heo word

14) Python program to Words Frequency in String

from collections import Counter

s = "hello world hello everyone"

word_freq = Counter([Link]())
print(word_freq)
# Output: Counter({'hello': 2, 'world': 1, 'everyone': 1})
15) Python program to Remove All Duplicates from Given String in
Python
s = "geeksforgeeks"
res = "".join([Link](s))
print(res) # Output: geksfor

DataTypes
9) Python program to interchange first and last elements in list
# Initialize the list
my_list = [1, 2, 3, 4, 5]

# Swap first and last elements

my_list[0], my_list[-1] = my_list[-1], my_list[0]

print("List after swapping:", my_list)

# Output: [5, 2, 3, 4, 1]

10) Python program to Swap Two Elements in a List

# Define the list
a = [10, 20, 30, 40, 50]

# Indices to swap (for example: swap elements at index 0 and 4)

i, j = 0, 4
# Swap using tuple assignment
a[i], a[j] = a[j], a[i]

print(a) # Output: [50, 20, 30, 40, 10]

16) Python program to find the Sum of all items in a Dictionary
d={'a':100,'b':20,'c':30}
Result=sum([Link]())
print(Result)
o/p: 150
17) Python program dictionary duplicate values append
d={'a':100,'b':200,'c':300,'e':100,'f':200}
Result = {}
for key,value in [Link]():
if value not in [Link]():
Result[key] = value
print(Result)

18) D/B Deep copy and shallow copy

Shallow Copy: Create a new object but references the original objects
for nested elements ([Link]()) in copy module).
Deep Copy: Creates a completely independent copy, including nested
elements (eg. [Link]()) in copy module).

19) D/B List and Tuple

Interview Audio Q & A
1) Explain your experience and day 2 day task in your current
project?
A) 1) Total 8+ years of experience in to IT industry. Out of 8 years I
worked as a data engineer about 4 years and remaining 4 years
worked as a Hybris developer.
2) It’s an e-commerce domain. Where will be create e-commerce
application like b2b and b2c.
3) Later on I got a chance to work on azure services in my first
company.
4) I worked on companies like Capgemini, cognizant and my
previous company Tech Mahindra.
5) I worked on Azure data factory, data bricks, Pyspark, sql and
python

6) My last project is Haleon. Haleon is a healthcare company. This

company is one of the world's largest global consumer health
providers such as Sensodyne and centrum.

7) Where will get customer data from multiple sources to the

target system.
And source system like we get data from Tibco source system and
also like flat files and also like sql server from there we will to
some other jones the final target system like as azure synapse.
 8) Actually we will be have into sprints. Generally our sprint will
be in up to 10 days. In 1 sprint I will have 1 task. The task like
where I need to delete the Decommissions delta tables. For
that I have to write the script to delete the delta table’s data
but not the structure of the table we want to keep the
structure of the table and also optimize the delta table.
9) Coming to my day 2 day activates like we will be having daily
calls with our team. If I’m struck any ware I seeking help with
my lead and then I got to forward.
10) Copy data from Rest-API to ADLS/Blob Storage.
11) Skip Empty Files Scenarios.
12) Download from API – Unzip and copy scenario projects.
[Link]
id=1bmmtkc5NLZQ4iU6Qz_7di-juuxA2kT1t&export=download
13) Delete file scenario.
14) Copy Latest file scenario.
15) Delete file from blob storage if file size is greater than 1MB.
16) Copy multiple tables using adf.
17) How to count number of files available in blob storage
account.
18)

2) May know what are all the things you consider to create pipeline?

Key Considerations for Creating an Azure Data Factory (ADF)

Pipeline When designing and building an Azure Data Factory
pipeline, consider the following aspects to ensure your solution is
robust, scalable, and maintainable:

1. Define Objectives and Requirements

  Clearly outline the business goals and data integration
requirements.
 Identify data sources, destinations, and transformation needs.
 Set performance benchmarks and KPIs to measure pipeline
success

2. Pipeline Structure and Modularity

 Design pipelines with a clear, organized structure for easier

management and troubleshooting Use modular and reusable
components to simplify updates and maintenance Parameterize
pipelines to make them dynamic and reusable for different data
scenarios

3. Activities and Workflow Design

 Choose the right mix of activities: data movement (e.g., Copy

Data), transformation (e.g., Data Flows), and control flow (e.g.,
ForEach, If Condition)

Use parallel execution (e.g., ForEach with parallelism) to improve

performance when processing multiple tables or files

4. Linked Services and Datasets

 Set up linked services to securely connect to all data sources and

destinations

 Define datasets that represent the structure and location of your

data.

5. Integration Runtime Configuration

  Select the appropriate integration runtime (Azure IR, Self-hosted
IR, or SSIS IR) based on your data movement and transformation
needs Optimize IR location and size for performance and cost
efficiency

6. Data Movement and Transformation Optimization

 Use incremental loading to process only new or changed data,

reducing resource usage and improving speed Compress data
during transfer to save bandwidth and speed up movement
Minimize unnecessary data movement and transformations by
processing data close to its source

7. Error Handling and Monitoring

 Implement robust error handling with retry policies for transient

failures
 Set up alerts and notifications for pipeline failures or performance
issues using Azure Monitor. Monitor pipeline performance and
resource usage regularly to identify and resolve bottlenecks

8. Security and Compliance

 Secure connections to data sources and destinations using

managed identities or secure keys.
 Apply data governance and compliance policies as required by
your organization

9. Scalability and Performance Testing

 Test pipelines with sample datasets to establish performance

baselines

Adjust batch sizes, parallelism, and compute resources to optimize

throughput and cost
10. Documentation and Collaboration

 Document the pipeline design, logic, and dependencies for future

reference and team collaboration

By considering these factors, you can create Azure Data Factory

pipelines that are efficient, reliable, and aligned with business
objectives, while also being easy to maintain and scale as your data
needs grow

3) How can be access database in pipeline? Any methods or steps to

connect database in pipeline to access the data?

To access a database (such as Azure SQL Database, SQL Server, etc.) in

an Azure Data Factory (ADF) pipeline, you need to establish a secure
connection and configure your pipeline to use it. Here are the main
methods and steps:

1. Create a Linked Service

A Linked Service in ADF defines the connection information to your data

source (database).

 Go to the Manage tab in your ADF instance.

 Click on Linked services.
 Click + New to create a new linked service.
 Select your database type (e.g., Azure SQL Database, SQL Server,
etc.).
 Enter the required connection details (server name, database
name, authentication method).
 For Azure SQL Database, you can use authentication methods like
SQL authentication, Managed Identity, or Key Vault secrets

 Test the connection and create the linked service.

2. Use the Linked Service in Your Pipeline

 In your pipeline, add activities such as Copy Data, Lookup, or

Stored Procedure.
 When configuring the activity, select the linked service you
created as the source or sink (destination).
 For data movement or transformation, define datasets that point
to tables or queries in your database, using the linked service for
connection

3. Special Cases: On-Premises Databases

 If connecting to an on-premises database (like on-prem SQL

Server), you must set up a Self-Hosted Integration Runtime (SHIR)
to securely bridge your on-premises environment with ADF in the
cloud

4. Using Azure Key Vault (Optional for Security)

 Store sensitive connection information (like passwords or

connection strings) in Azure Key Vault.
 In the linked service creation, select Azure Key Vault and
reference the secret for your connection string

4) How we will email in Azure data factory?

To send email notifications in Azure Data Factory (ADF), you

typically use one of the following methods:

1. Using Azure Monitor Alerts (Recommended for Most Scenarios)

 Create an Alert Rule:

In the Azure portal, go to your Data Factory resource, navigate to
 the "Monitor" section, and select "Alerts & Metrics." Create a new
alert rule based on pipeline metrics such as failures or successes

Configure Action Group:

During alert rule setup, add an Action Group with the "Email" action.
Enter the recipient email addresses.

Trigger Notification:
When the alert condition (e.g., pipeline failure) is met, Azure Monitor
automatically sends an email to recipients in the Action Group

This method is native, easy to set up, and works for both success and
failure notifications.

2. Using Logic Apps or Azure Functions (For Custom Emails)

 Create a Logic App:

Build a Logic App that sends an email using Outlook, SendGrid, or
SMTP.
 Trigger from ADF:
In your pipeline, add a "Web Activity" that calls the HTTP endpoint
of your Logic App. Pass dynamic content (like pipeline status) if
needed

 Flexible Content:
This method allows you to customize email content, recipients,
and formatting.

Summary Table

Method Use Case Setup Steps

Create alert rule → Add
Azure Monitor Standard pipeline
action group with email →
Alerts success/failure alerts
Receive email on alert
Logic Custom email Build Logic App → Trigger
 Method Use Case Setup Steps
Apps/Azure via Web Activity in pipeline
notifications/content
Functions → Receive custom email

5) How can we load the data from on_premise to azure?

To load data from on-premises systems to Azure, you typically use

Azure Data Factory (ADF), which provides a secure and scalable way to
move data from on-premises databases or file systems to cloud storage
or databases. Here are the main steps and methods involved:

Steps to Load Data from On-Premises to Azure

1. Set Up a Self-Hosted Integration Runtime (SHIR)

The SHIR acts as a secure bridge between your on-premises

environment and Azure Data Factory.

Install the SHIR software on a machine in your on-premises network.

Register the SHIR with your Azure Data Factory instance using an
authentication key

This allows ADF to securely access your on-premises data sources.

2. Create Linked Services

In ADF, define Linked Services for both your on-premises data source
(e.g., SQL Server, Oracle, file system) and your Azure destination (e.g.,
Azure Blob Storage, Azure Data Lake, Azure SQL Database).
The linked service for on-premises sources must use the SHIR you set
up

3. Define Datasets

Create datasets in ADF that represent the data structures (tables, files,
etc.) you want to move.

4. Build and Configure the Pipeline

Use the Copy Data activity in an ADF pipeline to move data from the
on-premises source to the Azure destination

Configure source and sink (destination) datasets and select the SHIR for
the source linked service.

5. Run and Monitor the Pipeline

Trigger the pipeline manually, on a schedule, or via an event.

Monitor execution and troubleshoot any issues using ADF’s monitoring

tools

Summary Table

Step Description
Self-Hosted
Install and register SHIR on-premises to enable
Integration
secure data movement
Runtime
.
Configure connections to both
Linked Services on-premises and Azure data
stores
.
Datasets Define the data structures to be moved.
Pipeline & Copy Create pipeline with Copy Data activity to
Activity orchestrate the transfer

.
Use ADF monitoring tools to track and manage the
Monitoring
migration process

Key Points

SHIR is essential for securely connecting ADF to on-premises resources.

Copy Activity is the core component for moving data.

You can migrate to various Azure destinations, including Blob Storage,

Data Lake, or Azure SQL.

ADF supports both batch and incremental data loads, and can be
scheduled or triggered as needed

6) How to split big file in to smaller ones using mapping data flow?

To split a big file into smaller ones using Azure Data Factory Mapping
Data Flow, follow these steps:

1. Create a Data Flow in Azure Data Factory

Go to your ADF workspace and create a new Mapping Data Flow.

Add a Source transformation and point it to your large file (e.g., CSV,
JSON) in your data lake or blob storage. No need to set schema if you
just want to split files without transformations
2. Add a Sink Transformation

Add a Sink transformation to write the output files.

In the Sink settings, specify the output folder where the smaller files
will be written.

3. Configure Partitioning in the Sink

In the Sink's Settings tab, look for the File name option and set a
pattern (e.g., output[n].csv) so each partition gets a unique file name

Go to the Optimize tab of the Sink.

Set Partitioning to Round Robin (recommended if you don't have a

natural key for partitioning)

Set the Number of partitions to the number of smaller files you want to
create. For example, if you want to split into 10 files, set partition count
to 10

4. (Optional) Dynamically Set Number of Partitions

If you want to split based on desired file size or row count, you can:

Use a Get Metadata activity in your pipeline to get the source file size
or row count.

Calculate the required number of partitions in a pipeline variable.

Pass this variable as a parameter to your data flow, and use it to set the
partition count dynamically

5. Execute the Data Flow from a Pipeline

Save your data flow.

In your pipeline, add an Execute Data Flow activity and select your data
flow.

Run the pipeline. After execution, you will see multiple output files in
your destination folder, each containing a partition of the original data

Key Techniques

Round Robin Partitioning: Evenly distributes rows across the specified

number of output files; useful when you don’t have a natural partition
key

File Name Pattern: Use the [n] pattern in the file name to generate
sequentially numbered output files

Dynamic Partitioning: Use pipeline parameters and metadata to

calculate and set the number of partitions for flexible splitting based on
file size or row count

References:

Microsoft Docs, community Q&A, and tutorials on partitioning and

splitting files with ADF Mapping Data Flows

Summary Table

Step Action
Source Point to large file (CSV/JSON)
Sink Set output folder and file name pattern
Use Round Robin; set number of partitions (static or
Partitioning
dynamic)
Execute Run data flow via pipeline to generate multiple smaller files
This approach is fully supported in Azure Data Factory Mapping Data
Flows and does not require external tools or code.

7) Any kind of situation got struck then how can you handle it?
8) Any mistakes done your project and how you learn from that?

9) What is the main requirement in your project?

The main requirement in an Azure Data Factory project is to

orchestrate and automate the movement and transformation of data
between various data stores, both on-premises and in the cloud

. To achieve this, you must define and configure several key

components:

Pipelines: Logical groupings of activities that perform units of work,

such as ingesting, transforming, and loading data

Activities: The specific actions within a pipeline, like copying data or

running data transformations

Datasets: Definitions of the data structures used as inputs and outputs

for activities, representing data in sources and destinations.

Linked Services: Connection information for external resources

(databases, storage accounts, etc.), enabling ADF to access and move
data.

Integration Runtimes: Compute infrastructure used by ADF to move

and transform data, including options for cloud and self-hosted
environments (the latter is essential for accessing on-premises data).
In summary, the core requirement is to design and configure these
components so that data can be reliably ingested, processed, and
delivered to target systems, fulfilling your business or analytical
[Link] projects involving on-premises data, setting up a self-
hosted integration runtime is a critical step to securely connect local
resources to Azure Data Factory.

Spark & Pyspark Questions

1) How to run from one Notebook to another Notebook?
2) How to connect the Storage account in a data bricks?
3) What is the syntax we use read a csv file using spark identifying
headers?
4) Define dedicated SQL pool definition?
5) Define Stored Procedure? Why we used Stored Procedures?
6) Which category come under Azure data bricks?
7) Can we use multiple languages in one data bricks notebook?
8) How many clusters we have in Azure Data bricks and what are the
functions?
9) To unmounted mount point what command will use?
10) If we need to know the schema of our data frame what is
the command?
11) How to define a schema in a data frame?
12) How can be rename a column in pyspark?
13) To check the previous versions of the data which spark
command will use?
2nd Interview
14) Why Pyspark used in Databricks?
15) What is Cluster? What is different Clusters?
16) What is client mode and cluster mode used in scenarios?
When you running pyspark app in local what memory and which
memory used? Which memory utilize to running that application?
 17) Difference between RDD’s and data frames and datasets?
18) Write a program to suppose we reading a csv file and we
need to check wither particular given keyword is existing in the
text file or not? That keyword I want to pass as argument in the
spark command and same thing from argument you need to read
in the code and check it?
A) Databricks PySpark Program: Check if a Keyword Exists in a CSV File

Below is a Databricks-ready PySpark program that:

 Reads a CSV file into a DataFrame.

 Accepts a keyword as a parameter (argument).
 Checks if the keyword exists anywhere in the CSV (in any column,
any row).
 Prints whether the keyword was found.

You can use Databricks notebook widgets ([Link]) to pass

arguments interactively or via job parameters.

python
# Databricks notebook or job script

from [Link] import SparkSession

from [Link] import col
import sys

# Define widgets for input parameters (Databricks standard)

[Link]("csv_path", "")
[Link]("keyword", "")

# Read widget values

csv_path = [Link]("csv_path")
keyword = [Link]("keyword")
# Read the CSV file into a DataFrame (with header)
df = [Link]("header", True).csv(csv_path)

# Combine all string columns into a single column for searching

from [Link] import concat_ws

string_cols = [f for f, t in [Link] if t == "string"]

df_combined = [Link]("all_text", concat_ws(" ", *string_cols))

# Check if any row contains the keyword (case-sensitive; use lower() for
insensitive)
found =
df_combined.filter(col("all_text").contains(keyword)).limit(1).count() >
0

if found:
print("Keyword found!")
else:
print("Keyword not found.")

How to Use in Databricks

1. Add the code to a notebook cell.

2. Create widgets at the top of the notebook:
o csv_path: Path to your CSV file (e.g.,
/dbfs/mnt/data/[Link])
o keyword: The keyword to search for.
3. Run the notebook, setting the widget values.
4. The output will print whether the keyword was found.
 19) How many actions perform sofa?
20) Functions, Pre-defined functions, user defined functions,
Betterment of modeling on pyspark
21) Sql query to find joins
22) Sql. Query to find ind hyd and pak lahore without using
where clause
23) Sql query to find top 5 customers with highest purchase
amount last 3 months
24) What are index types
25) What is diff between datalake and Delta table
26) Write a code to perform scd2 type
27) What is out of memory exception in spark and when do you
get it
28) what is narrow and wide transformation
29) what is csv and parquet file difference
30) what do you access azure datalake in databricks
31) how do you get 5th highest salary without using window
functions
32) what is a view, how will u create it
PYTHON CODING
1)Python program to check Prime Number?
def prime(n):
Flag=1
If n==0 or n==1:
print ('not prime')
elif n>1:
for i in range(2,n):
if n%i==0:
flag=0
 break
if flag==0:
print('not prime')
else:
print('prime')

prime(4)

Not Prime
2) Python program to find the Factorial of a Number? 5*4*3*2*1
def fact(n):
if n==0 or n==1:
return 1
else:
print(n)
return n* fact(n-1)
fact(5)
3)Python program for nth Fibonacci number? 0,1,1,2,3,5,8,13
Def feb(n):
If n==1:
Return 0
Elif n==2:
Return1
Else:
Return fib(n-1)+fib(n-2)
Feb(8)

4)Python program for Sum of squares of first n natural numbers?

12+22+32=1+4+9=14
Def natural(n):
Sum=0
For i in range(1,n+1):
Sum+=i*i
Return sum

Natural(3)
5)Armstrong number is a number that is equal to the sum of its digits,
each raised to the power of the number of digits in the number. For
example, 153 is an Armstrong number because 1^3 + 5^3 + 3^3 = 153.
def arm(n):
sum=0
temp=n
length=len(str(n))

while temp>0:
digit=temp%10
sum+=digit**length
temp=temp//10
if n==sum:
print('armstrong number')
 else:
print('not')
arm(5)

6) Python program to find Sum of Array

7) Python program to find Largest Element in an Array
8) Python program to Array Rotation
9) Python program to interchange first and last elements in list
10) Python program to Swap Two Elements in a List
11) Python program to check if a String is Palindrome or Not
12) Python program to Reverse Words in a Given String in Python
13) Python program to How to Remove Letters from a String in Python
14) Python program to Words Frequency in String
15) Python program to Remove All Duplicates from Given String in
Python
16) Python program to find the Sum of all items in a Dictionary
17) Python program dictionary duplicate values append
18) D/B Deep copy and shallow copy
19) D/B List and Tuple

PYSPARK CODING
Q) What is Spark?
A) It is General Purpose in Memory Computation Engine.
So generally Spark is used for the computation for processing for
large amount of data set like big data and big data solution for the
analytical purpose use Spark.
1) General Purpose means Data cleaning, Query and machine
learning.
In Spark each and every operation we can perform or do in spark
itself.
2) Computation means Amazon S3, HDFS Storage any kind of storage
used in SPARK.
3) In memory – it is nothing but a RAM. SPARK does all computation
in IN Memory.
SPARK is faster than others.
Q) Spark Architecture.
A) It’s a Master-Slave kind of Architecture of distributed framework.
We try to process whole data into multiple machines. It works on the
concepts of clusters.
A clusters is a group of machines that works together to process and
analyze data.
Spark distributes the data and computations across multiple nodes in
the clusters, allowing for parallel processing and faster data processing.
One of the nodes acts like as master and rest act as worker nodes.
Q) What is RDD?
A) RDD stands for Resilient Distributed Dataset.
 Dataset is nothing but the data that we provided.
The distributed means the input data is stored across all worker
nodes.
Resilient means fault tolerance (Handling failures efficiently).
Eg.
List=[1,2,3,4…..10000000]
RDD is distributed collection of dataset in memory
Rdd1= load file from hdfs
RDD2=[Link]
[Link]()
There are 2 types of operation happens in rdd
1. Transformation
2. Action
Q) DAG and Lazy Evaluation
A) DAG – Direct acyclic graph
Will be have any kind of transformation the execution will not be
happen but its try to create one kind of graph. In that graph only
execution plan will be mention order by its.
Lazy Evaluation – It’s nothing but until unless you will call action
execution will not happened. That’s why spark called it as a lazy
evaluation.
Practice Lab
1) Create Notebook, Cluster and create table?
2) How to create dataframe, dataframe using csv file, read
dataframe, see schema of dataframe?
3) How to read json file, multiline json file, create dataframe using
json file?
4) Select Function
df =
[Link]("/FileStore/tables/emp_1-
[Link]",header=True,inferSchema=True)

[Link]()

[Link]("emp_id","salary").show()
[Link](df.emp_id,[Link]).show()
[Link]([Link][1:5]).show()
5) With Column
df =
[Link]("/FileStore/tables/emp_1-
[Link]",header=True)
[Link]()

[Link]("salary1",[Link]*10).show()

from [Link] import lit//add

new column based on constantvalue
[Link]("country",lit("usa")).show()

[Link]("increment",lit(0)).show()
 [Link]()
from [Link] import col

df1=[Link]("salary",col("salary").cast("I
nteger"))

[Link]()
[Link]()

[Link]("loc","location").show() //
column rename

6) Filter in Pyspark
df =
[Link]("/FileStore/tables/emp_1-
[Link]",header=True)
[Link]()
[Link]([Link]=='india').show()
[Link]([Link]!='india').show()

[Link](([Link]=="manish")&([Link]=="10000
")).show()
[Link](([Link]=="manish")|
([Link]=="10000")).show()
[Link]([Link]("r")).show()
 [Link]([Link]("l")).show()
[Link]([Link]("%ul%")).show()

7) Distinct and drop duplicate

df =
[Link]("/FileStore/tables/emp_1-
[Link]",header=True,inferSchema=True)
[Link]()
df1=[Link]()
[Link]()
df2=[Link](['emp_id','salary'])
[Link]()

8) Sort and order by

df=[Link]("/FileStore/tables/emp_1-
[Link]",header=True,inferSchema=True)
[Link]("salary").show()
[Link]("salary").show()
[Link]([Link]()).show()
9) Group by

df=[Link]("/FileStore/tables/emp_1-
[Link]",header=True,inferSchema=True)
[Link]("address").sum("salary").show()
[Link]("address").count().show()
[Link]("address","emp_id").sum("salary").sho
w()
[Link]("address").max("salary").show()
[Link]("address").min("salary").show()

10) Joining

emp_df=[Link]("/FileStore/tables/employe
[Link]",header=True,inferSchema=True)
emp_df.show()

dept2_df=[Link]("/FileStore/tables/
department_2.csv",header=True,inferSchema=True)
dept2_df.show()

emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'inner').show() //inner Join

emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'left').show() //left Join

emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'right').show() //right Join

emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'fullouter').show() //fullouter

emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'leftsemi').show() //leftsemi
emp_df.join(dept2_df,emp_df.emp_id==dept2_df.use
r,'leftanti').show() //leftanti

11) Union and union all

import pyspark
from [Link] import SparkSession

data1 =
[("James","Sales","NY",90000,34,10000), \
("Michael","Sales","NY",86000,56,20000), \
("Robert","Sales","CA",81000,30,23000), \
("Maria","Finance","CA",90000,24,23000) \
]

columns=
["employee_name","department","state","salary","
age","bonus"]

df =
[Link](data=data1,schema=columns)
[Link]()

data2 =
[("James","Sales","NY",90000,34,10000), \
("Maria","Finance","CA",90000,24,23000), \
("Jen","Finance","NY",79000,53,15000), \
("Jeff","Marketing","CA",80000,25,18000), \
("Kumar","Marketing","NY",91000,50,21000) \
 ]
columns2=
["employee_name","department","state","salary","
age","bonus"]

df2 =
[Link](data=data2,schema=columns2
)
[Link]()

[Link]()
[Link]()

[Link](df2).show()

[Link](df2).show()

[Link](df2).distinct().show()

12) Fillna

df=[Link]("/FileStore/tables/
[Link]",header=True)
[Link]()

[Link]("").show()
[Link]("unknown").show()
[Link]("").show()
[Link]("",["city"]).show()
[Link]("",["city"]).[Link]("other",
["population"]).show()

13) Collect
14) Struct type
from [Link] import
StructField,StructType,IntegerType,StringType
data = [(1,'manish','usa'),
(2,'mani','usa'),(1,'nish','usa')]

schema =
StructType([StructField(name='id',dataType=Integ
erType()),

StructField(name='name',dataType=StringType()),

StructField(name='location',dataType=StringType(
))] )

df=[Link](data,schema)
[Link]()
[Link]()

15) Pivot and unpivot

df = [Link](data,columns)
[Link]()

[Link]("Product").pivot("Country").sum("Amou
nt").show()
df1=[Link]("Product").pivot("Country").sum("
Amount")
[Link]()
from [Link] import expr

[Link]("Product",expr("stack(2,'Canada',Cana
da,'China',China)as
(country,total)")).show()

16) Udf in pyspark

data = [("Finance",10), \
("Marketing",20), \
("Sales",30), \
("IT",40) \
]
Columns = ["dept_name","dept_id"]

df =[Link](data,Columns)
[Link]()

def addon(a):
return a+1
addon(10)

from [Link] import LongType

def addon(a):
return a+1
addon_df=udf(addon,LongType())
[Link]("dept_name","dept_id",addon_df("dept_i
d")).show()

17) Dataframe transformation

data=[(1,'manish',10000),(2,'rani',50000),
(3,'sunny',5000)]
columns=['id','name','salary']

df =[Link](data,columns)
[Link]()

from [Link] import upper

def uppername(df):
return
[Link]("name",upper(df,name))

from [Link] import upper

18) Create or replace temp view()

data=[(1,'manish',10000),(2,'rani',50000),
(3,'sunny',5000)]
columns=['id','name','salary']

df = [Link](data,columns)
[Link]()
 [Link]("employee")

%sql
select *from employee

%sql
select max(salary) from employee

csv –dbfs
df = [Link](“location
path”,header=true,inferSchema=True)

19) Windows function using pyspark

data=[(1,'manish','india',10000),
(2,'rani','india',50000),(3,'sunny','UK',5000),
(4 ,'sohan','UK',25000),
(5,'mona','india',10000)]

columns=['id','name','country','salary']

df =[Link](data,columns)
[Link]()

from [Link] import Window

 from [Link] import
row_number

window =
[Link]("country").orderBy("salary")

[Link]("rn",row_number().over(window)).sh
ow()

from [Link] import Window

from [Link] import col

window =
[Link]("country").orderBy(col("salar
y").desc())

[Link]("rn",row_number().over(window)).sh
ow()

from [Link] import Window

from [Link] import rank

window =
[Link]("country").orderBy("salary")

[Link]("rn",rank().over(window)).show()

from [Link] import Window

from [Link] import
dense_rank
 window =
[Link]("country").orderBy("salary")

[Link]("rn",dense_rank().over(window)).sh
ow()

20) Date format function

data = [("2022-03-15", "2022-03-16
12:34:56.789"),
("2022-03-01", "2022-03-16
01:23:45.678")]
df = [Link](data,
["date_col", "timestamp_col"])
[Link]()

from [Link] import *

[Link]("date_col",date_format("date_col","yyy
y/MM/dd").alias("date")).show()

[Link]("date_col",date_format("date_col",'yyy
y-MMMM-dd')).show()

[Link]("date_col",date_format("date_col",'dd-
MMMM-yyyy')).show()
21) Partition by
22) Explode function
23) Cache and persist
Interview Questions
1) Flatten the data
2) Find out the first not null values
3) Find out the total null values count in each column
4) Replace null values with the mean salary of all employees
5) Replace null values with the column with default value as
department
6) Find customers who have placed orders on consecutive days
7) Find out the employee salary greater then manager salary
8) Find out the users who log in and log out multiple times during a day.
a) What was there last login time?
b) What was there last logout time?
c) What was there first login time?
d) What was there total login count?
e) What was there last login duration?
f) Total duration per user?
Azure Databricks Interview Questions:

1. What is Azure Databricks?

Azure Databricks is a robust platform for large data analytics built on

Apache Spark. It is simple to use and one can quickly install it on the
Azure server. Data engineers who wish to work with big data hosted in
the cloud often use Databricks because of its excellent integration with
the other Azure services
2. What is the use of auto-scaling in Azure Databricks?

The auto-scaling functionality of Databricks enables users to

automatically scale the cluster up or down with their demands.
Ensuring users are only using the resources they require helps save
time and money.

3. What are the major benefits of Azure Databricks?

Though Azure Databricks is based on Spark, it supports many other

programming languages such as Python, R and SQL. To integrate these
with Spark, Databricks converted these languages on the backend
through application performance indicators (APIs). This eliminates the
users' requirement to learn any additional programming language for
distributed analytics. Azure Databricks is highly adaptable and simple to
implement, making distributed analytics much easier to use.

4. What are the different kinds of clusters in Azure Databricks and what
are the functions of each?

'Azure Databricks has four cluster types. Interactive, job, low-priority

and high-priority. Interactive clusters help with data exploration and ad
hoc queries. These clusters provide low latency and high concurrency.
We utilise job clusters for batch job execution. We can automatically
scale job clusters to match the requirements. Low-priority clusters are
less expensive than other cluster types but offer low performance.
These clusters are suitable for tasks, such as development and testing,
that may require lesser performance. High-priority clusters are more
costly than other clusters, but they provide the best performance.
These clusters are suitable for production-level workloads.'

5. What is the use of Kafka in Azure Databricks?

Azure Databricks uses Kafka for streaming data. It can help collect data
from many sources, such as sensors, logs and financial transactions.
Kafka is also capable of real-time processing and analysis of streaming
data.

6. How do you manage the Databricks code while working with a team
using the team foundation server (TFS) or Git?

Azure Databricks allows easy notebook integration with Git, Bitbucket

cloud and TFS. The integrating process differs slightly depending on the
service we integrate. After the integration, Databricks code functions
similar to a project clone. To effectively manage the Databricks code, I
start by creating a notebook, committing it to the version control
system and then updating it.

7. Can you run Databricks on private cloud infrastructure?

 'Amazon Web Services (AWS) and Azure are the only options available
now. Databricks utilises open-source Spark. We can develop our own
cluster and run it in a private cloud, but in that case, we miss out on
Databricks' full administration capabilities and features.'

8. What do you understand by mapping data flows?

'Microsoft offers mapping data flows which do not require coding for a
simpler data integration experience, as opposed to data factory
pipelines. It is a graphical method for designing data transformation
pipelines. It helps transform the data flow into Azure data factory (ADF)
activities and execute as part of ADF pipelines.'

9. Reading Data:

To read data from different sources in Databricks using PySpark, you

can use [Link]() with appropriate options for the specific
file format (e.g., csv, parquet, json).
Delta Lake provides ACID transactions, scalable metadata handling, and
data versioning, making it advantageous for reading and writing data in
Databricks.
Data reading performance can be optimized in Databricks by
partitioning data, using appropriate file formats, caching data, and
leveraging cluster resources effectively.
df = [Link]("dbfs:/path/to/[Link]", header=True,
inferSchema=True)

10. Writing Data:

Data can be written to different formats in Databricks using PySpark by

using [Link]() with the desired format (e.g., parquet, delta)
and appropriate options.
Optimizing data writing performance in Databricks involves
considerations such as batch size optimization, tuning parallelism, and
leveraging Delta Lake features like schema evolution and data
versioning.
Schema evolution can be handled in Databricks by using features like
schema merging in Delta Lake, while data versioning can be managed
by leveraging Delta Lake's time travel capabilities.

[Link]("dbfs:/path/to/[Link]")

11. Calling Notebooks:

You can call another Databricks notebook from a notebook using the
%run command followed by the notebook path and parameters if
needed.
Calling notebooks in Databricks is useful for modularizing code,
promoting code reusability, and enhancing collaboration among team
members.
Best practices for organizing notebooks in Databricks include using
version control, creating libraries for shared functions, and
documenting notebooks effectively for clarity and reproducibility.

%run "/path/to/another/notebook" param1=value1 param2=value2

12. Cluster Management:

Auto-scaling in Databricks clusters dynamically adjusts the number of

worker nodes based on workload demands, optimizing resource
utilization and cost efficiency.
Databricks offers different types of clusters such as Standard, High
Concurrency, GPU, Machine Learning, and Job clusters to cater to
various workload requirements.
Monitoring and optimizing cluster performance in Databricks involves
analyzing cluster metrics, tuning cluster configurations, leveraging
instance pools for faster startup times, and optimizing resource
allocation based on workload characteristics.
13. Default location of file saving in Databricks
14. Defaut file types of Any files if we are saving in databricks

NEW UI OF DATABRICKS Unity Catalog

1) create database [Link]
2) create volume [Link]
3)
df=[Link]("/Volumes/development/data/fil
es/order/[Link]")
4) [Link]()
5) display(df)
6) [Link]()
7)
df=[Link]("csv").options(header=True,
inferSchema=True).load("/Volumes/development/
data/files/order/[Link]")
display(df)
[Link]()

8)
df=[Link]("/Volumes/development/data/fi
les/order/[Link]",multiLine=True)
display(df)
9) df =
[Link]("json").option("multiline",Tru
e).load("/Volumes/development/data/files/order/
[Link]")
display(df)
10) Select Transformation
11) Filter Transformation
12) With column and without column
13) Drop Duplicates () VS Distinct ()
14) Group by Transformation
15) Joins
16) Union and Union all

Interview Preparation Q&A

1) Running total sales problem - SQL

A) select * , sum(amount) over( order by sale_date) running_total from
sales;
2) 3rd highest salary problem – SQL
A) with cte as(
select *, row_number() over (order by amount desc) rn from sales
)
Select * from cte where rn=3
3) Find Duplicate elements in a LIST – WITHOUT Inbuilt Functions –
Python
A) nums = [1,2,3,2,4,5,1,6,1]
d= {}
a = []
for i in nums:
if i in d:
d[i]+=1
else:
d[i]=1
for key , value in [Link]():
if value>1:
[Link] (key)
a

4) Reverse List – Python

A) num = [10, 20, 30, 40, 50]
a = []
for i in range(
len(num) - 1,
-1,
-1
):
[Link](num[i])
a

5) Sales by product per month - Pyspark

Dataframe:
data = [ ("Jan", "Mobile", 1000), ("Jan", "Laptop", 2000), ("Feb", "Mobile", 1500), ("Feb", "Laptop",
3000), ("Mar", "Mobile", 1200), ]

df = [Link](data, ["month", "product", "sales"]) [Link]()

A) [Link](“month”).pivot(“product”).sum(“sales”).show()

Python coding questions & answers

1) Find Duplicate elements in a LIST – WITHOUT Inbuilt Functions –
Python
A) nums = [1,2,3,2,4,5,1,6,1]
d= {}
a = []
for i in nums:
if i in d:
d[i]+=1
else:
d[i]=1
for key , value in [Link]():
if value>1:
[Link] (key)
a

2) Reverse List – Python

A) num = [10, 20, 30, 40, 50]
a = []
for i in range(
len(num) - 1,
-1,
-1
):
[Link](num[i])
a

3) Remove duplicates from list

A) nums = [1,2,3,2,4,1,5]
l = []
for i in nums:
if i not in l:
[Link](i)
l

4) Find the largest element in a list

A) l = [2,3,101,4,5,6]
max = l[0]
for i in l:
if i>max:
max=i
max

5) Count word frequency in a list

A) text = "data engineer data python sql data
engineer"
l = [Link]()

d ={}
for i in l:
if i in d:
d[i]+=1
else:
d[i]=1
d

6) Find factorial of a number

A) def fact(n):
if n==0:
return 1
else:
return n*fact(n-1)
fact(5)

7) Reverse a string without using built-in reverse

A) a = 'hello'
str =""
for i in a:
str = i+str
str

8) Check if given number prime or not

A) def prime(n):
if n<=1:
return False
for i in range(2,n):
if n%1==0:
return False
return True
prime(5)

9) Find even number from list

A) a = [1,2,3,4,5,6,7,8,9,10]
l =[]
for i in a:
if i%2==0:
 [Link](i)
l

Pyspark Questions & Answers

PYSPARK Coding Question & Answers
7.🔥 Top PySpark Coding Interview Questions for Data Engineers |
Missing Numbers & JSON Parsing

[Link]
v=sMDJYOXTLKw&list=PLOlK8ytA0MgjX4QwSbkwGcyzo7iM_8DSK&inde
x=7
Dataset –
1. data = [(1,), (2,), (3,), (5,), (7,), (8,), (10,)]
df = [Link](data, ["order_id"])
2. data = [ (1, '{"product":"Laptop","price":50000,"brand":"Dell"}'),
(2, '{"product":"Phone","price":30000,"brand":"Samsung"}'), (3,
'{"product":"Tablet","price":20000,"brand":"Apple"}') ]
df = [Link](data, ["id", "json_str"])
1 ) Find missing numbers/orders in given sequences ?
A) list = [(i,) for i in range(1,11)]
Print(list)
df_new = [Link](list,[‘order_id’])
display(df_new)
df_new.subtract(df).show()
2) JSON String Parsing – Extract values from json ?
A) from [Link] import *
[Link]("id",get_json_object(col("json_str"),"$.product").alias("produc
t"),
get_json_object(col("json_str"),"$.price").alias("price"),
get_json_object(col("json_str"),"$.brand").alias("brand")
).show()
3) Sales by product per month - Pyspark
Dataframe:
data = [ ("Jan", "Mobile", 1000), ("Jan", "Laptop", 2000), ("Feb", "Mobile", 1500), ("Feb", "Laptop",
3000), ("Mar", "Mobile", 1200), ]

df = [Link](data, ["month", "product", "sales"]) [Link]()

A) [Link](“month”).pivot(“product”).sum(“sales”).show()

4. Top PySpark Data Engineer Interview Questions With Answers

| Duplicate Scenario

[Link]
v=Bm1hIAmxcSI&list=PLOlK8ytA0MgjX4QwSbkwGcyzo7iM_8DS
K&index=10
GITHUB DATASET –
data = [
(1, "Alice", "TXN1001", 250, "2025-09-20"),
(1, "Alice", "TXN1001", 250, "2025-09-20"), # exact duplicate row
(1, "Alice", "TXN1002", 300, "2025-09-21"), # same customer, diff
txn
(2, "Bob", "TXN1003", 400, "2025-09-20"),
(2, "Bob", "TXN1003", 400, "2025-09-20"), # exact duplicate row
(3, "Cathy", "TXN1004", 150, "2025-09-21"),
(4, "Dan", "TXN1005", 200, "2025-09-22"),
(4, "Dan", "TXN1006", 500, "2025-09-23"), # same customer, diff
txn ]
df = [Link](data,
["customer_id","name","txn_id","amount","txn_date"]) display(df)
[Link] Duplicate Rows?
A. [Link]()
[Link] exact duplicate rows?
A. from [Link] import *
[Link]([Link]).count().filter(col(“count”)>1).show()

6. Get unique customers (by customer_id)

A. [Link]([‘customer_id’]).show()

7. Get unique transactions per customer (customer_id, txn_id)

A. [Link]([“customer_id”,”txt_id”]).show()
8. Find customers with more than 1 transaction
A.
[Link]([“customer_id”,”txt_id”]).groupBy(“customer_id”).c
ount().filter(col(“count”)>1).show()
9. Find duplicate transactions(where txn_id repeats)
[Link](“txt_id”).count().filter(col(“count”)>1).show()

PYTHON Interview Questions & Answers

1) Find second Largest Element from List
A) nums = [10,4,29,40,83,66]
sorted(nums)[-2]
2) Reverse Integer
A) n=1234
Int(str(n)[::-1])
3) Missing Number (1 to n)
A) nums = [1,2,3,4,5,7]
n = len(nums)+1
sum_nat = n*(n+1)//2
sum_nums = sum(nums)
sum_nat-sum_nums
4) Reverse words in sentence
A) S =”I Love Python”
‘ ‘ .join([Link]()[::-1])
5) Anagram check
A) # Two Strings are anagrams if they use the same characters with
same frequency, But possible in a different order.
# “listen” and “silent” True of anagram
# ”Hello” and “World” False
S1 = “listen”
S2 = “silent”
sorted (s1)==sorted(s2)
6) Palindrome check
A) s = ‘racecar’
If s[::-1] ==s:
Print(“Palindrome”)
Else
Print(“not a palindrome”)
7) Combine list
A) names = [“alice”, “bob”,”Charlie”]
Scores = [85,90,95]
List(zip(names,scores))
SQL Coding Questions & Answers
dataset

CREATE TABLE Orders ( order_id INT , customer_id INT NOT NULL, order_date DATE NOT NULL,
amount DECIMAL NOT NULL );

-- Customer 201: one order in every month of 2024

INSERT INTO Orders VALUES ( 1, 201, '2024-01-15', 220.00), ( 2, 201, '2024-02-10', 180.00), ( 3, 201,
'2024-03-12', 210.00), ( 4, 201, '2024-04-18', 200.00), ( 5, 201, '2024-05-05', 190.00), ( 6, 201, '2024-06-
22', 230.00), ( 7, 201, '2024-07-09', 250.00), ( 8, 201, '2024-08-14', 260.00), ( 9, 201, '2024-09-03',
240.00), ( 10, 201, '2024-10-27', 280.00), ( 11, 201, '2024-11-06', 300.00), ( 12, 201, '2024-12-19',
320.00);

-- Same-day multi-customer orders (to practice “above daily average”)

INSERT INTO Orders VALUES ( 13, 202, '2024-03-10', 100.00), ( 14, 203, '2024-03-10', 200.00), ( 15,
204, '2024-03-10', 300.00), ( 16, 202, '2024-07-07', 450.00), ( 17, 203, '2024-07-07', 450.00), ( 18, 204,
'2024-07-07', 150.00),

-- Repeated same amounts per customer (to practice duplicates) ( 19, 205, '2024-01-10', 400.00), ( 20,
205, '2024-02-10', 400.00), ( 21, 205, '2024-03-10', 350.00), ( 22, 205, '2024-04-10', 350.00),

-- More 2024 activity ( 23, 202, '2024-01-05', 500.00), ( 24, 202, '2024-04-22', 500.00), ( 25, 203, '2024-
02-14', 150.00), ( 26, 203, '2024-04-20', 150.00), ( 27, 204, '2024-02-11', 700.00), ( 28, 204, '2024-02-18',
300.00), -- 2025 rows (helpful for “last 90 days” depending on when you run) ( 29, 206, '2025-07-01',
600.00), ( 30, 206, '2025-08-01', 650.00), ( 31, 207, '2025-09-15', 120.00);

1 .Customers who placed orders in the last 30 days?

A. Select current_date(),current_date()- interval 30 days
2. Find customers who placed orders in all months in 2024?
A. Select customer_id from orders where year(order_date)=2024 group
by customer_id having count(distinct month(order_date))=12 O/P 201
For check 201 select * from orders where customer_id=201
3. Find customers who placed the same order amount more than once.
A. Select customer_id,amount,count(*) from orders group by 1,2
having count(*)>1
4. Get orders above the average amount for that same day.
A. Select *from orders o1 where amount>(select avg(amount)
avg_amount from orders o2 where o1.order_date=o2.order_date)

SQL Coding Question & Answers

1) Running total sales problem - SQL
A) select * , sum(amount) over( order by sale_date) running_total from
sales;
2) 3rd highest salary problem – SQL
A) with cte as(
select *, row_number() over (order by amount desc) rn from sales
)
Select * from cte where rn=3

1 .Show Employee with their manager’s details?

A. Select e.emp_id, e.emp_name manager_name from employees e left
join employees m on e.manager_id=m.emp_id
2. Find the 2nd highest distinct salary overall
A. with cte as (
Select * , dense_rank() over(order by salary desc) rn from employees
)
Select * from cte where rn=2
3. Top 3 earners per department average salary
A. with cte as(
Select * , dense_rank() over(partition by dept order by salary desc) rn
from employees
)
Select * from cte where rn between 1 and 3

4. Flag employee above department average salary

A. with cte as(
Select emp_id ,emp_name,salary,
Avg(salary) over(partition by dept) dept_avg,
Case when salary > avg(salary) over (partition by dept) then True else
False end sal_greater from employees
)
Select *from cte where sal_greater = ‘true’

1. Compare employee salary vs their manager (with names)

A. Select e.emp_id,e.emp_name,[Link] emp_salary , m.emp_name
manager_name, [Link] manager_salary from employees e left
join employees m on e.manager_id=m.emp_id
PART-2
Preparation of DATA Engineer
1)Tell me about yourself?
2)What are tools and technologies you used?
3)Project Architecture?
4)What is down stream and what is up stream?
5)Which file format and how you doc data?
6)diff b/w list and tuple?
7)diff b/w list and dictionary?
8)Suppose you are working with Rest API then how you can fetch data
as well load
data in python and pyspark?
9)list=[1,2,2,3,4,4,6] write a python program to remove duplicate
without any function
as well as det?
10)diff b/w array and list?
11)what is fact table?
12)What is dimension table?
13)explain star schema?
14) What is HDFS and what is its architecture?
15)What is partition in pyspark and how do you Optimisation partitions
in pyspark?
16) What is Partitioning vs Bucketing
17) Genkins?
18) Where you build a cdc pipeline and type is SCD type-2 and
implement in AWS
and sources are two 1 is SQL DB and 2 is MONGO [Link] you have to
implement
SCD type-2 and you have to maintain in the dataware house at then.
How you created it? Explain SCD type-2 cdc pipeline? usus

19)designing the pipeline or responsible on the pipeline?Rolls &

Responsibles?
A) Design the pipeline
1) We received semistructured data from [Link] give xml file
aroung 15gb [Link] parts those xml file based on xsd schema.
2) After that we parts those to semi structure to structure CSV file.
3) Then we write data back to S3 and pick those csv files and do some
XN Transformation
Setup and apply loadtodata.

20)What components involved in your pipeline?like s3,glue

transformation AWS components?
A) 1) for data injection (or) data pickup we used spark dataframe to S3.
2) Amazon EMR for script implemented. we get access throw Amazon
ECP.
Project related questions:
--------------------------

21) Project Architecture?

A)
Data Source Ingest Process
Serve
:------------------------- :
CSV : DataBricks :
: :
JSON :
SPARK :---------> PowerBI
:-------------------------:
SQL 2| 3| 4|
Azure 1
|--------------------------------- |
REST API ---> Data Factory ---> | Bronze
Silver Gold |
| | | | | | ||
| |
| Azure Data Lake
Delta Lake |

Storage |
|------------------------------------|

STORE

DataSource:
----------
1) We have many sources data is coming like CSV file we have that is
maintain
by ground team they putting some ware in the clouds.
2)json file we are also putting in the different source.
3)we have diff databases like oracle,mysql those databases are
mainly uses for
transformations data as well as some sql data also is their.
4)RestAPI directly cominig from the API's.

Ingestion:
---------
1)so we have multiple datasources so we have to bring that data
to one place that is
every one to pick the data and they are use that to their
transformation purpose.
2)for the data injection purpose we use azure [Link] will
help like bring
data from multiple source to one cloud platform like adlsjen2.
 Process:
-------
1)Data processing from datafactory to databricks.
2)Databricks normally follows medallion architecture for diff
purposes and diff teams
access the data.
3)The Medallion Architecture is a data design pattern for
Databricks lakehouses
that organizes data into three layers—Bronze, Silver, and Gold—
to progressively
improve data quality. The Bronze layer stores raw, unprocessed
data,
the Silver layer contains cleaned and validated data, and the Gold
layer holds
business-ready.

Serve:
-----
1)Finally we have Power-BI for build in dashboards.

22) Spark Architecture?

A) 1) If you want process large amount of dataset then we can use
Spark.
 2)Spark generally use for computation purpose.
3)Spark follows Master Slave Architecture
1 2 3
Master Slave
4)Master node(Driver Manager) --> Cluster Manager ---->Worker
node()
---->Worker node()

---->Worker node()
5)you have a 1Master Node and multiple Worker Node.
6)Driver Manager(node) convert’s code into multiple jobs. This jobs
send to a worker node.
7)Cluster manager will have the information like one which particular
node the resources are available and where the process can be
happened that information gives to the driver manager. This driver
manager sends all the jobs to the worker node. In a worker node with
the help of the executer the multiple tasks got created its actually try to
process your data and send back result to the master driver manager.
8)In Worker Node (3) job execution and process will happened.

23)What is RDD?
A) RDD stands for Resilient distributed dataset.
1)dataset is nothing but the data that we provided.
2)It means the input data is stored across all worker nodes.
3)Resilient means which is handling failure's very efficently.
 There are 2 types of actions happend in RDD
1)Transformation(TR)(MAP,FLATMAP)-When will perform Tr then
execution not happend.
2)action (MAX,COUNT)- When will perform action then only execution
will happend.

Process of RDD:-
1)First it will try to load file from HDFS. RDD1 = load file from
hdfs(1TR)
2)It's try to perform a filter operation. RDD2 = [Link] (1TR)
3)Finally it will show me the result. [Link]() (action)

When ever try to write any TR then its try to create a graph for you.

DAG and Lazy Evaluation:-

1)DAG means Direct Acyclic Graph.
2)In this graph it will try to create 1 by 1 one kind of graph through
that graph
only we will able to know how the execition will happen and this
graph is nothing
but a [Link] is the one graph through know how the execution will
happen.

Lazy Evaluation:-
 It is nothing but when you call action execution will not happen so
that is called LE.

24) Where we are fetching data in data engineering?

A) data is fetched from a variety of sources depending on the use case
and
[Link] data already their in the cloud like adlsjen2.
AWS S3 bucket.3rd party also.
Sql directly connect with data factory.

25) What was your role in this project?

A) Inetially I worked on injection.
my role is injection data from database to ADLS gen2.

26) What processing doing like batch processing or realtime

processing?
A) Batch processing we using. as per business requriments daily or
weekly or
monthly we are processing.

27) What are your team size and how can you dealing with
stackholders?
A) we are 5 members of team and we have multiple stackholders
like sales and retails we deal with directly onsite clint.
28) How you get the requriments ? who gave the requriments? like
monthly grouping?
A) businees team and team lead meeting with stackholders.
So they devided into sprints like 3 to 4 sprints.
every sprint for the 2 weeks.
every sprint devided into diff tasks or stories.
so we get jira stories for that particular tasks.

29) How partition tasks first injection or processing or transformation?

A) Based on requriments of sales teams diff diff purposes.

30) what are agail ceramonies you are doing?

A) Agile ceremonies are the regular meetings/events that structure
work in Agile
(especially Scrum) so the team stays aligned, delivers value each sprint,
and continuously improves.
In most teams, “Agile ceremonies” usually refers to four core
meetings.
1. Sprint Planning
2. Daily Standup (Daily Scrum)
3. Sprint Review
4. Sprint Retrospective
31)from transformation to till powerBI reports or Deltalake tables are
you faced any sudden spike in the data?any pipeline running longtime
or any memory leakage?
A) yes, sudden spike happens in starting of the month or sometimes
starting of the year. unexpected increase in data volume or data points
at a certain stage of the pipeline. because of add many customers. That
time size of data will increase. when more customers and more data
size increased then we setup configuration
like auto scalability it increases clusters size. cluster size settings and
auto scaling configuration

32) Have you try to optimizing the code to reduce the identification
time?
A) not that much idea i'm not that side to change the code.

33) What are the file formats you worked?current project what you
used?
A) generaly used row based file formats used in projects.
mostly i used parket and deltalake also use Json and csv files.

34) are you face any changes in the schema? for exe your table have
receiving 4 columns so how without breaking your pipeline?
A) We chose delta [Link] we enable delta format then schema
will change.
merge schema.
35)How to check data accuracy or consistency through out in your
pipeline?
A) We check the source data count with sink data count(Target).
if we check any missing data by using count the source with final data.

36)through out your pipeline how you test?using any testing tool or
unit testing?
A) Inetially we do sample test for [Link] pytest
framework using
for unit testing.

37) In databricks coding which one is pyspark or sql?

A) Both we are using pyspark and sql.

38)Are you faced any pipeline failures in production? any issues you
faced?
what are the issues?
A)when production side most of the performence issue,some times
memoryoutoff the
error.

39)what are default Shuffel partitions?

A)shuffle partitions refer to the way data is redistributed across
different nodes (partitions) during shuffle operations.
shuffle partitions in Azure data engineering dictate how the data is
split and
moved around across distributed tasks during shuffle, helping manage
workload
distribution and performance in Spark-based big data processing.

40)How are you monitering the pipeline?

A)We have custome logs to store logs in cloud and mostly we have
edge monitering tool we are using.

41)Is their any framework is their to keep the logs?

A) We directly check in the folders.

42)which versioning tool or VCS(versin control system) you are used?

What were the deployment process? deployment code to main branch?
A) we use GIT hub version controller only. we have diff branches like
dev,test and production. first we develop code in dev once that code
approval from manager then that code goes to next level.

43) what was the frequently pipeline running?

A)we have few pipelines, some pipeline daily ,some other pipeline
weekly,
quarterly pipelines also we [Link] on business.

44)Have you done any optimization? either datafactory or databricks?

45)Have you use bucketing some were?

: Round 1 & Round 2 – Technical :

---------------------------------

A. Introduction and Experience

46)Briefly introduce yourself and walk us through your journey as a
Data Engineer so far.
47)Current project details: technologies, data architecture, and
responsibilities.
48)Average data volume handled and strategies for efficient
processing.
49)Challenges faced in your projects and how you overcame them.

B. Azure Data Factory (ADF)

50)What is ADF? Is it an ETL or ELT tool, and why?
 A) Azure Data Factory (ADF) is Microsoft's cloud-based data
integration service for
creating, scheduling, and orchestrating data pipelines.

51)Explain a linked service and how to create one?

A)A linked service in Azure Data Factory (ADF) is the definition of a
connection to
an external resource (like Azure SQL Database, Blob Storage, a REST
API, etc.),

52)Difference between linked service and dataset in ADF?

A)A linked service tells ADF how and where to connect (the
connection), while a dataset
tells ADF what data inside that connection to work with (the data
structure and
location). They always work together: every dataset is built on top of a
linked service.

53)Integration Runtimes (IR): Types and use cases?

A)Azure Data Factory (ADF) uses Integration Runtimes (IR) as the
compute infrastructure
to execute data integration activities like data movement,
transformation,
and orchestration across cloud, on-premises, or hybrid environments.
 54)Triggers in ADF, especially tumbling window triggers.
A)In Azure Data Factory (ADF), triggers are objects that define when a
pipeline should
run, allowing automated and repeatable execution instead of manual
runs. They can pass
parameters into pipelines, support dependencies, and help build
robust, time- or
event-driven data workflows.

55)Moving pipelines from development to production: ARM templates

for deployment.
A)Azure Data Factory (ADF) pipelines move from development to
production through
Continuous Integration and Continuous Deployment (CI/CD) using Git
integration
and ARM templates, ensuring consistency across environments like
dev, test/UAT,
and prod.

[Link]
56)Write a query to find the second-highest salary in a table.

[Link]
57)Function to find the top 3 largest numbers in a list.
[Link] Validation and Schema Management
58)Handling data validation using SQL or Python.
59)Managing schema changes in PySpark over time.
A) Suppose we have one storage a

F. PySpark and Spark

60)Why is RDD considered resilient and fault-tolerant?

61)Lazy evaluation in Spark and its impact on performance.

62)Difference between persist() and cache() in Spark.

63)Difference between reduceByKey() and groupByKey().

64)DataFrames vs. RDDs in PySpark.

65)What are the key differences between DataFrames and RDDs in

PySpark?

66)How do you manage schema changes in PySpark when processing

data over time?

G. Azure Databricks
67)What is a mount point in Azure Databricks? How to mount ADLS
Gen2 to Databricks?
 A)copy data from adlsgen2 storage account to access in databricks
then we create
mount point in databricks.

:Managerial:
------------
This round assessed my ability to work within a team, take ownership
of projects, and
collaborate effectively with stakeholders.

H. Professional Background
68) Can you introduce yourself and provide a brief overview of your
career?
69) Team size and roles of other members in your project.

I. Project Architecture
70)Describe the architecture of your current project and your role in
its design and
implementation.

J. Data Warehouse Design

71)Star Schema vs. Snowflake Schema: Which one you implemented
and why?
 72)SQL Analytical Functions
73)Differences between ROW_NUMBER(), RANK(), and
DENSE_RANK() and their usage in project
scenarios.

K. Stakeholder Collaboration
74)Strategies for smooth communication between data scientists,
business teams, and
developers.

Databricks and Delta Lake

-------------------------
75) dbutils Function?
A) Databricks Utilities, accessed via dbutils, provide built-in functions
for
interacting with the Databricks environment in notebooks.
Main Modules
These utilities cover file system operations, job management,
notebooks, secrets,
and widgets.
•[Link]: Handles DBFS tasks like listing (ls), copying (cp), removing
(rm),
 and creating directories (mkdirs).
•[Link]: Manages notebook flow, including running other
notebooks (run) and
exiting with values (exit).
•[Link]: Supports job features like setting task values
([Link]) and
running jobs

76) Moving Files in DBFS?

A) Use [Link]() or the %fs mv magic command to move files or
directories
within Databricks File System (DBFS). These utilities handle renaming
and relocation
efficiently, with support for recursion on directories.

Basic File Move

Move a single file using full DBFS paths starting with dbfs:/.
text
[Link]("dbfs:/source/path/[Link]",
"dbfs:/destination/path/[Link]")

77) Job Cluster in Databricks?

A)Job clusters in Databricks are compute resources created
automatically by the job
scheduler for running automated, non-interactive workloads like ETL
pipelines or
batch jobs. They start when a job begins, execute the tasks, and
terminate automatically
upon completion to optimize costs and resource usage. Unlike all-
purpose clusters,
job clusters cannot be restarted or shared for interactive work.

78) Lazy Evaluation in Spark?

A) It is nothing but when you call action execution will not happen
so that is called LE.

79) Managed vs External Tables?

A)Managed tables in Databricks (under Unity Catalog) store both data
and metadata
in Databricks-controlled storage, with full lifecycle management
including automatic
data deletion on drop. External tables point to user-specified cloud
storage locations
(e.g., S3/ADLS), managing only metadata while leaving data intact on
drop.

80) Delta Lakehouse Architecture?

A)Delta Lakehouse architecture combines a cloud data lake with data
warehouse features
using Delta Lake as the storage layer plus governance, compute, and BI
on top.
It is usually organized in a medallion (bronze–silver–gold) pattern to
progressively
refine data for analytics and AI.

81) Bronze/Silver/Gold Layers?

A) Medallion (Bronze/Silver/Gold)
Layer Purpose Typical Data
Bronze Raw ingestion Uncleaned batch/stream data from source
systems
Silver Cleaned & conformed Joins, filters, quality checks,
standardized schemas
Gold Business-ready Aggregations, KPIs, dimensional models for
BI/ML

Data typically flows from raw event or batch sources into bronze
tables, then through
ETL/ELT to silver and gold tables using streaming and batch
pipelines on
the same Delta storage.
82) Deployment Process?
A) Databricks deployment follows CI/CD pipelines using Databricks
Asset Bundles (DABs)
to version code, notebooks, jobs, and infrastructure as YAML files in Git
repositories.
This automates building, testing, and deploying to isolated
dev/staging/prod workspaces
via tools like GitHub Actions or Azure DevOps.

83) Scheduling Jobs in Databricks?

A) Job scheduling is when you have a data. So you will have a
Notebook and write a code on it and that code will be schedule daily
basis or weekly or monthly basis. This is how we will try to schedule.
Particular time and particular Notebook will be running and it will be
load in particular table. That particular table will create dump or will
create reporting table.
Databricks schedules jobs through the Jobs UI using simple intervals or
cron expressions for automated execution on job clusters.
Navigate to Workflows > Jobs, select a job, and add a Scheduled trigger
to define timing in UTC.
Cloud and Spark-Related Questions
---------------------------------
PART-3
Q) Self-introduction including current role, projects, and key
responsibilities?
Describe your current project, including technologies, architecture, and
responsibilities.
A)
1) Hi, This is Rajesh. I Completed my masters MCA VIT
Vishakhapatnam. Totally I have 8+ years of exp in IT. coming to this
data engineer position I have 4+ years of exe I there. Previously I
worked with NS Data Systems, Tech M, Cog, CapG and Regnant.
2) Coming to my previous project it’s a healthcare domain.
And I used Various technologies in this project like Azure Data Factory,
Azure Databricks, Apachi Spark and I Have a good knowledge in pyspark
, sql and python . For code deployment purpose we use GIT hub version
controller only. we have diff branches like dev, test and production.
first we develop code in dev once that code approval from manager
then that code goes to next level.
We are in the retail side. Lots of data will be generated in the sales side
like daily sales data, products data.
so our business team needs to track like sales performance how
monthly sales going on and product price predictions and Sometimes
customer’s attentions like sometimes we losing customers then why we
are losing customers. SO for that we build pipelines.

3) Coming to my daily BIU(business intelligence unit) is use to take care

of various pipelines and we are getting data from different data
sources like Amazon S3, CSV and Json files and RestAPI directly coming
from the API's after that doing some kind of transformation processing
over that and it’s going to that specific Target. So my Target also S3
bucket. Yes, this is my daily BIU Task.

Project Architecture
5)Data Source:
1)Many sources of data is coming like CSV file. SO our ground team
maintain that data and they putting some ware in the clouds.
2) we are also putting Json file in different source.
3) we have diff databases like oracle,mysql. Those databases are mainly
used for Transformations the data as well as some sql data also is there.
4)RestAPI directly coming from the API's.
6)Ingestion:
1)so we have multiple data sources and bring that data to one place
that is every one to pick the data and they are used to their
Transformation purpose.
2)For the data injection purpose we use azure datafactory. This will
help like bring data from multiple sources to one cloud platform like
adlsjen2.

7)Process:
1)Data processing from datafactory to databricks.
2)Databricks normally follows medallion architecture for diff purposes
and diff teams access the data.
3)The Medallion Architecture is a data design pattern for Databricks
lakehouses that organizes data into three layers—Bronze, Silver, and
Gold—to progressively improve data quality. The Bronze layer stores
raw, unprocessed data, the Silver layer contains cleaned and validated
data, and the Gold layer holds business-ready.

8)Serve:
1)Finally we have Power-BI for build in dashboards.

Q) Discuss the average data volume handled and strategies used for
efficient processing?
A)
Average Data Volume Handled
This refers to the scale of data you typically work with in your projects.
For example, you might say: “We process 500 GB of transactional data
daily” or “Our pipelines handle millions of records per hour.”
Interviewers want to gauge whether you’ve worked with small,
medium, or big data workloads, since handling large-scale data
requires different tools and optimizations.
Strategies for Efficient Processing
This is about the methods and technologies you use to ensure data
pipelines run smoothly, quickly, and reliably.
Based on the surrounding topics in the guide, strategies could include:
Azure Data Factory (ADF): Using linked services, datasets, and triggers
(like tumbling windows) to schedule and optimize workflows.
PySpark/Spark: Leveraging lazy evaluation, caching vs. persisting, and
choosing between reduceByKey() vs. groupByKey() for performance.
Schema Management: Handling schema evolution in PySpark to avoid
pipeline failures.
Databricks Mount Points: Efficiently connecting to storage (like ADLS
Gen2) for streamlined access.
SQL/Python Validation: Ensuring data quality before processing to
reduce rework and errors.

Q) Highlight challenges faced in your projects and how you overcame

them?
A) yes, sudden spike happens in starting of the month or sometimes
starting of the year. unexpected increase in data volume or data points
at a certain stage of the pipeline. because of add many customers. That
time size of data will increase. when more customers and more data
size increased then we setup configuration
like auto scalability it increases clusters size. cluster size settings and
auto scaling configuration.