Hi
This is srinivas. I have total 10 years of experience.
Started my career as a SQL developer later I moved to BigData Stack.
Here I worked on Hadoop,Hive,Pyspark.
Later I got Moved to Azure stack.
Here out project aim is to migrate data from multiple source systems to Azure
Gen2Lake
and a build datalake.
In this I have done migration from onpremises sql server to Gen2Lake and build a
datalake.
For this we used azure services Azuregen2lake,Azure Keyvault, Azure Databricks,
Azuresql and Azure Data Factory.
Over all I am good at Pyhon,SQL,Hadoop,Hive,Pyspark,Azure Databricks and Azure Data
Factory.
-------------------------------------------------------
0-2 years : Introduction
Hi
This is srinivas. I have total 2 years of experience.
Started my career as a Azure Data Engineer.
Here out project aim is to migrate data from multiple source systems to Azure
Gen2Lake
and a build datalake.
In this I have done migration from onpremises sql server to Gen2Lake and build a
datalake.
For this we used azure services like Azuregen2lake,Azure Keyvault, Azure
Databricks,
Azuresql and Azure Data Factory.
====================================================
Explain you current Project:
----------------------------
My recent project is migration project from on premises sql server to
Azuregen2Lake.
In this we got requirement to migrate more than 200 tables in to Azure.
For this first we have to prepare metadata.
In our project we created metadata table azure sql .
Here Metadata contains What are the tables we have to migrate
What are the columns of the table
What is the incremental Column
What is the last incremental value
What are all sensitiv columns
What is the table load type(incremental or full
sync)
In our project using ADF we built pipelines.
Pipeline contains activities.
Using look up activity we connected to Azure sql and get metadata information .
This look up activity output is input to foreach activity.
In side foreach we used copy activity, notebook activity and stored procedure
activities.
For copy activity source is on premises sql server and sink is Gen2lake.
Here to connect to on premises sql server we used self hosted IR.
Here for copy activity we have to create respective linked services and data sets.
Here we have to provide the query , This is prepared from metadata table.
This query execute on source and result in to sink which is gen2 lake as json
format.
Next we used to notebook to process the data.
In the notebook using mount point connect to gen2lake and start processing data.
Here create a dataframe on json files apply transformations like
adding etldate columns
encrypt sensitive columns
apply any case conditions
join operations
date transformations
and then finally loaded to stage table.
Here in our project we used all delta tables only.
In our project we are maintaining different types on zones.
Stage
History
Trusted
Refined
Here stage is truncate an overwrite.
History is append one.
Trusted sometimes we are merging and sometimes we are appending.
and finally Refined zone based on use cases we joined multiple tables
and loaded into Refined zone.
These Refined zone data we are provied to powerBI team to generate the reports.
Here in our project we used scheduling triggers for the pipelines execution.
=======================================================