INTRODUCTION TO AZURE
DATA FACTORY
What is Azure Data Factory
Azure Data Factory is Microsoft’s cloud-based ETL service for scale-out server less
data integration and data transformation that allows to create data-driven workflows
for orchestrating data movement and transforming data at scale.
Existing SSIS (SQL Server Integration Services) packages can also be shifted to Azure
and run with full compatibility in ADF.
Example - suppose, there is a very popular game, and, when people play that game,
many logs get generated. Those logs are stored in Azure Data Lake Storage (ADLS).
ADLS is mainly used to store big data (huge amount of unstructured data). Say, the
customer information, marketing or campaigning related information are stored in on
premise databases.
The logs from ADLS are extracted and transformed into some meaningful data and
loaded into Azure SQL Database. On top of the Azure SQL Storage, reporting tools like
PowerBI is used to generate meaningful reports, e.g. - monitoring trends regarding
which advertisements are users watching more while playing the game etc., to take
business decisions which helps running the business.
The entire process of extracting, transforming and load can be easily be achieved with
Azure Data Factory (ADF).
What Azure Data Factory is Used
Big data requires service that can orchestrate and operationalize processes to refine
the enormous stores of raw data into actionable business insights. Azure Data Factory
is a managed cloud service that is built for complex, hybrid Extract-Transform-Load
(ETL), or, Extract-Load-Transform (ELT), and, data integration projects.
Top Level Concepts in Azure Data Factory
Pipeline - A Pipeline is a logical grouping of activities that performs a unit of work in
sequential order. A Data Factory might have one or more Pipelines.
Example - a Pipeline can contain a logical group of activities in sequential order, where
the first activity ingests data from an Azure Blob Storage to Hive, and, the second
activity runs a Hive query on an HDInsight cluster to partition the data.
Activity - An Activity represents a processing step in a Pipeline.
Example - a Copy Activity might be used to copy data from one data source to another
data source.
Linked Services - Linked Services are much like connection strings, which define the
connection information that is needed for Data Factory to connect to external
resources, e.g., Azure Blob Storage.
Example - a Linked Service for Azure Blob Storage specifies a connection string to
connect to the Azure Blob Storage account.
Data Sets - Datasets represent data structures within the data stores, which simply
point to, or, reference the data to use in the Activities.
Example - an Azure Blob Storage Dataset specifies the blob container and the folder
that contains the data to be used in the Activities.
Triggers - Triggers determine when a Pipeline execution needs to be kicked off. There
are different types of Triggers.
Creating First Azure Data Factory
Pre-requisites -
➢ Azure Subscription
➢ Azure Storage Account
Process - The task is to copy data from one storage account to another storage
account.
➢ Step 1 - Open the Azure portal ([Link]).
➢ Step 2 –
• Step 2.1 - Create a Storage Account by first clicking on the “Create a
resource” icon from the Azure services.
• Step 2.2 - Type “Storage account” and press Enter.
• Step 2.3 - Now, click on the “Create” button.
• Step 2.4 - Provide “TestEnvironmet” as Resource group, and,
“demostorageoindrila1” as Storage account name. Leave the rest of the
options as it is and click on the “Review + create” button.
• Step 2.5 - The validation is passed as seen in the image because all the
information provided in the last step was correct. Now, click on the
“Create” button.
• Step 2.6 - Deployment has started and it will take 1-2 minutes.
• Step 2.7 - Deployment is completed. Click on the “Go to resource”
button. If this button is disabled, click on the “Refresh” link. The “Go to
resource” button will get enabled.
• Step 2.8 - The storage account just created “demostorageoindrila1” is
opened.
➢ Step 3 -
• Step 3.1 - Create a container by first clicking on the “Containers” link.
• Step 3.2 – Now, click on the “Container” link.
• Step 3.3 - Provide “democontainer” as Name, and, click on the “Create”
button.
• Step 3.4 - The container “democontainer” is successfully created as the
message is displayed at the right top corner of the page. Now, click on
the container name “democontainer” to go inside the container.
➢ Step 4 - Now, create a folder “textFiles” inside the container “democontainer”
and upload a text file “[Link]” from the machine.
• Step 4.1 - Now, first click on the “Upload” link.
• Step 4.2 - Click on the file browser icon to upload file from the machine.
• Step 4.3 - Select the text file “[Link]” and it will be displayed in the
textbox for “Files”. Click on the “Advanced” dropdown.
• Step 4.4 - Provide “textFiles” as Upload to folder. Leave the rest of the
options as it is and click on the “Upload” button.
• Step 4.5 - The folder “textFiles” is generated.
• Step 4.6 - Clicking on the folder “textFiles”, the text file “[Link]” is
displayed inside it.
• Step 4.7 - Now, Clicking on the three dots (…) in the line of the file name,
the Context Menu for the file is displayed. Click on the “View / edit”
menu.
• Step 4.8 - The content of the file is displayed along with line numbers.
• Step 4.9 - In the home page, the added storage resource
“demostorageoindrila1” is displayed under the “Recent resources”
section.
➢ Step 5 -
• Step 5.1 - Create an Azure Data Factory by first clicking on the “Create
a resource” icon from the Azure services.
• Step 5.2 - Type “data factory” and select the first option “Data Factory”.
• Step 5.3 - Now, click on the “Create” button.
• Step 5.4 - Provide “DemoADFOindrila” as Name, “TestEnvironmet” as
Resource group, and, tick off Enable GIT. Leave the rest of the options
as it is and click on the “Create” button.
• Step 5.5 - Deployment is completed. Click on the “Go to resource”
button. If this button is disabled, click on the “Refresh” link. The “Go to
resource” button will get enabled.
• Step 5.6 - The Azure Data Factory account just created
“DemoADFOindrila” is opened.
• Step 5.7 - Click on the “Author & Monitor” link.
• Step 5.8 - The Azure Data Factory is opened in a new tab in the same
browser in edit mode.
• Step 5.9 - In the home page, the added ADF resource “DemoADFOindrila”
is displayed under the “Recent resources” section.
➢ Step 6 - Now, the goal is to create a Pipeline to copy the text file “[Link]”
from the folder “textFiles” and paste inside the folder “outputTextFiles”. If the
folder “outputTextFiles” does not exist, ADF will create a folder “outputTextFiles”
inside the container “democontainer” of the data storage account
“demostorageoindrila1”. If the folder “outputTextFiles” exists, ADF will use the
folder. So, ADF will have to interact with the data storage account.
• Step 6.1 - Now, first click on the “Author” link.
• Step 6.2 - Since, ADF will have to interact with the data storage account,
a Linked Service needs to be created which will have the connection
string to the data storage account “demostorageoindrila1”.
✓ Step 6.2.1 - Click on the “Manage” icon.
✓ Step 6.2.2 - Click on the “New” link under the Linked services.
✓ Step 6.2.3 - Select “Azure Blob Storage” from the “Data store”
section and click on the “Continue” button.
✓ Step 6.2.4 - Provide “LinkedService_DemoStorageAccount” as
Name. Select “Free Trial” from the Azure subscription dropdown,
and, “demostorageoindrila1” from the Storage account name
dropdown. Click on the “Create” button.
✓ Step 6.2.5 - The Linked Service
“LinkedService_DemoStorageAccount” is created inside the ADF
account “DemoADFOindrila” to connect to the data storage
account “demostorageoindrila1”.
• Step 6.3 - Since, the goal is to copy the text file “[Link]” from the
input folder “textFiles” and paste inside the output folder
“outputTextFiles”, data sets need to be created for both the “textFiles”
files of the input folder, as well as for the output folder.
✓ Step 6.3.1 - Now, first hover on the “Datasets” menu and click
on the three dots (…) to create the input data set.
✓ Step 6.3.2 - Click on the “New dataset” menu.
✓ Step 6.3.3 - Now, select the “Azure Blob storage” and click on
the “Continue” button.
✓ Step 6.3.4 - Since, the type of files to be dealt with are text files,
“Binary” format should be selected. Click on the “Continue”
button.
✓ Step 6.3.5 - Provide “InputDataSet” as Name,
“LinkedService_DemoStorageAccount” as Linked service. Click
on the file browser icon.
✓ Step 6.3.6 - Select and double click on the container name
“democontainer”.
✓ Step 6.3.7 - Select and double click on the folder name
“textFiles”.
✓ Step 6.3.8 - Select the file name “[Link]” and click on the “OK”
button.
✓ Step 6.3.9 - So, the data set “InputDataSet” is pointing to the
text file “[Link]” of the folder “textFiles”, which is inside the
container “democontainer”. Click on the “OK” button.
✓ Step 6.3.10 - The input data set “InputDataSet” is created.
✓ Step 6.3.11 - Now, again hover on the “Datasets” menu and click
on the three dots (…) to create the output data set.
✓ Step 6.3.12 - Click on the “New dataset” menu.
✓ Step 6.3.13 - Now, select the “Azure Blob storage” and click on
the “Continue” button.
✓ Step 6.3.14 - Since, the type of files to be dealt with are text files,
“Binary” format should be selected. Click on the “Continue”
button.
✓ Step 6.3.15 – Provide “OutputDataSet” as Name,
“LinkedService_DemoStorageAccount” as Linked service. Click
on the file browser icon.
✓ Step 6.3.16 - Select the container name “democontainer”. Click
on the “OK” button.
✓ Step 6.3.17 - Since, the output folder “outputTextFiles” does not
exist, it can’t be browsed. So, provide the folder name
“outputTextFiles” in the Set properties section.
✓ Step 6.3.18 - The file name can be provided as either the same
file name from the input folder “textFiles”, or, can be provided as
a different file name. Here, provide a different text file name
“[Link]”.
So, the data set “OutputDataSet” is pointing to the text file
“[Link]” of the folder “outputTextFiles”, which is inside
the container “democontainer”. Click on the “OK” button.
✓ Step 6.3.19 - The output data set “OutputDataSet” is created.
• Step 6.4 - Now, the Pipeline to copy the text file “[Link]” from the
input folder “textFiles” and paste inside the output folder
“outputTextFiles” needs to be created.
✓ Step 6.4.1 - Now, first hover on the “Pipelines” menu and click
on the three dots (…) to create a Pipeline.
✓ Step 6.4.2 - Click on the “New pipeline” menu.
✓ Step 6.4.3 - Now, “pipeline1” is created. Expand the tool “Move
& transform” from the Activities section, and, drag and drop the
Activity “Copy data”.
This Activity copies data from a source storage area and pastes
to a target storage area.
✓ Step 6.4.4 - Leave the properties settings of the “General” tab in
the “Copy data” Activity as it is.
✓ Step 6.4.5 - Go to the “Source” tab of the “Copy data” Activity to
provide information of the data set for source. Select the created
“InputDataSet” as Source dataset.
✓ Step 6.4.6 - Go to the “Sink” tab of the “Copy data” Activity to
provide information of the data set for destination. Select the
created “OutputDataSet” as Sink dataset.
✓ Step 6.4.7 - Now, before running the Pipeline, click on the
“Validate all” link to check if any configuration related error
exists in the Pipeline.
✓ Step 6.4.8 – No errors were found.
✓ Step 6.4.9 - Now, before publishing the Pipeline, Linked Service,
and, the Data Sets to the ADF, check if the data is actually copied
from the source folder to the target folder in the container of the
data storage. To do so, click on the ”Debug” link.
✓ Step 6.4.10 - Debug process takes a few seconds to complete.
The Debug process is successful.
✓ Step 6.4.11 - Now, in the container “democontainer” of the data
storage account “demostorageoindrila1”, the output folder
“outputTextFiles” is generated.
✓ Step 6.4.12 - Go inside of the “outputTextFiles” folder. The
output text file “[Link]” is generated inside this folder.
✓ Step 6.4.13 - Now, Clicking on the three dots (…) in the line of
the file name, the Context Menu for the file is displayed. Click on
the “View / edit” menu.
✓ Step 6.4.14 - The content of the file is displayed along with line
numbers. This content is the same as the input file “[Link]” of
the folder “textFiles” in the same container “democontainer”.
✓ Step 6.4.15 - The “Copy data” Activity does not copy the text file
“[Link]” and pastes into the output folder “outputTextFiles”
and renames it.
Instead, the “Copy data” Activity opens the text file “[Link]” in
the input folder “textFiles” in the container “democontainer” and
copies the data. Then the “Copy data” Activity creates the output
folder “outputTextFiles”, and, creates the output file
“[Link]” inside the folder. After that, the “Copy data”
Activity pastes the copied data into the output file
“[Link]”.
✓ Step 6.4.16 - Now, click on the “Publish all” button.
✓ Step 6.4.17 - The Pipeline “pipeline1” and the Data Sets
“InputDataSet” and “OutputDataSet” need to be published. Click
on the “Publish” button.
✓ Step 6.4.18 - The Pipeline “pipeline1” and the Data Sets
“InputDataSet” and “OutputDataSet” got saved in the ADF.