------------------------------------------------------------------------------------------------------------------------------------
Azure Data Engineering with AI
[Trainer: Mr. Ravi 19+ years of experience] [Duration: 3 Months]
Module 1 : Introduction to Data Engineer
● What is Data ?
● Why Data Engineering ?
● Who is Data Engineer ?
● Data Engineer vs Data Analyst vs Data Scientist
● What does Data Engineer do ?
● Data Engineering Architecture
● Data Engineer Workflow
● Real world examples
● Tools Used by Data Engineer
Module 2 : Cloud Computing
● Before Cloud Computing
● What is Cloud Computing
● Cloud Providers
● Advantages of Cloud Computing
● Types of Cloud Computing
○ Deployment Model
○ Service Model
● Global Infrastructure
● Regions & Availability Zones
● Core Characteristics
● How to choose right region
Module 3 : AWS Fundamentals
● AWS Introduction
● AWS VPC basics
● Internet Gateway (IGW)
● Route Tables (RTs)
—----------------------------------------------------------------------------------------------------------------------------
● Network Access Control List (NACL)
● NACL Rules
● Subnets
● Elastic Cloud Compute (EC2)
● Elastic Block Storage (EBS)
● Relational Database Service (RDS)
● Simple Storage Service
● Storage Classes
● Object Lifecycle
● Identity & Access Management (IAM)
● IAM components
● Hands-On
Module 4 : Database Overview
● Database History
● What is Database ?
● DBMS vs RDBMS
● Types of Data
○ Structured
○ Un-structured
○ Semi-Structured
● Types of Databases
● RDBMS fundamentals
● Key RDBMS concepts
○ Primary key
○ Foreign key
○ ACID properties
○ Normalization
● What is NoSQL ?
● Types of NoSQL Databases
● When to use SQL vs NoSQL
Module 5 : Big Data
● Traditional Database Limitations
—-----------------------------------------------------------------------------------------------------------------------------
● Introduction to Big Data
● Sources of Big Data
● Characteristics
● How Big Data is processed?
● Before Distributed Computing
● What is Distributed Computing
● Real-World Examples
Module 6 : SQL Basics
● What is SQL ?
● What can SQL do ?
● How does SQL work ?
● Key components of SQL
● Rules for writing SQL commands
● Different SQL commands
○ DDL
○ DML
○ TCL
○ DQL
○ DCL
● CREATE
● ALTER
● DROP
● TRUNCATE
● Constraints
● Keys in SQL
○ Unique key
○ Primary key
○ Foreign key
○ Candidate key
○ Composite key
○ Surrogate key
● SQL Joins
○ Inner join
○ Left join
—----------------------------------------------------------------------------------------------------------------------------
○ Full join
○ Right join
● Window functions
● Sub query
● Common Table Expressions (CTEs)
● Views
● Stored Procedures
● Functions
● Transform
● Date Functions
● Aggregations
● SQL Transactions
Module 7 : Azure Fundamentals
● Introduction to Azure
● Azure Global Infrastructure
● Azure Management Infrastructure
○ Management Group
○ Subscription
○ Resource Group
○ Resources
● Identity & Access Management (IAM)
○ Azure Active Directory (Azure AD / Entra ID)
○ What is Azure AD ?
○ Authentication vs Authorization
○ Service Principle vs Managed Identities
○ Azure RBAC (Role Based Access Control)
○ Azure Users, Groups, Roles
● Azure Virtual Network (V-Net)
○ Subnet
○ Route Table
○ Network Security Groups (NSG)
● Virtual Machine(VM)
● Virtual Machine Scale Set (VMSS)
● Vnet Peering
—----------------------------------------------------------------------------------------------------------------------------
Module 8 : Azure Storage
● Azure Storage Account
● Types of Storage Accounts
● Blob Storage
● Blob Hierarchy
● Types of Blob Storage
● Azure Data Lake Storage Gen2
● Access Tiers
● Blob vs ADLS
● ADLS Hierarchy
● Data Lake Zones
● Data Replication Policies
● Security in Storage
● Hands-On
Module 9 : Azure Data Factory (ADF)
● What Pipeline ?
● Why pipelines are required ?
● Core components of pipeline
● ETL vs ELT
● What is Azure Data Factory
● Azure Data Factory Architecture
● Azure Data Factory Portal UI
● Key Concepts of ADF
○ Pipelines
○ Activities
○ Linked services
○ Datasets
○ Triggers
○ Data Flows
○ Integration Runtimes
● Create Azure Data Factory Workspace
● Data set and Linked Service in ADF
● Data ingestion in Data Lake
—----------------------------------------------------------------------------------------------------------------------------
● Copy Activity in ADF
○ General
○ Source
○ Sink
○ Mappings
○ Settings
○ User Properties
● Copy data using REST API in ADF
● Get Metadata Activity
● If Condition Activity
● For Each Activity
● Expression Builder in ADF
● Parameterized Pipeline
● Data Flow
● Data Transformation
○ Select
○ Filter
○ Conditional Split
○ Derived Column
○ Aggregate
○ AlterRows
● Triggers
○ Tumbling Trigger
○ Scheduled Trigger
○ Storage Event Trigger
● Set Variable Activity
● Real Time Scenario in ADF
● Validate Schema & Schema Drift
● Delete Activity
● Debugging Pipeline
● Execute Pipeline Activity
● Logic apps
● Azure Key Vault
● Event hub
○ Kafka vs Event Hub
○ Publish and receive messages
—----------------------------------------------------------------------------------------------------------------------------
○ Event Hub demo
● Orchestration & Monitoring
● Integrating ADF with Github
● * ADF Exercises (Real-Time Scenarios)
● * Hands-on Project
Module 10 : Python Basics
● Python Basics
● Variables
● Data Types
● Operators
● Collections
● Functions
● Exception Handling
● Conditions
● Loops
Module 11 : Data Modeling
● What is Data Modeling ?
● Conceptual Data Model
● Logical Data Model
● Physical Data Model
● Frameworks built on data modelling
○ Star Schema
○ Snowflake Schema
● Star Schema vs Snowflake Schema
● When to use Star & Snowflake Schemas
Module 12 : Databricks with PySpark:
● Sign up for the free edition
● Explore Databricks UI
● Dataframe Basics
● Distributed Computing, Spark, and Databricks
—----------------------------------------------------------------------------------------------------------------------------
● SQL and joins in Spark
● Reading Spark plans with explain
● Spark Architecture
● Transformations vs Actions and Lazy Evaluation
● Narrow vs Wide Transformation
● Partition and Parallelism: Repartition
● Partition and Parallelism: Coalesce
● Managed vs External Tables
● Medallion architecture
● Unity Catalog
● Acid & Time Travel
● About AWS Account & Delta Lake
● Delta Lake and Delta format
● Intro & Tech Architecture
● Catalog Setup
● Data Understanding and Upload
● Dimension Data Processing
● Fact Data Processing
● * Databricks AI Assistant
● * Databricks AI Functions
● * AI Powered Databricks IQ
● * Hands-On Project
Module 13 : Azure Synapse:
● Introduction
● Synapse Analytics Overview
● Azure Synapse Architecture
● Azure Synapse Analytics Workspace & its Creation
● Explore Synapse Analytics UI
● Dedicated SQL Pool
● Serverless SQL Pool
● Azure Synapse Spark Pool
● Medallion Architecture
● Distributed Tables in Azure Synapse
● Synapse Distribution Strategies
—----------------------------------------------------------------------------------------------------------------------------
● Data ingestion in Synapse
● Create views in Serverless SQL pool
● Copy Activity
● Hands-On
Module 14 : Microsoft Azure Fabric & Hands-On
Module 15 : End-to-End Data Engineering Project
● Practice Tests
● Case Studies
● Weak Areas Review
● Mock Interviews
● Hands-on Projects