Pre-requisites
Python Roadmap
SQL Roadmap Data Engineer
What is Data Engineering?
Data Engineering vs Data Science
Introduction Skills and Responsibilities
Data Engineering Lifecycle
Python is recommended
Choosing the Right Technologies
Python Java Scala Go
Programming Skills Learn the Basics
Understand Different Steps
Data Structures and Algorithms Data Generation 1
Git and GitHub Data Storage 2
Data Engineering Lifecycle
Linux Basics Data Ingestion 3
Networking Fundamentals Data Serving 4
Distributed Systems Basics Data Generation
Database
Data Normalization
Sources of Data APIs Logs
Data Modelling Techniques
Data Collection Considerations Mobile Apps IoT
CAP Theorem
OLTP vs OLAP
Learn SQL Indexing
Database Fundamentals Data Storage Transactions
Relational Databases
Slowly Changing Dimension - SCD
MySQL PostgreSQL
Horizontal vs Vertical Scaling Relational Databases
MariaDB Aurora DB
Star vs Snowflake Schema
Oracle MS SQL
Document Column NoSQL Databsases Graph Key-Value
MongoDB Cassandra Neo4j Redis
ElasticSearch BigTable Neptune Memcached
Data Warehousing
CosmosDB HBase DynamoDB
CouchDB
What is Data Warehouse?
Data Warehouse Data Warehousing Architectures Data Mart Data Mesh
Google BigQuery Other Data Architectures
Snowflake Data Fabric Data Hub
Amazon Redshift Metadata-first Architecture
Serverless Options
Data Lake
Databricks Delta Lake
Cloud Computing
Cloud Architectures
Snowflake
Onehouse Amazon EC2 ( Compute)
Data Ingestion S3 (Storage)
Amazon RDS (Database)
Batch
Glue (ETL)
Hybrid Types of Data Ingestion
AWS
Streaming
Azure Virtual Machines
Realtime
Data Pipelines
Azure Blob Storage
ETL Process
Azure SQL Database
Cluster Computing Basics Extract Data
Data Factory (ETL)
Transform Data
Azure
What is Cluster Computing
Load Data
Compute Engine (Compute)
Distributed File Systems
Data Pipeline Tools
Google Cloud Storage
HDFS Apache Airflow
Cloud SQL (Database)
dbt Luigi
Job Scheduling Dataflow
Perfect
Cluster Management Tools Google Cloud
Cloud Providers
Kubernetes
Apache Hadoop YARN Big Data Tools Apache Spark
Hadoop Ecosystem
Docker Kubernetes
HDFS YARN
Google Cloud GKE Containers & Orchestration
MapReduce
AWS EKS
Prometheus CI/CD GitHub Actions Circle CI
Datadog Sentry Monitoring GitLab CI ArgoCD
New Relic
Unit Testing
Integration Testing
What and why use them? Testing End-to-End Testing
Async vs Sync Communication Functional Testing
Messages vs Streams A/B Testing
Best Practices Messaging Systems Load Testing
Smoke Testing
Common Tools
Apache Kafka
RabbitMQ Infrastructure as Code - IaC Declarative vs Imperative
AWS SQS Idempotency
AWS SNS Reusability
Environmental Management
Visit the Data Analyst Roadmap Common Tools
Data Serving Terraform
Data Analytics
OpenTofu
Business Intelligence
AWS CDK
BI Tools
Authentication vs Authorization
Google Deployment Mgr.
Microsoft Power BI
Encryption
Streamlit
Tokenization
Tableu
Data Masking
Looker
Data Obfuscation
Data Quality
Data Lineage
Reverse ETL Security
Metadata Management
ETL vs Reverse ETL Data Interoperability
Reverse ETL Usecases Data Quality
Data Governance
Tools
Hightouch
Census GDPR
Segment Privacy ECPA
EU AI Act
Data and AI Regulations
Machine Learning
MLOps