Azure

• ADLS Gen2 (hierarchical namespace, ACLs, tiers)

• Azure Blob Storage, Azure Files, Azure Table Storage

• Azure Data Factory (pipelines, activities, datasets, linked services, triggers)

• Integration Runtime (Azure IR, Self-hosted IR, SSIS IR)

• Mapping Data Flows, Debug mode, Schema drift handling

• Azure Databricks (notebooks, clusters, Unity Catalog, Asset Bundles)

• Azure Synapse Analytics (serverless SQL pool, dedicated SQL pool, Spark pools)

• Azure SQL Database, Azure SQL Managed Instance

• Azure Event Hubs, Azure Stream Analytics, IoT Hub

• Azure Cosmos DB, Azure Cache for Redis

• Azure Key Vault, Azure Active Directory (Managed Identities)

• Azure Monitor, Log Analytics, Application Insights, Alerts

• Azure Purview / Microsoft Purview (catalog, lineage, classification)

• ARM templates, Azure DevOps (Repos, Pipelines), Git integration

Azure Data Factory (ADF)

• Pipelines, Activities, Datasets, Linked Services

• Copy Activity vs Mapping Data Flows vs Custom activities vs Databricks/HDInsight

activities

• Triggers: Schedule, Tumbling Window, Event-based, Manual/REST

• Parameterization, dynamic content, Lookup/ForEach patterns

• Incremental loads (watermarks), CDC patterns, file-based incremental (Get

Metadata)

• Error handling: retries, on-failure branches, logging, quarantining bad records

• CI/CD: Git integration, ARM templates, ADF release pipeline best practices

• Integration with Key Vault, Managed Identities, Private Endpoints

• Monitoring (ADF Monitor), diagnostic logs to Log Analytics

AWS

• Amazon S3 (lifecycle, versioning, encryption)

• AWS Glue (jobs, crawlers, Catalog), Glue Studio

• Amazon EMR (Spark/Hadoop cluster management)

• AWS Lambda (serverless compute), Step Functions (orchestration)

• Amazon Redshift, Redshift Spectrum

• AWS DMS (CDC), Kinesis (Streams, Firehose), MSK (Kafka)

• IAM, KMS (encryption), Secrets Manager

• CloudWatch (metrics/logs), CloudTrail (audit)

• Terraform for AWS, CloudFormation, CodePipeline/CodeBuild

• Athena (serverless SQL over S3), Glue Data Catalog

Google Cloud Platform (GCP)

• Cloud Storage (uniform vs. hierarchical, storage classes, lifecycle management)

• BigQuery (serverless architecture, partitioning & clustering, BI Engine, ML in

BigQuery)

• Cloud Composer (managed Airflow, DAGs, operators)

• Dataproc (managed Spark/Hadoop, cluster customization, autoscaling)

• Dataflow (managed Apache Beam, unified batch & streaming, auto-scaling)

• Pub/Sub (event ingestion, ordering, exactly-once delivery)

• Cloud Functions (serverless, event-driven compute)

• Dataform (SQL workflow orchestration, data testing)

• Dataplex (unified data governance, catalog, lineage)

• Cloud IAM, Cloud KMS, Secret Manager

• Cloud Monitoring (Metrics Explorer, Logs Explorer, Alerting)

• Cloud Build (CI/CD), Terraform for GCP

Data Warehousing & Modeling (DWH)

• Star schema vs Snowflake schema

• Kimball vs Inmon methodologies

• Slowly Changing Dimensions (SCD Type 0/1/2/3/4/6)

• Fact table design (additive, semi-additive), grain, surrogate keys

• Partitioning, clustering, sort keys, distribution keys (Redshift/Synapse)

• Materialized views, aggregate tables, OLAP vs OLTP distinctions

• Data vault fundamentals (if applicable)

• Data marts vs enterprise DW vs data lakehouse

Apache Spark & Databricks

• Spark architecture: driver, executors, cluster managers (YARN, Mesos, Kubernetes)

• RDD vs DataFrame vs Dataset; Catalyst optimizer and Tungsten

• Jobs → Stages → Tasks; Spark UI / event logs

• Partitioning strategies, repartition vs coalesce, shuffle mechanics

• Broadcast joins, shuffle joins, salting, skew mitigation

• Map vs mapPartitions, mapPartitionsWithIndex, partitioners

• Memory model: executor memory, storage vs execution memory, shuffle buffers,

spills

• GC tuning, serialization (Kryo), off-heap / Tungsten memory

• Structured Streaming (micro-batch, continuous), checkpointing, watermarking

• UDFs/UDAFs, Pandas UDFs, vectorized execution

• Delta Lake: transaction log, MERGE, time travel, OPTIMIZE, VACUUM, ACID semantics

• Databricks-specific: Jobs, Workspaces, Repos, Unity Catalog, Photon, cluster policies

ETL / ELT Patterns & Pipelines

• Bronze / Silver / Gold layering (lakehouse patterns)

• Batch vs Micro-batch vs Streaming architectures

 • CDC patterns and exactly-once semantics (idempotency, upserts)

• Orchestration tools: ADF, Airflow, Prefect, Step Functions

• Data contracts, schema registry, schema evolution strategies

• Metadata-driven pipelines, control tables and configuration-driven jobs

• Data Lakehouse Architecture: Combining the best of data lakes and data warehouses.
• Medallion Architecture: A specific, proven implementation of Bronze/Silver/Gold.
• Data Quality Frameworks: Integrating checks (e.g., with Great Expectations, Soda
Core) into pipelines.
• Orchestration: Deeper dive into Apache Airflow (Sensors, XComs, TaskFlow API).

Apache Airflow

• DAGs (Directed Acyclic Graphs) as core abstraction

• Architecture: Scheduler, Webserver, Executor (Local, Celery, Kubernetes)

• Operators (Python, Bash, SQL, Docker), Sensors, Hooks

• Task dependencies (set_upstream/downstream, bitshift operators)

• Jinja Templating for dynamic tasks and parameterization

• Variables, Connections, XComs for cross-task communication

• Scheduling and backfilling (start_date, schedule_interval, catchup)

• TaskFlow API for simplifying Python function-based DAGs

• Monitoring: DAG/tree views, logs, SLA misses, alerting

• Executors: LocalExecutor, CeleryExecutor, KubernetesExecutor

• Best practices: idempotency, task boundaries, efficient DAG design

dbt (data build tool)

• Models (SQL files), materializations (table, view, incremental, ephemeral)

• Project structure: models, seeds, snapshots, tests, macros

• ref and source functions for dependencies and DAG generation

 • Jinja templating for dynamic SQL and code reuse

• Incremental models and strategies (merge, delete+insert)

• Testing: generic tests (unique, not_null, accepted_values, relationships) and singular

tests

• Documentation: [Link], [Link], dbt docs generate/serve

• Macros for reusable SQL code and custom functionality

• Snapshots for tracking Slowly Changing Dimensions (SCD Type 2)

• Hooks (pre-hook, post-hook) and run operations

• Packages for sharing and reusing dbt code

• dbt Cloud vs dbt Core (CLI)

• The dbt build command (run, test, snapshot in one)

Snowflake

• Architecture: Hybrid of shared-disk and shared-nothing (Separation of Storage,

Compute, and Cloud Services)

• Virtual Warehouses: Independent compute clusters for scaling, auto-

suspend/resume, multi-cluster (for concurrency)

• Data Storage: Columnar format in micro-partitions, automatic clustering

• Time Travel: Data recovery (0-90 days), AT|BEFORE clause, Fail-safe

• Zero-Copy Cloning: Instant, cost-efficient copies of databases/schemas/tables

• Data Sharing: Secure data sharing (public and private), Data Marketplace

• Snowflake Data Cloud: Cross-cloud data platform (AWS, Azure, GCP)

• SQL Support: ANSI SQL, stored procedures, UDFs (JavaScript, Java, Scala, Python)

• Snowpark: DataFrame API for Scala, Java, Python - pushdown operations to

Snowflake engine

• Streams: Change Data Capture (CDC) on tables for processing incremental changes

• Tasks: Scheduled SQL or Snowpark scripts for workflow automation

• Materialized Views: Pre-computed results for faster query performance

• External Tables: Query data in external stages (S3, Azure Blob, GCS)
 • Stages: Internal (Snowflake) and external (cloud storage) locations for data files

• File Formats: CSV, JSON, Parquet, Avro, ORC with schema detection

• Continuous Data Pipelines: Combining Streams and Tasks for ELT

• Security: RBAC, network policies, OAuth, MFA, column-level security, row access
policies

• Data Governance: Tagging, object dependencies, access history

• Cost Management: Warehouse sizing, resource monitors, credit usage optimization

• Semi-structured Data: Native JSON/XML/Variant support with dot notation and

FLATTEN

• Data Loading: COPY INTO command, Snowpipe for continuous ingestion

• Partner Integrations: dbt, Airflow, Fivetran, Informatica, Tableau

SQL & Query Optimization

• Window functions, analytic functions, common table expressions (CTE)

• Indexing strategies, covering indexes, execution plans, statistics

• Join strategies (hash, merge, nested loop), join order, hints

• Partition elimination, predicate pushdown, materialized views

• Query profiling and optimization in Synapse/Redshift/Athena/BigQuery

Programming & Testing

• Python (pandas, PySpark), lambda, list comprehensions, memory management

• Multithreading vs multiprocessing, async I/O patterns for ETL tasks

• Unit testing (pytest), integration testing, mocking (SparkSession, DB connections)

• Data testing frameworks: Great Expectations, Deequ, dbt tests (where applicable)

• Packaging: wheels, conda/mamba, containerization (Docker)

DevOps, IaC & CI/CD

• Terraform: modules, tfstate management, backends, workspaces, tfvars patterns

• CI/CD pipelines: Git branching strategies, build/test/deploy for pipelines, notebooks,

infra
 • Artifact management, release gating, approvals, blue-green / canary deployments for
schema changes

Observability, Ops & Resilience

• Centralized logging (Log Analytics, CloudWatch, ELK), metrics and dashboards

• Alerting/SLAs, synthetic checks, runbook/playbook creation

• Lineage tracking and auditing (Purview, Data Catalogs, manual lineage capture)

• Cost monitoring and optimization (compute auto-termination, right-sizing, file

compaction)

Security & Compliance

• Encryption at rest and in transit, customer-managed keys (CMK)

• RBAC, fine-grained access control, Unity Catalog / Table ACLs, row/column masking

• PII handling: masking, tokenization, pseudonymization, data retention and deletion

workflows (GDPR)

• Network security: VNet, private endpoints, service endpoints

BI & Consumption

• Power BI / Tableau integration patterns (direct query vs extracts)

• Data marts, semantic layers, materialized aggregates, performance tuning for

dashboards

• Row-level security and embedding metadata (freshness, lineage) into BI

Metadata, Catalog & Governance

• Data cataloging tools: Azure Purview, AWS Glue Data Catalog, DataHub, Amundsen

• Business glossary, data classifications, policy enforcement, stewardship workflows

Miscellaneous / Advanced

• Graph processing (GraphX), MLlib basics and feature engineering at scale

• Streaming platforms (Kafka, Confluent, MSK) and event-driven architectures

• Cost modeling, capacity planning, data retention strategies