AWS Data Engineering — Comprehensive Reference Notes
These notes summarize the core AWS services used in modern data engineering pipelines, covering storage,
ingestion, processing, cataloging, and analytics. They are written as a study reference for interview preparation and
day-to-day pipeline design decisions.
1. S3 (Simple Storage Service)
S3 is the foundation of most data lakes because it is durable, cheap, and decoupled from compute. Objects are stored
in buckets and organized using prefixes that behave like folder paths, even though S3 itself is a flat key-value store
under the hood.
Key features to know for interviews: versioning (keeps historical copies of an object), lifecycle policies (auto-
transition objects to Infrequent Access, Glacier, or Deep Archive to save cost), and event notifications (S3 can
trigger Lambda, SNS, SQS, or EventBridge whenever an object is created or deleted).
Storage classes matter for cost optimization. Standard is for frequently accessed data, Standard-IA and One Zone-IA
are for infrequent access, Glacier Instant/Flexible/Deep Archive are for long-term cold storage with varying retrieval
times, and Intelligent-Tiering automatically moves objects between tiers based on access patterns.
Partitioning data in S3 by date (year=/month=/day=) or another high-cardinality key is critical — it lets query
engines like Athena and Redshift Spectrum skip irrelevant data (partition pruning), which directly reduces scan cost
and query latency.
2. AWS Glue
Glue is a serverless ETL service built on top of Apache Spark. It removes the need to manage clusters directly,
making it a common choice for scheduled batch transformations.
Glue Crawlers scan data sources (S3, JDBC, DynamoDB) and infer schema automatically, populating the Glue Data
Catalog. This catalog is the shared metadata layer that Athena, Redshift Spectrum, and EMR can all query against
without re-defining schemas.
Glue Jobs can be authored in PySpark or Python Shell. Glue also supports bookmarks, which track already-
processed data so that incremental jobs do not reprocess the same files on every run — this is important for building
idempotent, cost-efficient pipelines.
DynamicFrames (Glue's abstraction over Spark DataFrames) handle semi-structured or evolving schemas more
gracefully than plain DataFrames, which is useful when upstream JSON sources change shape over time.
3. EMR (Elastic MapReduce)
EMR provisions managed Hadoop and Spark clusters for large-scale distributed processing. It gives more control
than Glue — custom bootstrap actions, specific instance types, and long-running or interactive clusters — at the cost
of more operational overhead.
Cluster cost is controlled through instance fleets that mix On-Demand and Spot instances. Spot instances are
significantly cheaper but can be reclaimed, so critical master/core nodes are usually kept On-Demand while task
nodes use Spot.
EMRFS lets clusters read and write directly to S3 as if it were HDFS, which allows compute (the cluster) and
storage (S3) to scale independently — clusters can be terminated after a job completes without losing data.
Common interview topics include choosing between transient clusters (spun up per job, cost-efficient for batch)
versus persistent clusters (always on, better for frequent or interactive workloads), and tuning executor
memory/cores for Spark jobs running on EMR.
4. Athena
Athena is a serverless, pay-per-query SQL engine that reads directly from S3 using the Glue Data Catalog for
schema. There is no cluster to manage, and cost is based on bytes scanned.
Performance and cost both improve dramatically by using columnar formats like Parquet or ORC instead of row-
based formats like CSV or JSON, because columnar formats let Athena skip reading unused columns entirely.
Partitioning combined with columnar storage is the single biggest lever for reducing Athena cost — proper
partitioning can cut scanned data (and therefore cost) by well over 50 percent, which is a number worth having
ready for interviews.
CTAS (Create Table As Select) queries in Athena can be used to materialize transformed, partitioned, and
compressed datasets for downstream consumption.
5. Kinesis and DynamoDB Streams
Kinesis Data Streams is used for real-time, high-throughput ingestion — think clickstream data, IoT telemetry, or
log aggregation. Data is organized into shards, and throughput scales by increasing shard count.
Kinesis Data Firehose is the simpler, managed alternative for streaming data directly into S3, Redshift, or
OpenSearch without needing custom consumer code, often used when the goal is just reliable delivery rather than
custom stream processing.
DynamoDB Streams captures an ordered, near-real-time log of item-level changes (inserts, updates, deletes) in a
DynamoDB table. This is commonly wired to Lambda to build event-driven pipelines — for example, updating a
search index or triggering downstream aggregation whenever a record changes.
6. Redshift
Redshift is a columnar, MPP (massively parallel processing) data warehouse designed for complex analytical
queries over large volumes of structured data.
Distribution keys determine how rows are spread across compute nodes; choosing a good distribution key (often a
frequently joined column) avoids expensive data shuffling during query execution. Sort keys determine the physical
order of rows on disk and are chosen based on common filter or range-query columns.
Redshift Spectrum extends Redshift to query data sitting directly in S3 without loading it in first, which is useful for
querying infrequently accessed historical data alongside actively loaded warehouse tables.
7. Lambda, Step Functions, and EventBridge
Lambda provides serverless compute for short-lived tasks — file validation, small transformations, or triggering
downstream jobs — and integrates natively with S3, DynamoDB Streams, Kinesis, and EventBridge.
Step Functions orchestrate multi-step workflows (e.g., extract, then validate, then load, with retries and error
branches) using a visual state machine, which is often easier to reason about than chaining Lambda functions
manually.
EventBridge is an event bus used to decouple producers and consumers — a service can publish an event without
knowing which downstream systems will react to it, which keeps architectures loosely coupled and easier to extend.
Summary
Together, these services form the typical AWS data engineering stack: S3 for storage, Glue and EMR for
processing, the Glue Data Catalog as shared metadata, Athena and Redshift for querying and analytics, and
Kinesis/Lambda/EventBridge/Step Functions for real-time and event-driven orchestration.