AWS Data engineer

Course 1

The AWS Data Engineer Associate Certification: Explained

So first things first, is the data engineering associate certificate the right
certificate for you? To answer this question, let's go over the certificate. In
terms of prerequisites, AWS recommends the equivalent of 2 to 3 years in
data engineering or data architecture, and a minimum of 1 to 2 years of
hands-on experience with AWServices. Having said that though, technically
speaking, there are no hard prerequisites. You can still study for and attempt
the certificate without having these many years of experience. In terms of
achievements, this certificate should equip you with the knowledge and skills
needed to build systems and pipelines that collect, manage, and transform
your data into beneficial information. Your role as a data engineer revolves
around pipelines, so you'll manage the flow of the data by ingesting and
integrating it from various sources, and then ultimately publishing it into
reports. And if you like to view things from a visionary slash big picture
perspective, the purpose of your role is to make data accessible for data
scientists, analysts, and business specialists so that you can supply them
with the high-quality data needed for decision making. Having the exam is
broken down into 4 main domains. The first domain is data ingestion and
transformation. This is where you'll test your ability on sourcing and
transforming data in ETL pipelines. Next domain is data store management,
which is all about knowing how to pick the correct storage platform for the
right job and knowing how to configure it in an optimal matter. Data
operations and support is all about automating data processing via
AWServices. Lastly, data security and governance, as the name conveys, is
all about securing your AWS services, so it focuses on topics such as
authentication, authorization, encryption, and masking.

About the Exam

[Autogenerated] In terms of exam weightings, data ingestion and

transformation makes up 34% of the exam. Data store management makes
up 26%, data operations and support makes up 22%, and data security and
governance makes up 18%. The exam itself, as you probably know, is an
associate level exam, so it requires a higher tier of expertise and
specialization over the foundational level exams that cover broader and
more general concepts. In terms of the duration, you can take up to 130
minutes, so around 2 hours and 10 minutes to complete the exam. To
register, you'll be charged $150. In terms of the format, you'll be presented
with 65 multiple choice questions, and the questions will instruct whether to
pick one correct response or multiple correct responses. Having to take the
exam, you'll need to choose between a proctor test center or taking it
remotely via an online proctored option. Lastly, in terms of languages, the
exam is offered in English, Japanese, Korean, and simplified Chinese. You can
also request an accommodation if English is your second language. If
approved, you'll be given additional 30 minutes to complete the exam.

Scheduling the Exam

[Autogenerated] Now, let's talk about scheduling and taking the exam. If you
choose to take the exam through a test center, you'll obviously need to go to
the test center. So be prepared for any hiccups on the road, and if possible,
aim to arrive 30 minutes or so before the exam. You'll need to take your ID.
This can be your driver's license, passport, or some sort of an official ID card.
Lastly, be mentally prepared to work in a new environment. You might face
issues with a slow computer or a noisy environment. Nothing too crazy, but
it's definitely not a controlled home environment. So if you get distracted
easily, consider getting earbuds or getting into the habit of blocking out
noise. If on the other hand, you prefer having a more controlled environment,
you can go with the online proctored option. The process will start with you
using your personal computer to do a system check before taking the exam.
Here you'll check that your webcam and microphone are working as needed.
Someone will be watching you throughout the exam, hence why you'll need
to have your mic and camera up and running. Afterwards, you'll need to
install the proctoring software, and this is how the exam is delivered. It works
on presenting the exam, as well as shutting down all the background
programs on your computer while the exam is running. Therefore, to ensure
that everything is up and running, log in early. You can actually launch the
online exam up to 30 minutes prior to your scheduled time. Another
important piece to be mindful of is the setup of your room. So during the
check-in process, you'll be asked to take pictures of the room, and the
proctor will also get you to pan your webcam around the room. Basically,
you'll need to show that you only have one monitor in the room. That your
workspace is free of clutter, especially on your desk, and that you're in a
room where the door closes, and if they spot something that they don't like,
they'll ask you to fix it. For example, one time I had a mirror upon the wall in
front of my desk, and the proctor asked me to take it down. Other things to
keep in mind is that you're not allowed to take breaks, including bathroom
breaks. You're also not allowed to stand up during the exam. Likewise,
external interruptions aren't allowed either. So if someone enters the room
during testing, it's an automatic fail. Therefore, if you have children or pets,
make sure to lock everyone out. And lastly, just like the in-person exam,
you'll need to have your ID ready. You'll be asked to take pictures of the front
and back of your ID. To present it to the proctor during the check-in process.
Now, to wrap up, let's go over some final exam steps. Ideally, try to schedule
the exam early. You can even schedule the exam when you start studying to
ensure that you'll be able to take it on your preferred date. If you're not
pressed with a deadline, obviously you can wait till after studying to book the
exam. Just keep in mind that you might not find a spot right away. I waited till
the last minute one time and ended up taking the exam on the same day as
my daughter's birthday. Everything ultimately worked out, but it would have
been less hectic had I booked the exam ahead of time. In addition to
studying, I highly, highly recommend taking practice tests as many times as
you can and just aim to score at least 80% a couple of times. I wouldn't be
exaggerating if I said that this is as important as studying, because you'll
need to train your brain on how certain topics are presented. Next step is to
get hands-on by following the demos and hands-on labs in our playground
environment. And lastly, to avoid getting overwhelmed, try to split your
certification goal into various mini goals. For instance, you can aim to
complete a certain number of modules or clips per week. And I would urge
you to try to get into the habit of celebrating each one of those goals.
Besides the fact that it's more rewarding for many people, this actually
motivates them to keep up the momentum. And that's about it for the exam.
Where to go from here? Well, AWS recommends heading to the AWS certified
Security Specialty certification to go to the next level as a data engineer. My
guess is this probably has to do with the fact that the data engineering
certificate doesn't focus on security topics, which is still a critical domain for
data engineers. I hope that this clip helped you figure out whether to take
this certificate or not, and if so, I'll see you inside the course.

Data Engineering Fundamentals

Data Engineering Fundamentals

[Autogenerated] Data engineering revolves around collecting, processing,

and analyzing large volumes of data, which is why it's important to have a
grasp of the different stages that data goes through. The first stage is
evidently identifying and understanding data sources. Data can be sourced
from various origins, including databases. Objects, these include files and
media files. Internet of Things such as sensors, wearable devices, and other
smart technologies that constantly collect and generate a stream of data. As
well as mobile applications. Data lakes store raw data in its native format,
allowing for high scalability and flexibility. The data is typically ingested from
source into a cloud environment such as AWS, where it's usually stored in
Amazon S3. AWS glue can then be used to transform the data which will also
be stored in Amazon S3. Glue achieves this by cataloging, cleaning, and
crawling the data to infer the schema if needed. From here, the data gets
published down the stream for further analysis and processing, where you
can have the data published to Amazon Quick site for business intelligence
and data visualization. Amazon Sage Maker for deploying machine learning
models, Amazon OpenSearch for search and analytics capabilities, and
Amazon Athena for running SQL queries directly against data in S3. And
these are the stages of data in data lakes. Now let's zoom out a bit and
compare data lakes against data warehouses. As shown in the previous
diagram, data lakes are centralized repos for storing raw data in its native
format. It can store structured, semi-structured, and even unstructured data.
This means that you don't need a predefined schema. The schema is only
applied once the data is read or processed. This is known as schema on
wheat. Therefore, data lakes can accommodate various types of data,
including text, images, and videos. In terms of cost, storage costs in data
lakes are fairly inexpensive. Data lakes are also less time consuming to
manage, which reduces operational costs. Having, having, having a data
warehouse on the other hand, is a centralized repo that stores historical data
from various sources. Yes, you can also achieve this with data lakes, but
where data warehouses really differ is that first off, they can only store
structured and organized data, so you can't use it to store unstructured data.
In other words, it requires a pre-defined schema because it operates on the
schema on white principle. This means that a pre-defined schema must be
defined before the data is stored. Therefore, data warehouses are best suited
for transactional systems, customer relationship management, and ERP
systems. And in terms of costs, in data warehouses, storage costs can be
higher, particularly for structured and processed data. Additionally,
maintaining data warehouses typically requires more intensive management
efforts, potentially leading to higher operational expenses. And that's it for
the fundamentals. I look forward to seeing you inside the courses.

Data Ingestion

Intro to AWS Glue

Hello, everybody, and welcome to this leg of the data Certification's
Engineering Associate learning path. In this module, we'll be specifically
looking at data ingestion. We'll be talking about the services that specifically
pop up on the exam that deals specifically with data ingestion. We'll also be
looking at APIs that shift that incoming data. Let's get started. In this clip,
we're going to be talking about AWS Glue. AWS Glue is an ingestion tool for
data. It is a serverless solution for discovering, prepping, and combining
data, allowing you to pull data from different sources to build catalogs of
data operations. Glue also has multiple capacities, allowing you to do quite a
few different tasks within one service, allowing you to do things like creating
an ETL job, gathering, storing, possessing, and cataloging data, all within one
service. It also allows for a lot of third-party services to interact within your
Glue environment and your AWS environment as a whole. Okay, let's break
down some of the things you can accomplish with a Glue ETL job. For ETL
jobs, you have a couple of options when it comes to building out those jobs
within AWS, in this particular case, we're going to be specifically looking into
Glue. Again, Glue is a serverless data integration, allowing you to pull data
from various sources and cataloging it. With Glue, you're allowed to pull from
a lot of different data sources, the most popular, and common, being S3,
DynamoDB, and RDS. You're also able to further customize your experience
with ETL jobs by using different methods, including the utilization of Python
shell for cheaper jobs, Glue Streaming, which allows for you to catalog data
that is streaming in real time with the use of things like Kinesis, or if you
need a more visual way to build out your jobs, you can definitely utilize Cloud
Studio's to see a GUI, or graphical user interface, to see how your job
progresses through the process. You also have a few other ways to customize
your jobs, including the type of processors for the workloads that you're
using. There are four main different types of DPUs, or data processing unit.
The first is the G.1X, they're generally mapped to one singular DPU. These
are mostly used for memory-intensive loads, but they're also slightly more
cost effective, as you're charged per DPU that you're using. And then we
have the G.2X mappings, which usually map across two different data
processing units. Because they're mapped over those two different units,
they're usually really good for high-intensity volumes, like machine learning
or extreme transfers of data. Then we also have the G.025X, which is
mapped to a quarter of a DPU. It's not really useful for anything high volume,
but it is also one of the cheapest options. Finally, we have the older,
standardized version, which allows you to utilize older versions of Glue, think
version 1.0 or earlier. This also allows you to allocate your DPUs when the job
runs, setting your maximum capacity. Now, let's dig into where that data is
after it's been processed and where it gets sent to, in this case, a data
catalog. Your data catalog allows you to manage your metadata and store
your data in general, allowing you to gain more information about your
datasets from your databases. You also get more detailed information about
tables being pulled into your catalog, this information includes columns, data
types, and a lot of other attributes that have to do with how many rows there
are, where your data comes from, and things along those lines. Another
great thing about a catalog is that you can automatically get information
about your database from the schema, allowing you to process your data
more quickly and effectively than manually mapping your schema from your
database to the catalog. Another aspect of Glue is the Studio. The Glue
Studio allows you to build out your ETL jobs and workflows, to provide you
with more visualization of that particular step. This allows you to incorporate
other services within those jobs and workflows, allowing you to process and
transform your data. It also allows you to scale your data processing with an
influx of data coming in to your ETL jobs. The two main things with Studio are
that it is a visual way to write ETL jobs and it helps you to orchestrate and
monitor your entire build. The visualization of build options means that you
can drag and drop your functions into the build to create the perfect job to fit
your needs. And when I say orchestration, this means you can automate the
whole data pipeline, utilizing the ETL job and pipelines information, allowing
you to do things like set them to run on a set schedule, be that specific times
or intervals, or it can respond to events within your environment, things like
data arrivals into a database, or data being uploaded into an S3 bucket.
DataBrew is a way to process your data visually. When you get your data to a
state to be processed, you can easily use SQL to process or sort your data. It
also allows you to get into the nitty gritty of your data, exploring and
interacting with your data to get more in-depth knowledge of what's going on
within your dataset as a whole, meaning you can really learn the details of
everything that's going on in that table. DataBrew also allows you to
transform your data with a lot of easy, built-in options to make sure your
data is as presentable and readable as possible. Realistically, what you need
to know for this exam is the basics between Studio and DataBrew, so let's
jump into that. Glue Studio and DataBrew are both visualization tools,
allowing you to see information and steps in how your data is both being
processed and the flow of it through your environment. The main benefits
are different when you're looking at these two services. For Studio, you have
everything you need to monitor any and all jobs that are built within Studio
and allow you to see what's happening within your jobs, while DataBrew is
more built around data profiling, meaning you can really find some patterns
and anything else going on within your information. The other thing you have
to consider is that the two main focuses are for these particular services.
Studio focuses on getting transformation and storing your data, while
DataBrew is much more focused on utilizing your data, allowing you to see
patterns, the odd tweaks within your data trends that make everything come
together. So, now that we have a basic grasp of what some of the service
actually does, let's talk about some of its use cases. Why would you use Glue
first and foremost? It is a quick-and-easy way for you to build and complete
ETL jobs for your data. You can also get that data into a catalog, meaning
you can get a more variety of data from a lot of different sources. But it also
helps with the next point, which means that you can get a more unique
perspective on your data processing and transformation to get your data into
a more presentable state because of how many different sources and
potential viewpoints you have on this particular data. The other thing we can
consider is the main benefits of Glue, so let's dig into the main four to focus
on for this exam. The first is that Glue is a fully-managed service, meaning
that you really only have to focus on how to build on the top level of the
service without having to worry about the workings of the underlying
hardware, maintenance, and lifespan. Glue also has a way of collecting data
from a plethora of different sources and services, meaning you can have a
much wider net when you're collecting your data. It also allows you to inject
data in basically real time, meaning you can consume, process, and buffer
that data without worrying about delays. Finally, you've also got basically
real-time metrics and reporting coming into the streams of data, giving you a
place to see your inbound data all in one place. So something else we need
to cover while we're looking at Glue as a whole are triggers. Your ETL job can
be triggered in response to a variety of different things. The first trigger that
we're going to be discussing in this particular clip is the event-driven trigger.
This trigger can really help with your automation, allowing your workflows
and ETL jobs to respond to events that are happening within your
environment. And when I say events, I mean that it can be just about
anything. Some of the examples for events in general are things like, a
picture being uploaded into S3, a new dataset has been injected into your
DynamoDB table, or even something as simple as a notification service topic
getting an incoming message. These events can also come from
EventBridge. But you don't really need to know much about EventBridge in
the context of this exam just be aware that EventBridge can also produce
these events to trigger your Glue environment. The next type of trigger is a
time-based trigger, this, as the name implies, means that you can schedule
your workflows and jobs. This can be at specified intervals, like every other
Tuesday or a particular date and time, like the last Wednesday of a quarter to
be able to pull the most relevant data for a quarterly report. To really work
within these types of triggers and setting these dates and times, you need to
be at least slightly familiar with the concept of Crontab notation. This
notation allows you to set customizable frequencies, meaning you're able to
set the time to the exact time that you need, down to the minute. This
frequency can have different benefits and can help you with your business
needs, depending on how you're mapping out your environment. And finally,
we have the on-demand trigger, which allows you to start your workflow
based on your need, without the automation components of the other two so
you will need to start this process manually. Having an on-demand trigger
means that you have complete control over when this workflow or ETL job
starts and stops, meaning you really only have to run it when you absolutely
need to. This is also really helpful if you have small changes within your data
or if you're trying to get a better idea of how often you need to set your
automation up. With that, we have completed this clip. I hope that you were
able to learn something, and I look forward to seeing you in the next one.

Let's Talk Kinesis

[Autogenerated] Hello everybody and welcome back to this learning path for
the data engineering exam in AWS. In this clip, we're going to be talking
about another service that's pretty big on the exam, and that is Kinesis.
Kinesis is primarily a data streaming service. This service allows you to
stream data in near real time, allowing you to ship that data to a point that
you set within your environment. There are 4 main services under the Kinesis
umbrella, the first being a data stream, which is kind of the first step in the
data collection process. The data stream gathers data from a specified point
and sends its streaming. The next step is the fire hose, which helps you
move that data from the data stream to a collection point within your
environment. You also have under this symbella of video streams, which
allows you to gain that data from video streaming services. Things like
security cameras, or even things like doorbells. These streams allow you to
store and easily analyze this video information that's coming through in
basically real time. And finally, we have the managed Apache flink. Which,
while it doesn't have kinesis in its name, it does still settle under that same
umbrella. Flink is a way to analyze this data combined with Zeppelin
notebooks from Apache, meaning that you can use SQL to dig deeper into
the data that you streamed into it. It also helps to transform your data and
structure it into nice neat tables. With these services, you're able to collect,
deliver, and analyze any data that gets pushed into that stream. You're also
able to interact with AWS services in a lot of different third party services
with only a minimal amount of effort. Let's talk about some of the services
under that umbrella that we just brushed past a minute ago. The first of
those services that we're going to talk about is Kinesis Data streams. Data
streams are a serverless way to stream your data from your source to a
storage place. Allowing you to go from incoming data into a storage. These
streams are made up of a group of shards that hold the data records. A data
record contains a sequence number which acts like an identifier, a partition
key which allows you to group your data shards together. And then the data
blob, which is the actual data within the shards. Within these streams there
are really two sides. You have the data producers. And then you have the
data consumers. The data producers are things like applications, services,
incoming streams. Anything that allows your data to come into your stream.
The consumer is on the opposite side of the stream, allowing you to take that
information from those applications, services, or streams, and help process it
or store it. So, let's take a look at the data flow within that stream. Like we
talked about in the last slide, you have a data producer, which can be a lot of
different things, be it in Internet of Things device or servers or container, that
data is getting pushed into kinesis. It will hit that first data stream, flow down
the stream, and come into a fire hose. In that fire hose, we'll send it to a data
consumer. Which the most popular services are things like S3, Redshift, and
some third party software. Data fire hoses are a delivery stream, meaning
that you gain data from the data stream and deliver that stream somewhere
else. These streams allow you to take data that is streamed and send it to a
destination. This also allows for things like monitoring the flow of data that's
coming through the stream and the fire hose, and confirming that your data
has successfully gotten to its destination. Sending this data allows you to get
it cataloged or stored in AWS or third party services. You can also send it to
services that allow you to process and potentially transform that data, such
as A patchy fling. Apache flink really allows you to dig deep into your data.
Allowing you to process that data with programming languages, SQL
commands, or some other processes to allow you to transform your data or
just to get it settled. Flink allows you to create notebooks using Apache
Zeppelin and build out the notebook to run specific commands against your
data. It also allows you to monitor the changes within that processing period
as you're going through your notebook. So let's talk about some of the
common uses for kinesis as a whole. One of the first things that it's normally
used for is anomaly detection. Take for example, Internet of Things devices.
Internet of Things devices can be kind of finicky. You need to keep track of
any changes that happen within these devices or any security blips that
might pop up. It also really helps with log production and processing for
machine learning applications. As these types of applications can change
insanely quickly, being able to monitor these logs in real time can be
incredibly helpful to help with course correction when your machine learning
has a tendency to go different directions than you expected. Kinesis also
helps with pattern detection within your data. Meaning you can actually find
changes incredibly quickly and are able to shift priority in your data captures.
And finally, of course, we also have click stream analysis, meaning you're
really able to keep track of where and what your people are clicking on on
sites to help curate your site for your users. Now, let's talk about benefits.
Kinesis has 4 main benefits that you need to keep in mind for this exam. The
first being that it's managed by AWS, meaning AWS once again handles the
underlying hardware and you're just focused on the top levels of application
management and service management. It's also serverless, meaning that
you don't have to actively consider server restrictions when you're building
out your data streams. It also allows for basically real-time data streaming,
meaning you can easily keep track of your data and know the changes that
are happening within your streams. Finally, you have actually built in auto
scaling, meaning that as your processes grow and shrink, they will
automatically build out quickly and efficiently. So one of the other things that
you need to be aware of for this exam. Is the difference between the four
main kinesis family services and SUS or simple Queervices. These two
services have their benefits, the first thing being the data retention period.
For kinesis, data is stored for 24 hours by default, but can be stored up to 7
days, while SQS has a default period of 1 minute, but it can be adjusted and
information can be stored up to 14 days. For Kinesis, you can have multiple
consumers for the same data record. While SQS you only have a singular
consumer per message, as once it is read, it gets removed from the queue.
Kinesis's workflow process is that it processes the shards that come into its
streams in the order of the shards, not necessarily the order of the shards
themselves. To alliterate on that a little bit more, what I mean is that your
shards can come in in a couple of different functions and formats. And
although we have things like the identifier and the partition key to try and
keep everything grouped together as much as possible, that doesn't
necessarily mean that things won't come a little bit out of order if data cops
up more quickly in one area than the other. A normal SQSQ also doesn't
guarantee the processing messages in order unless you build it out
specifically as a first in, first out style of queue. And finally, you have the
processing of the shards, you can have related records within those shards
sent to the same processor. Well, SQS doesn't really support that. If you have
all of these data records that have data blobs that are on the same topic,
these shards can usually be sent to the same processor to allow for quicker
and faster uploads. SQS doesn't really support that as an option. One of the
other comparable services that you need to take into account for the
differences in general are the differences between Kinesis and Apache Kafka.
We'll go into a little bit more detail and what Apache Kafka is in a later video.
Just be aware that these are some of the main differences between these two
services at this time. One of the things that you need to really consider is
that the configuration process between these two services is different.
Configurability within kinesis is limited. The most configuration you can
realistically do is where to write the data at the end of the resources. where
with CAFTA, you get a lot more wriggle room on the configuration of how the
data gets written. Things like what type of file type it gets saved in, where it
gets saved, how it gets saved, all of that good stuff. That's a little bit more
configurable than you would get with Kinesis. However, when we're talking
about the overarching cost of either of these services, you need to
remember that Kinesis is an AWS service, meaning that it is cloud native.
Meaning that like most cloud services, it is a pay as you go model and has a
more compartmentalized pay structure. While Kafka is a free to use software,
you do usually have a higher overarching cost, as it requires more active
engineering hours to manage it correctly. Though once you get it managed
and correctly set up, it's usually good to just sit and run. And finally, there's
the fact that once again Kinesis is a cloud software. So that means that it
was designed for quick and easy implementation and management. Well,
Kafka places you and your engineers to do all of the heavy lifting when it
comes to implementation, maintenances and overarching management in
general. With that, we've completed this clip. I hope you were able to learn
something, and I look forward to seeing you in the next one.

Intro to Amazon RedShift

Hello, everybody and welcome back to the prep course for the AWS
engineering exam. In this clip, we're going to be talking about the AWS
service Redshift. So, what exactly is Redshift? Redshift is a fully-managed
data warehousing solution based on PostgreSQL. It has a lot of unique and
powerful features, making it a great solution for OLAP workloads. It supports
distributed workloads, as it's capable of processing data across multiple
nodes, and we'll see exactly how that works shortly. It also distributes these
workloads by storing data in columns instead of in rows. In terms of querying
capabilities, it offers the Redshift Query Editor, which is a web-based SQL
tool, allowing you to write and execute SQL queries, either directly in the
console or through the API. It also allows you to query petabytes of data. This
includes both structured data and semi-structured data. It can be deployed in
specific VPCs that have specific security and networking options to make
sure that it is completely isolated within that network. And, just like RDS,
Aurora, and DynamoDB, Redshift, too, also supports multi-availability zone
deployments. It's also offering a severless option, meaning that you can do a
lot more with it without having to be as quite as fiddly as some of the other
options. All right, now let's talk about some of the ways that you utilize
Redshift in the context of this exam. Redshift is a way for you to collect data
in a traditional data warehouse. This type of storage is like having a central
repository for your information. Data flows into the warehouse from a
transactional system, relevant databases, as well as other data sources. Data
warehousing also allows you to put together multiple data sources, so that
way you can run queries against all of your data, not just small groupings of
them, allowing you to save a lot of time and resources to not run repetitive
commands over and over and over again. Redshift also allows you to utilize
Log Analysis, meaning you can gain more information and data from your
logs. It also allows you to see more in-depth things like click-stream analytics
and hardware associations with these reports. In terms of architecture,
Redshift uses a cluster architecture, whereby each cluster is composed of
computing resources called nodes. They look kind of like this. You can have
up to 128 nodes per cluster. There are realistically two types of nodes within
these clusters, you have the leader node and a compute node. If you're
working in a singular-node cluster, you will have a singular compute node
and a singular leader node, and this is typically suited for small datasets or if
you're only deploying the cluster for testing purposes. Now, on the other
hand, if you're deploying a multi-node cluster, your cluster will consist of at
least two compute nodes, which is more suited for larger datasets or
production workloads. Okay, so now let's unpack the nodes a little bit more.
Let's look at the leader node first to take a look at what its responsibilities
really are. Okay, so these nodes are in charge of receiving queries from client
applications and parsing these queries and developing execution plans,
including execution plans that execute queries in parallel. Lastly, they
aggregate the results of these queries and return the results back to the
client. On the other hand, we have compute notes, and they have obviously
different responsibilities. They're in charge of partitioning the data into
smaller portions called slices, whereby each slice gets assigned a compute
node to achieve parallel processing. This processing is also known as slice
management. Compute nodes run the execution plans that the leader nodes
create and work on sending the intermediate data results back to the leader
node once they have gathered their data. And if we zoom out a bit, we can
see that the leader node is used to communicate with a client. And you can
see two lines here because realistically you either have a JDBC connection if
you're connecting using a Java-based application or an ODBC connection if
you're working with another API source. From here, the leader node
automatically distributes that data and query load across all the compute
nodes. And like we said, those compute nodes partition jobs into slices and
execute the plan. Once they're done, they send it back to the leader. The
leader then aggregates that results and sends it back to the client. This
process where work is distributed and data is separated into parallels across
multiple compute resources takes place under the umbrella of
massively-parallel processing, or MPP. So let's discuss some of the other
different types of nodes. Some of the more common node types include the
Dense Compute, or DC node, which is designed for demanding workloads
that need high-compute capacity, hence the name. And these are great for
workloads that are heavy on complex analytics and demand real-time
responses. The second type is a Dense Storage, or DS node, which are
designed to serve as large data warehouses. They are optimized for storage
intensive workloads, however, they typically offer slower performance, which
is why AWS recommends using RA3 nodes instead0. RA3 is distinguished by
its ability to scale, compute, and storage independently. It can automatically
offload data into S3 if a node grows beyond its capacity of a local SSD. And,
generally speaking, AWS recommends using this one over the Dense Storage
node models, as it is it a little bit more flexible with cloud in general. Okay,
so let's talk about the structure and preparation of a staging table. As you
can see, this is, well, a rather large block of code, so let's break this down
into chunks. To start off, we have the insert command, pushing data, in this
case, to the orders_summary table. These next few lines are information
about that data, customer key, order status, price, and so on. We then get
into the sorting section of this code, where we essentially rank the total price
interaction with a threshold. After that, it is splitting the table per order of
priority. If you have more important orders or what have you, it would split it
depending on what you need for it. And then, after that, we see the order is
being taken from the orders_staging table. So, let's talk about some scaling
options when it comes to Redshift. What happens when you need more
information, or if you find that your data warehousing options aren't nearly
as robust as you want them to be? You have a couple of options when it
comes to scaling outward. The first thing is you can add more Redshift
clusters, also referred to as concurrency scaling, because it allows you to
serve sudden spikes in concurrent read requests, thanks to Redshift's ability
to run queries in parallel. If you enable this feature, Redshift will
automatically add more compute power temporarily to meet the demand of
your high concurrency. Another scaling option is to query data via S3 using
the Redshift Spectrum functions, which enables you to query large volumes
of data in S3 without having to load the data into Redshift. In other words, it
allows you to jump to querying data without having to wait for the extract
transformation, load jobs, or ETL jobs to maintain access to that data. And
lastly, you can play with your clusters by scaling them either horizontally or
vertically. For instance, if you were horizontally scaling, you would scale in
and out by adding and removing nodes from the cluster. And for vertical
scaling, you can scale up or down by changing your node types. Realistically,
there are two ways to resize your clusters, the first is via what is known as
Elastic Resize. The second is referred to as Classic Resize. With a Classic
Resize, the operation copies tables and metadata from a source cluster to a
newly created table cluster. Now, what's the purpose of doing this? Well, one,
it allows you to increase and decrease the size of the cluster by adding or
removing nodes to the target cluster. But, keep in mind that the source
cluster goes into read-only mode during these resize operations. If this is
your source cluster, and this is your newly resized target cluster, the source
cluster will go into read only as soon as you start the resize operations. And
the next thing that you can achieve with a Classic Resize is changing the
node types of the cluster. And because this process copies all the data to the
new cluster, it ends up taking a decent more time to do this process. Lastly,
it's worth noting that this process doesn't retain the system log table of the
original cluster. However, things like user schemas, user data, user tables, all
of that gets copied from the source cluster to the target cluster. Now, let's
talk about Elastic Resize, and the main differences between the Elastic
Resize and Classic Resize. Well, let's first take a look at what happens when
you're resizing an existing cluster. It follows a similar approach in the sense
that you can add or remove nodes, but it does so via an in-place resizing
process, whereby nodes are added or removed in place of the existing
cluster. Therefore, the cluster becomes unavailable when you are adding or
removing the nodes during an Elastic Resize. On the other hand, if you
attempt to resize by upgrading the node type, a new cluster will be created.
So, let's say you have a source cluster and you initiate an Elastic Resize,
whereby you're upgrading the node types. Redshift will create a cluster
snapshot and use it to restore it to a new resize cluster. Similar to the Classic
Resize, the source cluster will be in read-only during this operation. And keep
in mind, you can only do this by upgrading the node types within the same
cluster type. Also, Elastic Resizing doesn't support downgrading node types.
But, in exchange to these limitations, it is much faster than Classic Resizing,
typically completing within minutes, all while retaining your system logs,
therefore, it's usually recommended as the first option unless you're thinking
that you might have to change the cluster type or downgrade your node
types. And with that, we have completed this clip. I hope you were able to
learn something from this, and I look forward to seeing you in the next one.

Loading Data to Redshift: Commands

Hello, everybody, and welcome back to this learning path for this AWS exam
on data engineering. In this particular clip, we're going to be talking about
how to load data into Redshift. And in order to talk about that, we really need
to know where that data can come from. In terms of loading data, Redshift
can load data from a variety of AWS services, including, but not limited to,
S3. AWS Kinesis, EMR, DynamoDB, DMS, and RDS. So realistically, you can
load data into Redshift using what's known as COPY command and these
COPY commands can really be utilized in one of two different ways. Option
one is to load data from S3 into Redshift, and how that essentially works is
you have a data source, you load that data from the data source into S3, and
you follow a COPY command from S3 into your Redshift. And being from S3,
you're able to easily use said COPY commands to move that data into
Redshift. And then, of course, you have the second option, and option two is
to load directly from your source database into Redshift. So, for instance,
let's say you had a data source of DynamoDB, you could use a COPY
command to copy data from there directly into Redshift without hopping over
S3. That hop into S3 can be useful, as having a secondary set of data can be
helpful in things like disaster preparedness and making sure that you always
have a copy of it somewhere. Now, let's go over the steps needed for option
one. First, you need a Redshift cluster. After that, you need to split the files
that you wish to upload into multiple different files. This enables the COPY
command to leverage that MPP architecture that we learned about in the
previous lesson and load data in parallels. Then, ultimately, upload the file
into S3. Once the file has been uploaded, you can then create an IAM role,
assign S3 read permissions to that role, and attach that role to the cluster.
And with the IAM role out of the way, you can finally use your copy command
to load data from S3. Now, let's look over what a direct COPY command
would look like. In reality, it would look something like this, and you need
these three blocks of information to really complete that command. But, let's
dig into what these three lines of text actually mean. In the first line of this
command, we're identifying the target table, that's going to be the
destination within Redshift that we're going to be copying the data to. And
then the second line here, which says from, is going to be where we identify
our source bucket, as well as where we're taking the data from in general.
And, of course, to perform any of these operations, we need to have the
necessary permissions to do so. So, this line here is providing an ARN for an
IAM role that is granting us the necessary permissions to read that file within
S3. And so that is the basics of what a COPY command would look like if
you're copying data into Redshift. But what if you wanted to achieve the
opposite? Let's say you need to export data to S, well, that's where the
UNLOAD command comes in, which looks something like this. In the first line,
we specify the query to retrieve the data from our Redshift table, which is
going to act like the source in this particular scenario. The second line here,
we specify our target destination to determine which S3 bucket to send the
results of that query to. And similar to the last command, we're provisioning
our IAM role to make sure that we have the necessary permissions to
complete this command. Now, you also get an extra line here that we didn't
have in the last one, which is this phrase here, which is FORMAT PARQUET,
which is specifying the target file format that we are saving into S3. So, one
of the other commands that we really need to be aware of in the case of this
exam is going to be the VACUUM command, which I know sounds a little silly,
but it does have a very useful process within the Redshift sphere. And
VACUUM does a lot of like what it says on the tin, it reclaims space within a
table that a command is looking at. In this first command here, we see a full
vacuum of the sales table. Now, this method can take a little bit longer than
some of the other options we will be looking into, and it requires more space,
as it locks the table down and makes a full copy of that table, and doesn't
really seal the old one until it's done. But this will end with a smaller
database file overall. You can also set the thresholds to how much of the file
is reclaimed or dealt with. So, in this particular case, as you see on the
screen, you see VACUUM FULL sales, sales being the identifying table that
we're going to vacuum within Redshift. So the next variant in the VACUUM
command that we're talking about is the sort function. The sort function
allows you to resort your rows within a database, assuming that a specified
amount is already been sorted. So, in the case of this example here, VACUUM
sort only, meaning we're not reclaiming space, we're just sorting out what
was already here, again, on the sales table to 74 percent. So, assuming that
your sales table hasn't been sorted yet or hasn't been sorted recently, you
can make sure that it will sort up to 75% of that table to make sure that all of
your data is smoothly inside of the database. Now, finally, you have
reindexing. Reindexing allows you to potentially gain more space and have
your queries be more effective because reindexing allows you to combine
with vacuuming to reclaim space and make your overarching Redshift cluster
more effective, in general. But reindexing also allows you to make sure that
nothing has fallen into places that it shouldn't within your database. Again,
just like the structure we had in some of the other commands, you have the
actual command that we're using, VACUUM in this case, the variable, which
is the command that we're going to be also using inside of vacuum, which is
reindex, and the table that we're using this particular command on, in this
particular case listing. With that, you have completed this clip. I hope you
have learned something, and I look forward to seeing you in the next one.

Intro to AWS Lambda

Hello, everybody, and welcome back to this jaunt for the learning path of the
exam prep for data engineering in AWS. In this clip, we're going to be going
over Lambda. The service is touched on in the exam, and there will be some
basics that you really need to get down in order to gain some of those
much-needed points, so let's start with the basics. Lambda is split into two
different parts and both those parts work well together. There is the Lambda
function, the function is the code and workflow that you can prep to
complete tasks when events happen that match the trigger that you set
within the function. And then you have the Lambda service, the service is
what allows you to handle those incoming calls and is the underlying process
of the functions that are happening. They also handle the routing of the
inputs and outputs from the functions themselves. So, let's talk about some
popular use cases when it comes to utilizing Lambda. One of the most
common is for IOT devices, or Internet of Things. You could easily keep track
of changes that are happening within your devices or within your
environments using Lambda in this particular way. The triggers can be
anything from changes within the data stream to new devices being added to
the stream, allowing you to gain and process logs quickly and effectively. You
can also have serverless websites, which can have triggers on things for like
click-stream analysis, changes in traffic, but you can also have more in-depth
information, how traffic flows on your site, or where exactly people are
looking. You also, of course, have event-driven architecture that is utilized
within Lambda itself. This event-driven architecture means that whenever a
trigger has happened within Lambda causing one of the functions to go off, it
will create an automatic response without having to deal with a person
having to go and interact with that event. And thankfully, because of the
multitude of triggers that Lambda supports, this event can be just about
anything. So, let's do the breakdown of some of the benefits of Lambda. One
of the first things that we need to consider is the fact that Lambda is an
AWS-managed service, meaning that you can really put your focus in on the
code, workflows, and triggers without having to worry about the nitty-gritty
undertones of how much processing power do we need in order to fire off this
function. Amazon handles that for you, you don't have to think about it. The
other thing to be aware of is that Amazon is in charge of things like your
security when it comes to your Lambda functions and the service as a whole,
meaning that you do have to play by Amazon the rules when it comes to
things like having a trigger react into a certain action within another service
in AWS. So, in order to delete this particular action, you need to make sure
that your ducks are in a row, make sure that your IAM role has access to
wherever you're sending your function or information to. This overall process
is designed to keep your environment safer, as well as keep Amazon's
infrastructure more secure. The last thing is, much like a lot of cloud
software's, this is a pay-as-you-go model since you aren't paying for things
like electricity or things like that, you're just paying for specific runs of the
code. The other thing to be aware of is you get the first million requests free
per month, even when you're out of your trial period, that is something to
consider, meaning that you can have a good balance of the number of
events that you can process while still fine tuning your architecture within
your environment and have it be essentially free. When we dig below the
surface, you do get a few more benefits, like the extremely easy way it is to
build that responsive environment to those triggers and events. You also
have six different languages supported in runtimes so that you were able to
build out your code, allowing you a lot of different customization options. You
also get to tweak your code within the browser itself, meaning you can run
your code, see it has a flaw, or isn't working quite the way you thought it
would, make a tweak, redeploy, and you'll have your new codebase. And let's
say you made a mistake with some of those tweaking and you want to roll it
back to a last generation, because of the fact that you have constant control
over what persistent changes happen within your code and what persistent
changes are deployed within your Lambda function, you can easily roll back
your function so that way you can have an earlier version of your code. High
level, this is what you're looking at when you're building out a workflow.
When you're in the AWS console, you can jump into the Lambda section of
that console and create a function. This process can be done by scratch by
selecting the individual pieces and parts, and then causing the CREATE
function from within the portal. You can upload a blueprint, which is your
code and configuration presets all bundled together, or you can use a
container image that can deploy your function from there. Now, if you're
building from scratch, you do need to set all of your variables, things like
name, runtime language, architecture, all of that good stuff. For all of these
options, you also need to have an executable role that has the permissions
to collect the information coming in and to send it to where it's going. After
you create a function, you can either manually write your code within the
console or you can upload your code in a bundle, be it a ZIP or DMG function,
either one of those will work. This is really the meat of what causes your
functions to run, it is the action that is taken once the trigger is hit. And
finally, you have to specify a triggering event. This can be anything that's
potentially happening in your environment, as there are a lot of different
services that work within Lambda to actually go with this workflow. So, now
that we've talked about the process of creating a function, which will give us
a unique ARN to process that information, let's talk about the actual
requirements needed to run a Lambda service. With Lambda, you have a few
requirements to actually run the service as a whole, the first and foremost
being that you need a Lambda function, or the backbone of the event
response. You also need an IAM role, or a Identity Access Management
Execution role, meaning that you have to have a role inside of AWS with the
correct permissions to run both your function and has access to the services
that you'll be utilizing. The next thing that you need to remember is that you
need to specify your memory size, meaning that when you build out your
function, you can dictate how much memory you're going to be utilizing.
Now, this is also an editable value under your configuration settings within
the console itself, and the settings are always flexible between 128 MB and
10,240 MB. The next thing that you need to be aware of and take into
consideration is execution timeouts. An execution timeout is specifically how
long your function will wait before it will give up on that programmed action.
The limit within AWS when it comes to Lambda is 15 minutes. The last limit
we really need to go over is event-source mapping, or triggering for your
function. This is how you invoke your Lambda function to respond to events
that are happening within a service or services that you have designated for
the function to watch. So, let's go to the whiteboard for this. This is the
basics of what your environment looks like, you have an S3 bucket, a
Lambda function, and a DynamoDB table, and they're all happily sitting in
within your environment, just relaxing and waiting for input. And then we
have Steve. Steve has some information that he needs to upload into S3, for
both collaboration on a project and to make sure the file is safe. Now, once
Steve has uploaded that file into S3, it will trigger a Lambda function. Now,
this isn't the only way to trigger our Lambda function, Betsy can also trigger
it by utilizing the ARN of that function, either trigger method works. And once
the trigger is fired off, the function itself will happen, which, in this case is
writing to a DynamoDB table, taking stock of that new file currently within
the S3 bucket. This is the basics of how a Lambda function happens and
really all you need to know when it comes to the exam. With that, we have
completed this clip. I hope that you've gained something from this lesson,
and I look forward to seeing you in the next one.

Triggering Data Ingestion

Hello, everybody, and welcome back to the test prep for the Data
Engineering Associates exam for AWS. In this clip, we're going to be talking
about triggers. So, the first major question you're probably going to ask what
is trigger and in what context? A trigger in the context of this exam is a way
to allow resources to access other AWS services when specific actions are
taken or other things happen. When an event happens within your
environment, you are able to cause actions to happen within other resources,
assuming you have your triggers and events set up correctly. In this clip,
we'll be looking at specifically S3 events, EventBridge, and Redshift Events.
All right, let's talk about S3 event triggers. These events are seen as rules
within S3, allowing for actions to happen when conditions are met. For
example, a triggering event could be a file has been removed from a folder
or a new folder has been created. You can use these events to trigger all
sorts of other actions within your environment. New files being uploaded into
S3 could kick off a Lambda function to process that new file and make sure
that it's saving space within your S3 bucket, or a file being removed could
set off an SNS Proxy to make sure that notifications are sent out that there
has been a change within your S3 environment. These rules can push events
to a lot of different services, both on AWS and through third parties. And now
we're going to touch on EventBridge triggers. So, for EventBridge triggers,
you can set a number of different options for what these triggers can be.
This, much like S3, can be done in either a rule that you have in place or an
event pattern that happens in your environment. If an event matches the
pattern definition, it will send that event to the target, which triggers an
action, usually, that is specified by the rules. If something happens within
your environment, say an EC2 alarm goes off within your environment, with
these triggers in place, it can send out another alert to an AWS service, say,
SES, to make sure an extra email gets sent out to the admin team to know
exactly what's going on within the environment. So, the last thing that we're
going to talk about is Redshift events. These events are a log of everything
that happens or even changes within the cluster, new data being uploaded,
changes to tables, or changes to the database as a whole. Whenever an
event happens within Redshift, you get notifications from the subscriptions
that you personally are subscribed to or have been set to depending on your
environment, and these logs can be held for weeks at a time. These events
can be incredibly detailed, including date, description, where the event is
coming from, and the source ID. All right, let's talk about the three main
components that come with creating an event-driven architecture. And these
three services that we're going to be looking at are Kinesis, EventBridge, and
the Lamba. For Kinesis, you're looking at Data Streams. These Data Streams
can be configured to act as a trigger, being able to push data from the
source to a singular place, which can trigger a number of different services,
even services like Lambda. But what you need to remember is that these are
mapped to usually a singular destination. You also have EventBridge, where
events can follow specific rules and patterns, and when they match those
patterns or rules, they can act as triggers for one or multiple destinations,
meaning that these events and triggers can cause multiple services to react
at once. And finally, we have Lambda, which is honestly the most flexible of
the three. You have the most options when it comes to trigger actions from a
number of different services and actions that will allow you to have a lot of
customizable options when it comes to all the events that happen and allow
to be mapped to a service. Each of these particular services is useful in their
own way and can be very handy in building out your event-driven
architecture. There isn't necessarily to say one of these services is better
than the other, it's mostly that they're utilized for different specific things
and can do things that the others can't. With that, we have completed this
clip. I hope you have been able to learn something, and I look forward to
seeing you in the next one.

Consuming Data APIs

Hello, everybody, and welcome back to your trip down the learning path for
the AWS Data Engineering Associates exam. In this clip, we're going to be
talking about consuming data via APIs. First and foremost, let's talk about
what an API is and what it stands for. API stands for application programming
interface, which is what allows a server and an application to talk to each
other, with loops that look something like this, sending a request to the
server and the opposite side sending a response back to your client. In this
case, the application is sending a request to the App Server for information,
and then sending a response back from the App Server to the application to
allow work to continue. So, now that we have an idea of how the call and
response for API works, let's look at the four main APIs that we'll be
examining in this particular clip. The first we're going to be talking about is
SOAP APIs, then RPC APIs, a WebSocket API, and finally, the REST API. So,
let's go a little bit deeper in what each of these API types means. First, we're
going to consider the SOAP API, which SOAP stands for Simple Object Access
Protocol, which realistically is the standard messaging protocol that utilizes
HTML and XML messages, allowing easier communications between the
client and server. Then we have the RPC API, which stands for Remote
Procedure Calls. This is a protocol that also provides communication systems.
However, it's going to be utilizing an operating system rather than
router-specific messaging protocols. In other words, let's say you have an
application client and that client requests in action to happen within a server,
and that server sends that output from the action back to the client. This is
an example of what an RPC API call looks like. And now, we've got the
WebSocket API, which is a way to send communications between a web
browser to a server. This API allows the messages to be sent to a server and
receive event-driven responses without having to pull the server for multiple
replies. And then we have the REST API, which actually does stand for
Representational State Transfer. This type of API is a flexible and lightweight
way of bringing together applications and connecting them to components to
other services within your architecture. So, let's talk about the Data
Exchange really quick, as that's one of AWS's main API tools and something
that might pop up on the exam. As the name suggests, the Data Exchange
API is a way for you to move data between services, allowing you to set up
subscriptions that access that data and information. You also use the API
options to export or copy that dataset to S3, and then you can use that data
with a variety of different analytic tools. The Data Exchange API really has
two different functions. When you are the controller or provider for this
information or data, you're allowed to curate what data gets pushed out to
the subscribers of this particular dataset. So, let's break down the Data
Exchange API into parts. There are two main parts of the API setup. You start
with your dataset, which is a collection of data that changes over time, be it
through updates, growth, or changes, or maybe just revisions, be it with a
new version or just incremental changes within that dataset, all of it counts.
You then have the asset, which boils down to a piece of data, which can be a
number of different things, be it a AWS Data Lake formation, structured data,
or image file that would be short on S3. It could also be a Redshift data
share, any of these are an option of what an asset would look like within the
Data Exchange API. But speaking of Redshift in Data Exchange API, which is a
way to access your Redshift database, you are able to access this data from
using a web-based service application, services like Lambda, SageMaker,
Notebooks, Cloud9. Thankfully, you don't need to have a persistent
connection to your database to have the data flow, instead you get a secure
HTTP endpoint to work with your SDK, or Software Development Kit, with
these endpoints. You can run SQL commands without having to manage the
connections. Another note that isn't really mentioned in the exam, but might
be useful in your real-world experience, these data APIs have to use
credentials. These credentials can either be stored in Secret Manager or by
temporary credentials. However, they are not enterable passwords you need
to remember that in case you are ever flowing through this process. So, let's
talk about the troubleshooting thought process when it comes to the Redshift
data API. The first thing you need to make sure is that the caller, or
requester, is able to actually make a call to that dataset, both by ability and
authorization. The next thing to consider is that you need to determine if you
have credentials to get into that dataset. That database's credentials could
be, again, stored through temporary measures or with Secrets Manager.
Generally speaking, it is always good to try and complete best practices
when it comes to security and keeping your data together. In this case, it's
considered a best practice to have your Secrets Manager set up to hold your
credentials, the encryption, generally speaking, is better on these particular
cases. And if you're having problems connecting to your database using the
API, your best bet is to start from the bottom of the stack, things like check
your IAM permissions, your HTTP and your SDK endpoints are pointed in the
correct places in order for you to correctly connect to them. Starting out with
these types of troubleshooting ideas in mind is always useful. Remember,
start at the bottom and work your way back up. And finally, the thing to
consider is when you're calling the API using the command-line interface or
AWS's CLI from code, or if you're using the Query Editor from Redshift within
the console. Both of these options are valid, they just have different
challenges depending on what you're looking at. Users with CLI access might
have a slightly easier time moving their data than if you're just doing it
through the console, though both of these options are valid. The last thing
that we're going to talk about in this particular clip is going to be the Lambda
API. Much like we saw in our previous clip on Lambda, the Lambda API is also
serverless, meaning low maintenance, high customization. In the Lamba API,
you are able to utilize API gateways and API different status types, which
support a lot of different functions, things like routing your data, serving
HTMI files, or even storing binary files. This also helps to control traffic and
flow through your APIs, allowing you to have more things like redirects. With
that, we have completed this clip. I hope that you were able to learn
something from this, and I look forward to seeing you in the next one.

Security and Connections

Hello, everybody, and welcome back to this leg of the data engineering exam
for AWS. In this clip, we're going to be talking about the basics in network
security, as well as some RDS protections when it comes to AWS, along with
some best practices for both of these topics. There is another module on
security later in the learning path, but these are just some of the basics that
you need to know before going into that leg, as well as what you need to
know for data ingestion, in general. So let's get started. So, one of the main
security features within your account overall is going to be what we call
NACL's, or Network Access Control Lists. NACL's are a stateless firewall.
Stateless, meaning that they inspect all traffic, regardless of things like
direction of the traffic or if that particular packet is part of a pre-existing
approved connection. They inspect everything, which is great because it
adds in the layer of protection for your environment as a whole. However, it
can cause issues if you don't account for certain types of traffic coming into
your environment. Speaking of that, NACL's allow you to approve or deny
traffic on a number of different protocols. They also reference things like port
ranges and allow you to flag specific types of traffic that's coming into your
environment. So, underneath the layer of the NACL, you also have security
groups. Security groups are another way for you to filter out the particular
traffic that's coming into your VPC. The thing about security groups is that
they are stateful, and a stateful firewall means that security groups are
inspecting packets and traffics with contacts that you've built into it, allowing
you to have a much more complex rule structure, as well as log more
detailed traffic events that are happening within your environment. Because
of this, security groups have a little bit more wiggle room when it comes to
building out your environment and dictating what comes and goes inside of
your environment. Security groups allow you to block or allow traffic via IP
address, blocks of IP addresses, other security groups within your account, or
even running services within your account. All of this is editable inside of the
AWS console, specifically under the EC2 section of the console. So now that
we've talked about the two main pieces of security function when it comes to
basic security in your environment, let's take a look at how that looks
architecturally. Whenever you're building out your architecture in any of
these particular contexts, you first start with your empty shell, which is your
VPC. This shell is how you build out your environment as a whole. Within the
shell, you will have your subnets, and those subnets will contain your
resources, and we'll help you sort what type of traffic you'll be allowing to
access those resources inside of those subnets. But on top of your subnets,
you're also dealing with security groups and your NACL's. Just like you can
see here, you can have multiple security groups within your environment,
and you can have them so that they are overlapping or not. As you can see
in this example, our private security group is isolated, meaning it doesn't
have access to the outside world, at least from what we can see in the
diagram, whereas the public subnet does have a security group and is
attached to the NACL's that are there, but has access to the outside world
through those NACL's and security groups touching the outside spaces. And,
as you can see in this example, your NACL and security group both inter
work together to help monitor and approve or evoke traffic going into your
environment. It is always important to remember that when you are doing
things like troubleshooting, if somebody has an issue of being able to
connect into your environment, you need to remember to always start from
the bottom. Start by looking at things like your NACL's to make sure that
something didn't occur that caused them to be denied, and then move up to
your security groups to check to see if there's something that's not approved
for their particular access. Okay, so let's talk about some of the best
practices when it comes to RDS security. In the last diagram, we saw the two
different subnets. Generally speaking, it's considered best practice to isolate
your RDS instances inside of a private subnet, meaning that the internet
doesn't really have access to it, making it harder to do things like DDoS
attacking, or time-out attacking. With RDS instance being in a private
subnet, does not necessarily mean that it does not have access to other
internalized resources, it just means that you'll have to map it out a little bit
differently using things like NAT gateways or endpoints that you're able to
connect into. But let's also talk about some of the best practices you really
need to be aware of for the sake of this particular exam. When you're putting
a security group in place, you really need to control what IP addresses they
can access it to and from, making sure that your developers are only coming
from a singular IP or a singular block of IPs to make sure that we don't have
people accessing it that are not supposed to. Generally speaking, it's also a
good idea to limit the resources that share that particular subnet within your
VPC. That way, you can always be aware and monitor the traffic that is
coming into your subnet. And finally, always allow your NACL's to really strip
down what traffic can come into your database, making sure you really only
have what you need to access it. With that, we finished this clip. I hope you
were able to pick up something from it and I look forward to seeing you in
the next one.

Data Transformation

ETL Pipelines and You!

Welcome back to your prep for the data engineering exam for AWS. In this
module, we'll be talking about data transformation. We'll be covering ETL
pipelines, using containers within those pipelines, dealing with data coming
from multiple sources, as well as optimizing costs. Let's get started. So, let's
jump in and talk specifically about data processing. So, what exactly is data
processing? Data processing is the transformation of data from the raw data
into meaningful and useful data that we can actually categorize. We have
been progressing for a while on how exactly we process data, and really
there have been three main focal points on data processing from one thing
to another. So, the first and foremost, when you're thinking about the journey
of data processing and how data evolves over time, think about the old
manual pen andpaper, like a teacher grading an essay, allowing you to write
down your data and to be edited to make sure that it is corrected easily.
Then we have the next step up, which is the more mechanical section, which
requires much less effort of getting your ideas down on paper, it's also more
streamlined and makes sure that we actually have everything, the same font
and sizing, that we had on the actual paper. And lastly, we have the
electronic side of things, specifically things with computer programming and
machine learning to process data even faster and more effectively. This data
processing track actually has a name, it's called the ETM, standing for
extract, transform, and load. So, let's look at a blueprint of how that data
processing steps actually work. First, the data gets pulled and/or extracted
from various sources to be pushed into our data lake, where it is validated
and transformed into readable data. Then, it is loaded into our target
database. Once it is in that database, it is able to be published into reports
for whatever business needs that the company has for it. This build process
allows for data to flow effectively and quickly, meaning that you can get
information quickly from your extraction sources all the way up to your
reports without much hassle. But let's slow down first, let's talk about the
most common ways of transforming your data. The first thing we're going to
talk about is filtering. This is a process of where certain columns and rows
are filtered out before saving and inserting that data into its final destination.
For example, let's say you have an application based off of these rows and
tables, and you need a table for your new data certification exam, you would
need to filter out the historical data for all your other tables, that way you
can have all the data for years that you have been collecting it for your data
certification. Another way to transform data is through mapping data by
taking an input from a data source and changing it, also referred to as
conforming it, to get the equivalent of that data in another format. For
example, you might need to transform some data fields from a local time
zone to a standardized time with date stamps, or you might need to
transform some text field to numerical fields for machine-learning apps that
specifically don't quite understand some things. For instance, if you have a
survey about the likes and dislikes to be used in recommendation engines,
you'd probably need to map it to something more objective, things like likes
are ones and dislikes are zero. Another method is deriving variables. When
variables are computed from other variables in the same dataset or other
datasets. For example, you might have a final price that is computed by
subtracting the discounted price from the original. The next occurrence we
have is for aggregating data. As the name implies, this is where you would
have extra columns or fields to summarize data by grouping and aggregating
that data across a particular dimension. The simplest example includes
calculating the sum, average, mean, or median of data across any specified
dimension, for example, the sum of prices or the average of time spent on an
application. Another way of transforming data is through duplication. This is
where you would work to remove duplicated data, which is effective because
it cuts down on storage costs, as well as query times. This process can take
place either during the transforming stage, so before the data is inserted into
its final database, or can happen after insertion. And all duplicating data
examples usually revolve around removing records that are identical across
all fields. Lastly, we have splitting data, this usually takes place when you're
refining or cleaning unstructured data by splitting certain fields. For example,
splitting the address field into street, city, and zip code names. So with all of
this information, let's now talk about the two main services in AWS that are
specific for ETL jobs. These two services are EMR and Glue. Well, we have
gone over Glue in this particular course, and you will be going over EMR,
which stands for Elastic MapReduce, later in this learning path. But for now,
let's talk about the main differences between these two services and why
you might use one over the other. These two services were built with
different thoughts in mind. EMR was designed with specifically the
integration of ETL in mindd. While both services have ETL in mind, Glue is
more focused on big-data processing, while EMR focuses on the ETL jobs
themselves. The next thing we have are the customization options. EMR has
more customization options for third-party support, while Glue looks closer at
the built-in capacities with other AWS services. Finally, we have how you load
your connections. EMR lets you be the linchpin in loading your data into your
connectors, while Glue allows for more automatic migrations and allows you
to have other ETL providers. Basically, these two are very similar services.
Glue is a quick-and-low effort way to execute ETL jobs, while you EMR is
more customizable and more specific for ETL jobs. Allowing for ETL jobs,
real-time streaming workloads, specifically for machine learning, and using
already existing infrastructure, EMRs flexibility comes with a more hands-on
style of running, but does usually result in a less-expensive version than Glue
because you're avoiding some of the serverless features that Glue really
touts, so another thing that we're going to cover in this particular clip is
going to be Apache. Apache is a software organization that's dedicated to
maintaining open-source software projects. In the big-data field, you might
be used to seeing some of these particular names. One of the popular ones
is Apache Hive, which is a data-warehousing software facility, dealing with
large datasets residing on distributed storage. You also have Hadoop, which
is very popular and in the big-data space, which is a software library
framework that allows you to process a large number of datasets across
clusters and computers, designed to scale up and down to meet your needs.
You've also probably heard of Avro, allowing you to run data serialization,
data serialization being the process of converting an object into a stream of
bits to more easily save and transmit it. Spark is a multi-language engine for
executing data, be it through engineering, science, or machine learning on
single mode machines or for clusters. Spark is one of the more flexible
services that Apache offers. Another name you might be seeing popping
around is Airflow, which is a way to programmatically work with your
workflows. In this clip, we'll be talking a little bit more in depth about Apache
Airflow and some of the services that work with it in AWS. If you want to dig
into some of these services with Apache deeper, I have listed them in the
resource tab, that way you can actually take a deeper look into them.
Apache Airflow allows you to create your workflows, allowing you to build,
schedule, and monitor your batches and Step Functions. Airflow is a scalable,
dynamic, and extensive, meaning that Airflow has a modular architecture
and a powerful Python skeleton, allowing you to use Python features to
create your workflows. Really, anyone with some Python knowledge can
deploy a workflow using Airflow, and you can use Airflow to build
machine-learning models and transfer data to manage your infrastructure
better. So now that we've gone over Apache Airflow, let's talk about how it
works specifically with AWS. AWS has a service, Amazon Managed Workflows
with Apache Airflow, or AMWAA. AMWAA is a way to enable you to build out
your orchestration, creating your environment that has your clusters, which
has all the pieces to run your workflows. You can build your orchestration by
uploading a DAG, or Direct Acyclic Graph file, into S3, which is all the pieces
to run your file. This service allows you to use the ETL pipeline to extract,
transform, or load your data and learning to run your applications against
that data. AWS is another way to deal with your orchestration as well. You're
able to use a GUI, or graphical user interface, that allows you to build out a
workflow, which can be a way to piece out your workflow with an
event-driven architecture. This is an example of what one of these workflows
can look like. In this example, there is a Hello World script that needs to be
run with two different states, Hello and World. Once the workflow sees these
two in a passed state, it ends in a passed state to complete the workflow.
Now, let's say the script is missing, as you can see under the choice state,
you have the is the script where it needs to be set to false, meaning it's not,
and true, meaning it is. We also see that the default is to assume that the
choice state is true, meaning the script is where it needs to be. So these Step
Functions allow you to build out a lot of different environmental and
orchestration options, hitting certain services within AWS, finding them in a
pass or fail state of whether they are running or not, and reporting back to
you once they've finished their job. Realistically, you have two different type
of workflows that you will find within Step Functions. One is the Standard
which is to execute each step exactly once, like in our example here, but you
also have the Express workflows which can execute things at least once,
meaning they can hit that same step several times within the function, but
has a set time that it will run for, so keep in mind that when you're building
out your examples. So now that we've talked about those two, let's talk
about what the difference is between AMWAA and Step Functions. So one of
the first points that we're going to hit specifically is auto scaling. Airflow does
allow for auto scaling, but you have to integrate it with other services first,
like Amazon's Fargate, which is a service from AWS that allows you to
manage and monitor your applications, not your infrastructure. While some
functions allow you to scale automatically without having to fiddle with your
steps, Airflow also allows you to map out your options of queries and develop
query execution plans. With Airflow, you also pay by the hour depending on
the specifics inside of your development. With Step Functions, it's also a
pay-as-you-go model, with the first 4000 transactions, or steps, through your
workflow being free per month. Finally, we have why you might use Airflow,
which allows you to manage complex codes and workflows, whereas Step
Functions are more suited for simplifying complex applications. The last
thing, specifically, we're going to go over in this clip is the Glue workflows.
Much like the Apache Airflow and the Step Functions that we saw in the
previous slide, Glue has workflow options as well, which can sometimes be
mentioned on your exam, so we're going to be briefly touching on it here.
Glue workflows allow you to create and see complex ETL jobs, which allow
you to build out and incorporate multiple crawlers, jobs, and different types
of triggers to cause those jobs to start running. Just like with Step Functions,
you could see a GUI to build out your flows and can have a crawler that can
dig into different data sources and can kick off ETL jobs from crawlers finding
information. You can also set triggers that allow you to kick off the crawlers
and ETL jobs from events that happen where within your environment. These
workflows can be built manually one component or resource at a time or can
be built overall quickly with a blueprint, easily uploaded into the console and
run from there. With that, we have completed this clip. I hope that you've
learned something, and I look forward to seeing you in the next one.

Using Containers in Data Pipelines

Hello, everybody, and welcome to this clip in the data transformation

module. The first thing that we're going to discuss today are the pipelines for
data. Why a pipeline, you ask? Well, we have data flowing in from a data lake
or data warehouse, then data gets transformed, and flows into a target DB,
which normally spits out a consumable report. So, in a lot of ways, having
data move from one stage to the next in this fashion, where we first ingest,
process, and ultimately publish, resembles a typical pipeline. Ingestion
happens here. It's all about dumping the source data into the pipeline,
whereas publishing happens here. It is all about delivering that data into a
fashion where it can be consumed by the users. Where things get interesting
is here in the middle where the processing happens, so let's unpack this
particular stage in the pipeline. So, one of the things that we can talk about
in that data-transformation stage is specifically our containers. Containers
were a buzzword for a while, but it does have a lot of substance behind it. A
container is a lot like these jars here. Containers are a way to virtualize an
operating system so it can run anywhere. These containers are often
confused with VMs or virtual machines. Containers are a much more
lightweight than VMs not taking up nearly as many resources. Containers
also virtualize on the OS level, while a VM virtualize on the hardware level.
We also need to remember that containers share an OS kernel and use a
fraction of a memory and hardware space that a VM requires. Containers also
hold specific software packages, that way, they can be used interchangeably
between other services inside of AWS. Instead of having to do a singular EC2
that has all sorts of software on it, you can containerize this process, allowing
each of the individual containers to make things a little bit more efficient.
When it comes to normal pipelines, traditionally, you have a monolith style of
pipeline, meaning you have all of your resources in the same place and
everything is written in a similar language that is understood by all the
components, easily allowing for these connections to run from one place to
another, usually on a single server, meaning you're a bit more restrictive as
the individual pieces are together in one instance. When it comes to
containers, you can utilize microservices, you definitely have more wiggle
room on how you build out your environment. Allowing you to build your
pipeline with your containers, each piece has a separate container. With
these containers, you can use different languages that are more efficient or
utilized in their particular pieces. And since containers are usually polyglot,
that means that they are able to understand different languages and still be
able to function correctly. You also get more agility with this particular style
of workflow, considering that you can tweak in the individual pieces to speed
them up or slow them down to make sure that the flow of information is
going correctly. And finally, with a more containerized system, you can also
build at scale. Let's take, for example, you don't need as big of a data store
for some incoming data, but you do need a higher processing power to make
sure that incoming data is dealt with correctly and put in the right places the
first time. Under a monolith system, you would have to upgrade the entire
system in order to get this to work, meaning that you would have a lot more
space and a decent more overhead for this particular project. Whereas if
you're using containers, you can simply scale back on your storage and up
your CPUs on the containers that are holding those particular services. AWS
has two main type of container services, the first is ECS, which is Elastic
Container Services, and then you have EKS, which is Elastic Kubernetes
Services. These are the two main container softwares that AWS normally has
in their catalog, so let's dig into these two options in a little bit more detail,
starting specifically with ECS, or the Elastic Container Service. This particular
service allows you to orchestrate your resources, which allows you to
containerize your applications and map out your resources to be more
efficient inside your environment, allowing you to launch, manage, and scale
as you grow, or shrink your environment with the waves of your business
needs. ECS is meant for simplicity, decreasing the number of decisions you
have to make around your infrastructure and security. ECS has three main
layers on how it works, capacity, which is your infrastructure layer, which has
all of your components on it, the controller, which controls where the
deployments happen, allowing you to manage your applications and
containers, and then you have the provisioning layer, which gives you the
tools to interact with schedulers for deploying and doing regular
maintenance on your containers to make sure that your information and
application are running smoothly. Now let's talk about EKS, or Elastic
Kubernetes Services. EKS has a very similar setup to ECS. They have the
similar ways to create and control the builds that happen within your
containers, with EKS you also gain the controls and maintenance for your
Kubernetes control panel, which gives you a big boost up in security and a
few more integrations. The extra services allow you to run Kubernetes inside
of your cluster, meaning you have a lot of the expandability and movability
of ECS, but with an extra layer of security on top of what was already there
by adding Kubernetes on top of it. This can add to the complexity, as
Kubernetes is not always the simplest service to interact with, but generally
speaking, it is considered a good idea to look into, assuming you have
someone that is knowledgeable in the Kubernetes environment. All right,
let's try and compare and contrast these two services. First, and foremost,
with ECS, you are utilizing AWS orchestration engines to build out your
environments and containers. While EKS also does this same orchestration,
the focus points on EKS are more towards your Kubernetes experience than
towards the containers themselves. With ECS, you are actively fine tuning
your auto scalability, allowing you to do small finetuning, including when and
what needs to be scaled. One of the things that EKS mostly focuses on is the
multi-cloud deployments, allowing you to build out your EKS environments
on more than just the AWS platform. ECS really focuses on your
microservices, much like we were talking about earlier in our container
section. While EKS also has its microservices set up, the thing that most
people focus on when they're dealing with EKS is usually the integration of
security, specifically when it implies with other AWs services. With that, we
have finished this clip. I hope that you've learned something, and I look
forward to seeing you in the next one.

Data Coming From Everywhere!

Hello, everybody, and welcome back to this module on data transformation.

In this particular clip, we're going to be specifically looking on data coming
from everywhere. The two main things that we're going to focus and are the
main focus of the exam when it comes to multiple source data integration is
multiple-source EMR and Redshift utilizing multiple-data warehousing. Let's
get started. So, let's talk about EMR multisource. But before we actually dig
into multisourcing, we first need to talk about what exactly EMR is. There's
another clip coming later in this learning path that will go over EMR in more
detail, but we're going to touch on the basics of what exactly it is so that we
can talk about multisourcing. So, let's talk about EMR. EMR is AWS's big data
solution for huge scale data processing, interactive analytics, and machine
learning. This big data solution builds off of open-source frameworks, usually
an Apache subset, think Hive, Spark, Presto, that kind of thing, and then runs
on various services within AWS. EMR allows you to run ECL jobs within a
cluster. These clusters can be anything from EC2, EKS, outpost, or
completely serverless options. With these data processing options, you can
also have a lot of AWS services that help you develop, run, visualize, and
help debug this data pipeline. It can also be used for data science
applications, SQL queries, and machine-learning workloads. EMR is a real
powerhouse when it comes to the data pipeline and ETL jobs as a whole. So,
now that we have a kind of an idea of how EMR works, let's talk about
multisourcing. With normal EMR, you have a cluster that looks like this box
here. These clusters consist of three types of nodes normally, a primary, or
master, node that orange one there, which manages the clusterrunning the
software to coordinate and distribute the data and tasks across the other
nodes. You have core nodes with software components that run tasks and
store data within your various systems. A lot of times you'll have it distribute
specifically to a Hadoop Distribute File System, or an HDFS. You also have
task nodes, which run tasks specifically to utilize that data and push it to
other sources. These particular nodes don't actually store the data, they
usually just transport it or transform it into something else. They're also an
optional type for these clusters, you don't necessarily have to have a task
node inside of these clusters, but you do need to have a primary or master
node, as well as a few core nodes in order to actually run the cluster as a
whole. For multisource nodes, you have multiple nodes that are acting as
that primary node, meaning that they can take data from multiple sources,
meaning you're able to process and store data across this cluster and cast a
wider net for bringing in more information and cataloging it correctly. Now,
let's talk about multi-data warehousing on Redshift. We're going to go over
the requirements, your prerequisites, and limitations, specifically. For this
exam, you really need to know some of the specifics when it comes to
building out your Redshift environment. So, for specific requirements, you
need to have your Redshift connected to a datashare. Redshift's multi-data
warehousing is currently only available on three different types of clusters. If
you have a serverless work group, it'll work just fine on there, but the actual
two types of clusters, like traditional Redshift clusters that it works on are the
ra3.4xl clusters and the ra3.16xl clusters. These are the only two clusters
that currently have multi-data warehousing available to them. Amazon is
going to be adding more clusters to this list. But as of this moment, and as of
this exam, these are the only three that it is currently available on. You also
need to have your metadata discoveries, which is when you connect directly
to your data share based on your view catalog data. The last thing that you
need to have on your warehouse is you need to have your encryption set up
correctly. For prerequisites, you need to have the following. You need to have
a data warehouse created, have loaded data with your Query Editor, have
loaded data into your S3, you also need to have a fundamental
understanding of SQL commands to navigate through the warehouse to build
out things correctly. So lastly, we have the limitations specifically for this
particular warehouse at the moment. Right now, you cannot connect to these
types of warehouses via the data API, you have to connect via a datashare.
Consumers are also unable to see permissions granted by that datashare to
their user base. There's also the fact that consumer writing to the datashare
objects will not trigger an auto analyze option, meaning you have to actually
run SQL commands once you have added new data or information into your
warehouse to analyze that new data to place it to where it needs to be. And
with that, we have completed this clip. I hope you were able to gain
something from this, and I look forward to seeing you in the next one.

Optimizing Costs

Hello, everybody, and welcome back. In this particular clip, we're going to be
talking about cost optimization when it comes to data transformation. We're
going to be going over three specific subjects when it comes to that
particular cost optimization. We're going to be going over Glue Python shells,
PySparks, and Spark jobs. Let's get started. Before we jump into those three
main targets, we're going to need to talk about something that actually
drives those particular processes inside of AWS. We need to specifically talk
about data processing units, or DPUs. A DPU is a computer processor that is
tightly integrated with a general-purpose CPU with a network-interfacing
hardware. DPUs allow you to make your processing jobs easier by allowing
you to dictate more resources to spread your load across multiple units,
meaning you can work faster and more effective. You're also dealing with a
cost being measured by how many DPUs you are utilizing. So, let's get into
the meat of this clip. The first thing that we're going to cover today is going
to be Apache Spark. Spark is a way to bring your data together, allowing you
to run jobs inside of AWS Glue and process data in a large batches. These
batches can be run as ETL scripts using glueetl through the command line.
Spark is also a framework that is utilized inside AWS EMR, therefore forming
features to Spark being your batch and data streaming, SQL analytics, data
science at scale, as well as machine-learning applications. PySpark, as the
name implies, is a Python API for Apache Spark, allowing you to use real-time
data processing with the flexibility of Python. The combination of Python and
Spark allowing the ease of use and ability to learn with Python, meaning you
don't have to learn anything specifically more complicated than your normal
Python language, and the active power of Spark being able to create multiple
jobs and handle a lot of data processing. AWS Glue supports the extension
for PySpark, allowing you to build out your ETL jobs reasonably from the AWS
CLI using Python or Boot3. The Python shell option in Glue is potentially
cheaper than just running Glue on its own, but let's break that down a little
bit. The Python shell for Glue is within the Spark environment. These shell
jobs allow you to run Python scripts, and you can use the various versions
based on what version of Glue that you are using to manage all of your data,
allowing you to easily keep up to date with both Python and Glue. This shell
is created and modified again through the CLI using the Python shell
command. The other ways that this particular shell makes your jobs a little
bit cheaper is you only need to utilize one DPU to get these jobs to actually
start running. Let's compare the two main options when it comes to doing
these ETL jobs through the Python shell of port Glue or for Spark, specifically.
Well, the Python starts with one DPU by default, it specifically is designed to
run lighter and quicker jobs, while Spark gives you a 10 DPUs by default,
meaning you can potentially run things faster as you spread out your data
resources across multiple different places, and allowing you to have more
processing power to deal with more data to be allocated. The payment
structure for both of these options are very similar, but not exactly the same.
With the Python shell, you are billed per hour with at least one minute
minimum. Spark, you are also billed per hour, but your bill is also variant on
what version of Glue you're running and the version of Python that
corresponds with that version of Glue. For example, Glue version 2.0 or later
has a 1-minute increment for bill time, whereas using 0.9 or 1.0 versions of
Glue has a 10-minute minimum. Finally, we have the fact that the Python
shell is designed to run light ETL jobs quick-and-simple load that don't
require a lot of resources, while Spark is designed to handle some of the
more hefty jobs. Spark is also designed specifically for scaling in mind,
allowing you to move horizontally, vertically, or both when it comes to
actually dealing with your ETL job with how much data you have coming in.
With that, we have completed this clip. I hope you've learned something, and
I look forward to seeing you in the next one.

Applying Programming Concepts

Infrastructure as Code (IaC) for Repeatable Deployments

Hello everybody, and welcome back to your prep course for the Data
Engineering Exam for AWS. In this module, we'll be talking about applying
programmatic concepts, specifically things having to do with Infrastructure
as Code, as well as some SQL concepts that you'll really have to get a handle
on in order to make it through this exam. Let's get started. So, what is IaC, or
Infrastructure as Code? Before we had IaC, we, as engineers or developers,
would have to build out our resources one by one, which is a whole thing.
When building out your environment at first, it's fine, with not as complicated
resources or security settings allowing you to build out your environment
however it grows. But, as time progresses, things obviously become more
complex, which means you have a lot of different chances to mess up your
builds by accident. This is where we can use IaC. Using software's like
CloudFormation or some third-party software's allow you to really standardize
your deployments and build out your whole environment in one go, making
certain that you don't have to worry as much about human error.
Realistically, you should consider utilizing IaC, or Infrastructure as Code, if
you're facing some of these issues, the first being that you're mapping out
your environment. Building out a blueprint using CloudFormation or some of
the other IaC options, allows you to see the flow of information and the flow
of traffic inside of your environment, meaning you're able to get a clear
picture of your overall environment. Another reason to utilize IaC is if you're
specifically building out your environment to respond to growth and changes
within your traffic. IaC allows you to take advantage of scaling options,
making sure your environment can scale with the changes that occur from
inside and out. There's also the automation to consider, building out these
environments can be complicated, and one small miss can make it so the
information just doesn't flow, or a small tick on a security platform can make
it so users can't interact with your environment. Though it can be really great
to have your blueprints down so you can really cut down on human error as a
whole or just error in general. Having an extra pair of eyes on a set of
blueprints always makes things a little bit easier. So, let's go over the main
options in AWS when it comes for IaC. The first, and probably the most
popular option, is CloudFormation, but that doesn't mean that it is the only
option. Another option that we have is the Cloud Development Kit, or the
CDK. We also have some third-party options when it comes to building out an
IaC for our environments inside of AWS. CloudFormation, specifically, is
AWS's built-in option for building out your environment and optimizing it for
any size. The Cloud Development Kit is kind of like a layer on top of
CloudFormation, allowing you to build your resources and environments in
different kinds of languages so you don't have to just stick with the YAML and
JSON specifically to build out your environment. We also have third-party
softwares that can still utilize with AWS, like Terraform, allowing you to sync
your environments with your code, making sure that they always match and
drift is less occur. It also allows you more wiggle room when it comes to
building out languages when compared specifically with CloudFormation. But
let's look at CloudFormation and the CDK case specifically since, realistically,
those are the main two that are going to be focused on this exam. So let's go
over the main points that you really need to understand in order to go into
this exam when it comes to CloudFormation. The first thing to dig into is that
CloudFormation utilizes templates written in YAML or JSON. Every time your
resources get built out, CloudFormation creates what's called a stack. These
stacks make it so that you can connect and build out to your resources all at
once. One of the things that really helps you make sure that you don't have
any drift or human error when it comes to your build out and templates,
specifically in your environments, is that CloudFormation utilizes change
sets. When you launch a stack or a new blueprint of resources, a change set
identifies the changes within your template compared to your environment
to make sure that you are aware of the actual changes that are going to
occur when you deploy your new resources. So the other option that gets
talked about very briefly in the exam is going to be the AWS CDK, or Cloud
Development Kit. This kit is a way to build out your environment using
developmental languages instead of just YAML or JSON, think closer to
JavaScript, Python, or C+. CDK really allows you more wiggle room when it
comes to your development cycles, as these are usually done inside of an
IDE, allowing for multiple people to actually build out at once. Also the type
of code that you utilize inside of a CDK can be declarative or imperative
statements. Declarative statements, meaning that you are relying on
supplying the exact results to the development, whereas imperative
statements are more for well-defined instructions. CDK also provides you a
library with your constructs or resources, these acts like building blocks that
contain everything you need to build out your resources. So, digging into the
main differences between CDK and CloudFormation, what we really need to
take a look at and dig into is the differences between the code styles. With
CDK, we have realistically three different levels of how our code is built out.
The first level to consider is the CloudFormation constructs, meaning that
you are using the same syntax as a CloudFormation and offer no abstracts,
which looks like this. This matches what a lot you would see inside of a
CloudFormation stack. The next level is an abstract construct, also known as
a curated construct, which are developed by the C K team and are usually
widely-used construct types, meaning that you can add cloud resources, but
have particular aspects that are not steadfast and defined like in
CloudFormation, and it looks like this. Finally, we have layer three, which is
known as pattern constructs, which allows you the highest level of
abstraction, allowing you to have prebuilt or custom patterns that you can
have multiple resources from one or more services. All right, so let's talk
about how exactly these services stack up against each other. Keep in mind
that you need CloudFormation in order to run CDK in order to deploy
resources, but let's still compare some of the basics about them. The first
thing that we can compare is how we build our code. With CDK, we can use a
lot of different code types to build out our infrastructure, whereas
CloudFormation allows you to utilize only YAML or JSON within your template
files. Next, we have the fact that CDK is more geared towards developers.
You get to thankfully write less code and deploy more. With CloudFormation,
you're dealing with more template files, meaning you always get a consistent
build out, but it can take more development cycles to actually get that
template file where it needs to be. We also need to consider the rollback
options for both of these resources. CloudFormation gives us a more visual
drift detection and can automatically rollback your build, informing you that
something went wrong through some of your code block, whereas CDK, you
don't get as much visual representation. Normally, when you're running a
CDK function, you'll just see that the build out failed and that you'll have to
try again. With that, we finished this clip. I hope I have cleared some things
up, and I look forward to seeing you in the next one.

Intro to SQL

Hello, everybody, and welcome back to the AWS Data Engineering Associate
learning path. In this clip, we're going to be talking about SQL, which stands
for Structured Query Language. This language is a coding style that looks a
little bit something like this. And when you use this code inside of a SQL
processor, usually a database that runs things like MariaDB or MySQL, you
are presented with information or a table that looks like this. As you can see,
this is a table with student IDs, the student first and last name, and then the
actual full name of that student, all of this information is given back to you
because of the code that you put in originally. So, when it comes to SQL,
there are a few services that AWS utilizes in relation to relational databases,
the first being RDS, which stands for relational database, which has a lot of
options when it comes to building out your databases, including eight
different flavors of build options. You also have Redshift, which we talked
about in an earlier clip, and AWS Athena, which is a service that allows you to
filter your data and see your data from places like S3 or Glue. Athena also
lets you get a more in depth look of how things are going inside of your data,
making it easier to see patterns and changes within it. So, let's talk about
some of the common commands that you will see with SQL, the first being
the SELECT command, which is a way for us to pull data from tables within
the database, giving you a detailed report on your database. Next, you have
the WHERE clause, which allows you to filter your data records to only pull
what you need from your tables. Then, you have AND, which works with
other clauses, allowing you to chain conditions together to build out the
perfect command that you're looking for in your database. You also have OR,
which allows you to filter your results, but you can filter things by different
markers, giving you more options to have specific result patterns. The next
few options are more, think of aggregators, things that bring your data
together. First off, you have ORDER BY, which allows you to sort your results
by conditions from your table. And finally, you have a GROUP BY, which
allows you to pull all sorts of records and make your own aggregated results
with things like count to have the number of results happening within a table
or database. So, some other commands you should probably remember for
your career journey, but are not necessary for the exam overall, these are
some of the commands that allow you to move things and modify your
database as a whole. The first is INSERT, which means that you add your
data into a table with customized options to add things to multiple tables.
Next, we have the CREATE statement, which allows you to actually build your
environment of tables and databases. After that, we have UPDATE, which,
much like it says on the tin, allows you to change existing records within
different tables and databases. And finally, we have DELETE, which allows
you to remove records from your tables or databases or delete full on tables
from your databases. So let's go over some necessary reference points when
it comes to building out your commands. First and foremost, remember, your
commands, clauses, and the like, will always be written in capital letters,
which can list the differences between commands and the description words
for things like your table names, and like. The next thing to be aware of that
brackets are to add optional things for you to add within your command. This
can be either to change the command overall or to dial-in things within your
command to get exactly what you need. You also have curly brackets, which
help you build your commands as well, but it allows you to choose between
different arguments to get the right data you need. After that, we can
consider pipes, not those kind of pipes. There we go. Pipes allow you to
choose between different arguments to get the right data. They also allow
you potentially to chain your commands together. Italics in commands are
things that you need to change. So let's say that you have some example
code that is being used within your documentation, you would need to
change what is in those italics, things like table names, to make sure your
command is actually working. The three dots, or ellipses, are a way to tell
people examining your code to repeat whatever comes before it. Then,
finally, you have quotation marks, which are for putting around strings within
your commands. You can use single quotes for literal meanings, like what's in
the quote has a specific meaning within your command, and you have
double quotes for identifying things, like tables, columns, and row names. So
let's talk about some of the argument syntax when it comes to SQL. The first
syntax you need to be aware of is the dollar sign, which allows us to look for
letters within a string which is sorted by a position. Next, we have the caret,
which allows you to create commands that look for one or the other
conditional argument being true, but not both of them. Then we have the
percent symbol, which is used to look for things in your data like specific
numbers within your order tables, or like in the example on screen, where
you have %at%, which leads to a data response on your report back. Then
we have the ampersand, which allows you to combine your operations and
invoke bitwise on your commands. So, bitwise is a way for you to add binary
values to your results that you're pulling for your tables, translating your
characters and statements into binary. So, let's go over some of the ways to
optimize your queries. Whenever you're doing an SQL command, try and
avoid select *, as it will pull back everything inside of your table, which can
not only take a while to come back as an actual report, but you also have to
worry about the strain it puts on the database as a whole. Try and aggregate
your commands, meaning that you can really pull pieces of data with case
expression, which saves you engineering cycles so you don't have to keep
selecting the same tables with the same commands over and over again.
Next, try and restrict your data predicates, meaning that you can filter and
restrict your data that returns in queries, meaning you don't have to bring
back your whole table. Another thing to consider is try using joins when they
are more effective. Another thing to consider is try using INNER joins when
you have to join your tables together, as they're a little bit more effective
than some of the other join types. We have another video specifically talking
about joins a little bit later in the module. Avoid unions unless you absolutely
need to combine SELECT statement results. These commands can really
make your database chug, but these commands can be useful when
necessary. Try and be as specific as possible, meaning that you can get back
exactly what you need without having to filter your results, saving you a lot
of time and effort overall. And finally, the EXPLAIN command is your friend.
This command is very helpful when it comes to showing you how your
command is going to run, and what is the reference point for what that
command is looking at. With that, we have finished out this clip. I hope
you're able to learn something, and I look forward to seeing you in the next
one.
Let’s Talk about SQL Joins

Hello, everybody and welcome back to the AWS Data Engineering

preparation learning class. In this clip, we'll be talking about SQL JOINs. JOINs
are a way to combine rows of two or more tables based on related columns
between them. For example, let's talk about some order listings. In this table
here, you have rows with the order IDs being a unique order on your site, the
CustomerID for the user accounts on the site, and finally, the OrderDates for
them. Now we have another table that has the customer names and the
agent that worked with the client on the account. We have a CustomerID
again, but this time, the ID is in reference to the customer's name and the
agent that worked with them. So, let's say that we wanted to have some of
our order information, but we need to have the customer's name associated
with it. Well, that's where JOINs come in. The command that joined these
tables together would look something like this. We see the SELECT statement
from the first table of Orders and the JOIN of them together with the
customer table combining CustomerIDs on both tables. This lets us have a
name in the new table, instead of just having a CustomerID, and that fresh
table that we just aggregated with those JOINs would look something like
this. Now, there are a couple of different types of JOINs, so let's dig into that.
Realistically, looking into SQL, there are four different types of JOINs. We
have INNER JOINs, LEFT JOINs, RIGHT JOINs, and FULL JOINs. Now, let's dig
into each of these different JOIN types in a bit more detail. So, let's first talk
about INNER JOINs, an INNER JOIN is much like the command that we saw on
our example, it allows you to return records that have matching values from
both tables, be that from matching columns or matching rows. However,
you're only pulling that data that matches, so you're not pulling data that
you don't necessarily want. Next, we have what's called a LEFT JOIN, or a
LEFT OUTER JOIN, which lets you return records from the leftmost table, or
your initial table, and only the matching records from your right or secondary
table, and it would look something like this diagram here. And then, we have
the opposite, which is a RIGHT JOIN, allowing you to pull records from your
secondary table and only matching records from your primary table. And
finally, we have the FULL JOIN, which is some of the biggest and roughest to
build with SQL, as it will return all of the records, as well as what matches on
both sides. For the purposes of this exam, you only really need to know that
the four JOIN types exist. So with that, you have completed this clip, I hope
that you were able to gain something from this, and I look forward to seeing
you in the next one.

SQL Aggregations and Pivoting

Hello, everybody, and welcome back to the data engineering certification
prep learning path. In this clip, we're going to be talking about SQL
aggregation. An example of what aggregation looks like in the SQL space, we
have this command. So, when it comes to aggregations, it basically means
you're bringing data together. Like we're seeing in this example, we're
bringing together data from the orders table, counting total sales and
shippings. The aggregation happens by using that GROUP BY and ORDER BY
that we see right here. GROUP BY allows you to group column two, which is
shipping in this case, and after grouping the column, it will put in an order of
total sales of the orders column within your database. Let's talk about the
five functions used to aggregate. First, we'll touch on MIN, meaning you will
bring back the smallest value in the selected column. For example, if you had
an order sheet again, you could bring back the data of the smallest price that
you have in your order sheet. Next, we have COUNT, allowing you to bring
back the number of rows within a dataset, which can be useful for knowing
how large your tables actually are. Back in our orders example, you can get
an idea of how many orders you have within that table by using the COUNT
function. We also have MAX, which is the exact opposite of MIN, which allows
you, in this case, to pull the largest order value from your table. We can also
get the total number of columns that have numerical values by using SUM.
And finally, we have AVG, which stands for average, and like it implies, pulls
the average values within numerical columns, things like pulling an average
pricing that you have in your order sheet. So, now that we have an idea of
what aggregation is, let's talk about pivoting. Pivoting is the ability to
combined data within the table. Let's take a look at an example. Looking at
this chart, we have three main columns, we have a Sales Person, we have
the Product that they're selling, and the Sales that they have accomplished.
Now, as you can see, John and Jane both have two entries, one being for
Apples and one being for Oranges. Now, if we were to build out a query like
this one here, now we have the Salesperson and the Product from SalesData,
and we're using this as a source of data for the pivot. Within the pivot, we
have the argument, COUNT, in Sales for apples and oranges and creating a
new table, and that new table would look something like this. That way, we
see we have the Sales Person and the two fruits that they are selling. Both
John and Jane have their accounts for the sales of both of the products in a
more concise format, allowing it for easier reads for people who aren't
necessarily as entrenched in data as data engineers are. So, with the
example out of the way, let's talk about some of the main building blocks on
how to pivot your data. The first thing that you need to remember is row
identification, these row identifiers contain values that have unique
identities. They don't have to be completely different, but they need to have
some differences compared to the other rows. Then you have the column
identifiers, which are much like the row identifiers, but you see them in the
combined result. Now we also have to keep in mind how we aggregate
things, which is the main values we're pushing together based on both of the
identifying factors that we have provided, both the row identification and the
column identification. Now, there are some specific steps that you need to
take when it comes to preparing your data for pivoting and aggregation.
There are three things you really need to take consideration when you're
prepping your data. The first is that you need at least one column to act as a
row identifier, with one column that will be used to make new column
headers and one column containing the values that will fill the pivoted table.
So, after you've prepped your data, you'll need to verify your data is
functional. There are really three main steps when it comes to verifying your
data after it's been prepped. Now, verification can happen while you're
prepping your data, it's just a matter of how you wish to do it. But here are
the three main things that you need to consider when you're verifying your
data for pivoting and aggregation. The first thing is removing any duplicate
data that may have slipped through the cracks, this can make it easier and
more concise when you're building out your new tables. Next, you need to
make sure that your data types are consistent. That way, you don't run into
any problems when you're running your data together. And finally, you need
to fill in any missing gaps within your data. That way, when you bring it
together, you don't have holes in your new table. And with that, we have
completed this clip. I hope that you're able to gain something from this, and I
look forward to seeing you in the next one.

Tumbling Window Aggregations

Hello, everybody, and welcome back to the Data Engineering Associates

exam. In this clip, we're going to be talking about a small thing that does pop
up on the exam every once in a while, so it's something you need to take
into consideration. In this clip, we will be talking about tumbling windows. So,
let's look at a more visual representation of what I mean when I say tumbling
window. In a tumbling window, you have a couple of main metrics that you
are dealing with. You are dealing with the data stream and then the specified
time within that data stream. This data can come in a few different ways and,
specifically, in different time intervals, but in dedicated self-contained
windows. These Windows are not overlapping, so think of it like a revolving
door allowing your data to come into a room, with a lot of different options
for the presentation that your data can come in during these windows. There
are three main services that allow for tumbling windows and their
modifications, the first one is Lambda, then you have DynamoDB, and finally,
Kinesis. Now, digging into a little bit more detail, with Lambda, you're
allowed to set up your specific tumbling windows within your functions.
These windows allow you to pull data from streams or events in bundles of 5
MB packets, with the actual window size being anywhere from 0 to 900
seconds. Next, you have DynamoDB, which can be both a destination and
triggering option within Lambda, but can help you build out your data from
the creation of these windows to make sure that you're not getting nonstop
data, but you're getting it in specific bursts that allow you a little bit more
control. And finally, you have Kinesis. Kinesis is one of the main streaming
services within AWS, so it can actively deal with the information coming
inside of these tumbling windows to make sure that it's only sending data
within those agreed-upon windows. Now, when it comes to the aggregation
of these tumbling windows, realistically, you have three main steps when it
comes to collecting your data into these groups. It, of course, always starts
out with your application code, how that information is going to be pushed
into that data stream and then sent into your window. Once this data hits
your SQL groupings from inside of these windows, you can filter, combine,
and view the new datasets within your primary database. And then, of
course, finally, you get the results from your data within these windows.
Now, because of this, there are two very distinct versions of windowed SQL.
You have the time-based windows, which is like what we see in Lambda,
which the window sizes are usually specified in terms of time, usually
through a timestamp within the application stream. Then, you also have the
row-based windows, which allows you to specify your window in the sense of
your incoming data sizes, you know specifics of how much data is coming
through on that stream and how many rows of data will be impacted by that
stream. There are, of course, other types of windows when you are dealing
with these kind of SQL styles, realistically, you're looking at about three that
you need to worry about in your career journey and then about one that you
actually need to worry about for the sake of this exam. for your career
journey. You need to be aware that there are three main types of windows,
them being staggering windows, tumbling windows, and sliding windows. For
the purpose of this exam, we're focusing on tumbling windows, but we'll give
you a brief overview of what the other two types of windows also do. With
staggering Windows, you're utilizing specifically keyed time-based windows,
so you're usually looking at specifically when that data arrives at your
window. This style of window allows for overlapping, meaning you don't have
to worry as much about when that data comes in, as it's unlikely to be out of
order or doesn't come in time at all. With tumbling windows, you're allowed
to open and close that window in scheduled intervals, which means that you
are only accepting data within those scheduled intervals, which can be useful
for a lot of different things, including, but not limited to, making certain that
you have security inputs in place, as it's much less likely for you to get
overloaded in your system if you only have it open and accepting information
at certain times. And finally, we have sliding windows, which allow you to
have a more continuous style of dataflow, but can be specified with fixed
time periods or row counts before actively closing out on the window, being
able to slide open and shut without much preparation. With that, we have
completed this clip. I hope you got something out of this, and I look forward
to seeing you in the next one.

Connecting to SQL Databases: JDBC & ODBC

Hello, everybody, and welcome back to the prep for the data engineering
associates exam for AWS. In this clip, we're going to be talking about the two
main ways that you're going to connect into the database. Realistically, you
just need to know the basics for this exam so that's all we're really going to
be diving into. The first way to connect is through JDBC, which stands for Java
Database Connectivity. So, let's talk about these connections. You first start
with your custom codebase that you've uploaded to the database. This can
be accomplished by using JDBC, which is broken down into two real parts, the
API and the driver. Both of these parts roll into a device that contains the
connections into the database. There are four main types of JDBC drivers, the
first is the JDBC-ODBC bridge that translates and converts the Java that's
coming from your code to an ODBC function. ODBC is another connection
method that we'll be talking about in just a minute. You also have native API
drivers that allow you to have the client-side libraries for your target
databases. Then, you have network protocol drivers, which uses software in
between to convert your JDBC into a traditional database call. These
softwares are normally called middleware and have a variety of different
methods. And finally, we have the database protocol drivers, also called thin
drivers, allowing your JDBC calls directly into vendor-specific database
protocols. Now, let's talk about the open-database connectivity, or ODBC.
This type of connection stems from an application sending information into a
driver that is usually localized on your device that your database is on, so
that driver sends that information to the data source and/or databases. And
here's kind of the main breakdown for the ODBC main components. The first
component is an application process that calls the ODBC functions and
submit SQL queries or commands into the database and requests. You also
have drivers that manage and load other drivers for every application that
sends a call to your database, with a centralized driver on an OS of the
device that your database is on. These drivers handle the calls and sends the
data back to its source, allowing them to see the information with the
database and call centers. The data source is just like it says on the tin,
where your data is stored and can be accessed by the drivers as a whole. So,
let's talk about the main differences in these two types of connections, JDBC
versus ODBC. One of the main things to be aware of is fairly straightforward
is JDBC is in the Java language only, while ODBC is much more open, allowing
you to program in just about any language. However, JDBC does allow itself
to be used on just about any platform. There's a lot of go between and
translator softwares allowing it to run off of most OSs, whereas ODBC mostly
is just a Windows option. When either of these drivers gets a call, they both
respond, but they do it in different ways. JDBC is object oriented, meaning
your data objects are seen as, well, objects and set bundles, while ODBC is
procedural, meaning it follows more of a set list inside of its calls, ticking off
boxes as it's going down the call itself. With that, we have completed this
clip. I hope you're able to gain something from this, and I look forward to
seeing you in the next one.

The Big Data Puzzle Coming Together

Hello, everybody, and welcome back into this journey to gain your data
engineering certification through AWS. In this clip, we'll be talking about how
the puzzle that is big data comes all together. Let's get started. So, let's
wade into the data ecosystem for a moment. The ecosystem in this example
is going to be a modern sales forecasting. So, what do you need to get
started in this environment? The first thing that you need to do is find a data
source to gain the insights on your sales and products. After you find the
data sources that you can get to, you need to be able to transform your
analytics from your data pools into a price leverage, meaning that you'll
have more of an idea of what people are looking to pay for your product or
service. And finally, you can leverage AI when it comes to getting more
information out of your data pools and analytics, allowing you to do things
like having more predictable outcomes and making sure that your inputs
actually follow a strategical pattern. So, let's build out a data lake. We can
start by taking in our data ingestion points. From there, you get options on
how to process your data, which you have the options to either process that
information by taking it down locally or by following blueprints within AWS to
deal with that information. You can also bounce between these two options
before moving on to the next step, which is cleaning your data. Cleaning
your data does things like getting rid of duplicate information, filling in any
gaps that are within your data, and making sure that your data is actually
relevant to what you're looking for. Now, once your information is all cleared
and analyzed, it's a good idea to set it through some security policies, be
that through compliance policies, encryptions, or some other options that are
available within AWS. After your security within the data lake is secured, we
can make our data more presentable for an analytic possibilities throughout
our environment. We can also make it generate reports for people who aren't
as data minded. So let's talk about the life cycle of the data lake as a whole.
Realistically, it starts with an ingestion and transfer, allowing you to gain new
information within the lake and transferring that to the processing options.
This moves into security and storage. The data moves into security, allowing
you to lock down your information to disable people from seeing what they
shouldn't. And once the security is set, you can set up storage for your data.
Being that you can set up places to contain your information, then you can
transform your data for various reports and business uses. And once that's
set up, you can get to analyzing and monitoring the data that's coming in
and flowing into your storage space. Once you get all of your transformation
and business uses set up, you realistically are going to monitor and analyze
the overall pipeline to make sure everything works the way that it should,
and fulfills your business needs. So, let's talk about the overall stages of
data. So, most data comes from a data source, be that a file type like S3 or a
data link. Now, when it comes to these data sources, they can offer two
different ways of processing. The first, being through real-time ingestion and
the second being through specifically data storage. Both of these options are
valid, it just depends from there which way you're processing. From here,
after the data source is ingested using services like Kinesis, or stored using
services like S3, it then promptly can move into the actual analytics stage,
which can be done via batch processing, machine learning, or using a stream
processing using things like Managed Apache Flick. And, you don't have to
necessarily go in this particular order when it comes to your processing, you
can go in just about any way you want to, your real-time ingestion or storage
spaces can really enter into any of these particular options without much
effort. Once they've been processed either by services like a Glue or Apache
Flink or even some of our machine-learning objects, we then go into the
analytic data store, meaning that you can do things like RDS or Redshift to
keep a track on what exactly is happening within your data stream, and
allows you to really dig in deep to what your data is actually saying. And
through this analytics, you can get things like reporting and dashboards.
Realistically, these things can go in a multiple of different directions. These
reports and analytics stores don't have to necessarily go in one direction, as
we see here. There are a lot of different options when it comes to building
out your stages with building out your data. It's very customizable, meaning
there's a lot of different ways to get to your end goal, which is your reports
and dashboards, making it presentable anyone within your business
operations to see what your data is saying. With that, we have completed
this clip. I hope you were able to gain something from this, and I look forward
to seeing you in the next one.

Conclusion

Summary

Hello, everybody, and welcome back to the last module of this leg of the
learning path for the AWS Engineering Associates exam prep. First of all,
congratulations, you've made it through the ingestion portion of this learning
path, but you are well on your way to being ready to ace this exam. Let's go
over some quick touch points from this leg so that way you can have a quick
refresh before you start on the next one. Here is a brief overview of the
topics we covered in the first module. Now, let's look at some of the main
touch points from all of these particular lessons. So a quick recap on AWS
Glue, Glue is an ingestion tool, it is a serverless solution for discovering,
prepping, and combining data, allowing you to pull data from different
sources to build catalogs for data operations. Glue has multiple capabilities,
allowing you to do quite a few different tasks within one service, allowing you
to do things like creating ETL jobs, gathering, storing, processing, and
cataloging data all in one service. It also allows for a lot of third-party
services that can interact within your Glue environment. The three main
types of triggers within Glue are event based or things that are happening
within your environment, time based, so that the way they happen on a
schedule, or on-demand based, which is you are personally in charge of
starting and stopping these particular events. Amazon Kinesis is primarily a
data-streaming service. The service allows you to stream data in near real
time, shipping that data to a point that you set within your environment, and
analyzing your data to gain more relevance from your data as a whole. There
are four main services under the Kinesis umbrella, the first being the Data
Stream, which is kind of the first step in your data-collection process,
gathering data from the specified point. The next step being Firehose, which
helps you move that data from the Data Stream to the collection point within
your environment. You also have under this umbrella Video Streams, allowing
you to store and easily analyze that video information that's coming through
from things like CCTV cameras or Internet-of-Things devices. And finally, you
have the Managed Apache Flink, which, while it doesn't have Kinesis in its
name, does still settle underneath this umbrella. Flink is a way for you to
analyze your data, combining it with Zeppelin notebooks, meaning that you
can use SQL to dig deeper in your data. You're able to transform your data
and structure your data in neat, easy-to-read tables. These services allow
you to collect and analyze your data with active streaming data. They also
allow you to interact with AWS services and multiple third-party services with
very minimal effort. Redshift is a way for you to collect data in a traditional
data warehouse. This type of storage is a central repository for your
information, data flowing into this warehouse from transactional systems,
relational databases, as well as a few other sources. Data warehousing also
allows you to put together multiple data sources so that way you can run
your queries against all of your data, not just small groupings of data,
allowing you to save time and resources, so you don't have to keep running
repetitive commands. Redshift also allows you to utilize Log Analytics,
meaning you can gain more information and data from your logs, seeing
more in depth on things like click-stream analysis and hardware associations
within your report. And finally, you have the business intelligence, which gets
you incredibly detailed and easy-to-read information on customizable
dashboards. These dashboards and reports make it so that anyone should be
able to understand the data that you're linking to them, so that way even the
most non-data savvy should still be able to gain information from your
resources. Now, let's talk about how we can load that data into Redshift. You
can load data into Redshift using what's known as COPY commands, and you
can do this through one of two ways, option one is to load data from S3. So,
with this option, you would need to load data into S3 first, then use your
COPY command to copy that data into Redshift. Your data source can be
things like RDS, a Kinesis Firehose, or any other type of data source. Then
you can load to S3, so, in other words, if your data is in S3, you can easily
use COPY commands to move that data into Redshift. Option two is that you
load your data in from a specified database, so you would move that data
specifically from your source database directly into Redshift. So, for this
example, let's say your data source is DynamoDB, you can use COPY
commands to copy the data directly to Redshift without hopping over S3. So
you also need to remember that there are three main trigger types when it
comes to data ingestion within your AWS environment, S3 event triggers,
EventBridge event triggers, and Redshift event triggers. Now, let's have a
quick rundown of how you can set up a Lambda environment. Remember this
is the example of your environment, it's currently waiting for input. Steve has
some information that he needs to load into S3 for both collaboration on his
project and to make sure that the file is safe. Now, once Steve has loaded
this file into S3, it will trigger a Lambda function. But that isn't the only way
to trigger that function, Betsy can also trigger that function utilizing an ARN
from the function. Either trigger method works, and once the trigger has
been fired, your function happens, which, in this case, allows you to write to
a DynamoDB table, allowing you to take stock of the new file that is currently
within the S3 bucket. This is the basics of how a Lambda function works. Now
let's talk about API, which stands for application programming interface,
which is really just the operation of allowing your server and your application
to talk to each other in a loop that looks something like this. Sending a
request, and then the opposite side sending a response. In this case, the
application sending a request of the App Server for information and then
sending a response back from the App Server to the application. When it
comes to your basic security architecture, I want you to remember that,
specifically, you are looking at a VPC that is built out like a shell with your
subnets inside of it, your security groups on top of that, and your NACL then
allowing access from the public back out. Things to also remember, that
security groups helped you filter who has access to your VPC, it is a stateful
control process, and allows you to add and block traffic, whereas NACLs, or
Network Access Control Lists, are stateless ways for people to access into
your VPC and subnets. NACLs are also a layer of security for your VPC and
acts as one of your first line of defenses in firewalls. In the next module, we
really focus on data transformation. Here are the clips that we covered in this
particular module, let's go over them in a little bit more detail. So, let's
review the blueprint of how the data processing works in steps. First, you
have data that gets pulled or extracted from various sources, it gets pushed
into a data lake where it is validated and transformed into readable data, and
then it is loaded into a target database. Once in that database, it can be
quickly published into reports for your business needs. These build options
happen very quickly and effectively, allowing for information to really flow
like a river through your environment, allowing you to quickly change tactics
if necessary. So the other thing to keep in mind when you're taking this exam
is that there are two main container types that work within the data
structure, the first being the ECS, which is your Amazon Elastic Container
Service, and then EKS, which is Elastic Kubernetes Services. Let's also do a
quick review of what EMR is. EMR is AWS's big-data solution for huge scale
data processing and interactive analytics, with some sprinkles of machine
learning no top of it. This big-data solution builds off of an open-source
framework, usually an Apache subset, think Hive, Spark, Presto, that kind of
stuff, and then runs on a variety of different services within AWS. EMR allows
you to run with a cluster, that cluster can be anything from EC2s, EKS
containers, or it could also be something like outpost or some other
completely serverless options. With these data processing options, you can
use a lot of AWS services to help you develop, run, visualize, and debug your
pipelines. It can also be used for data-science applications, SQL queries, and
machine-learning workloads. Now I can touch on the three cost-cutting
measures when it comes to ETL jobs. The first method, which is Apache
Spark, the PySpark, and the Python shell for Glue. That first method in
Apache Spark allows you to unify analytics and data. The next method is
PySpark, which is a Python SPI for Apache Spark. And finally, we have the
Python shell option within Glue, which is potentially cheaper than just Glue
on its own. The other way that this particular style makes jobs slightly
cheaper is that it defaults to one DPU to actually run Python shell jobs within
Glue. The third module we went over was applying programming concepts.
Here are the clips that we covered in this particular module, let's go over
them in a little bit more detail. So, as a reminder, there are three main IAC
options, which are CloudFormation, which is the first and probably the most
popular when it comes to AWS, we also have the Cloud Development Kit, or
the CDK, then we have third-party options like Terraform. CloudFormation is
AWS's built-in option for building environments, optimized for any size. The
Cloud Development Kit is a layer on top of CloudFormation, allowing you to
build out your resources and environments in different languages. And then,
of course, you have third-party software's like Terraform, allowing you to sync
your current environment with your Terraform files, making sure that your
environment always matches your code, and allowing you to build your
resources with buildings happening in different languages. So, next, let's talk
about SQL, or the structured query language. This language is a coding style
that looks something like this. Now, when you use that code, you are
presented with information or a table that looks something like this. As you
can see, on this table, you have four different columns of information, the
studentID, the first name of the student, the last name of the student, and
then the full name of the student. Now let's talk about the various types of
joins you have when you're talking about SQL. There are four different types,
the one on the far left, which is your Inner Join, which allows you to only take
back things that match from both databases; Left Joins, which takes
everything from a primary database and the matching from the secondary
database; Right Joins, which, much like Left Joins, take everything from a
secondary database and the matching primary database; and then, finally,
you have Full Joins, which pulls everything from both databases, including
what's matching between them. Now, we're going to touch on pivoting,
which is the ability to combine data from tables. In this example, we have
this particular chart, which has three main columns, being the Sales Person,
the Product, as well as the items being sold. As you can see, Jane and John
both have two entries, one for Apples and one for Oranges. With this
particular query, however, we would be able to have the Salesperson and the
Products from the Sales data table, and we are using this as a source table
for our pivot. Within the pivot, we're using the argument of COUNT in Sales
for both apples and oranges, and creating a new table, which looks like this,
where we have the Sales Persons have individual rows, but the Apples and
Oranges are two individual columns. This allows you to combine some of that
information and makes it a little more easy to read in general. Next, as a
reminder about Tumbling Windows, you have a couple of main metrics,
specifically the stream of incoming data over a specified period of time.
Within these specified periods of time, you have these individual windows
where data is flowing through the windows in various shapes and formats.
But these windows never specifically overlap, allowing you to make sure that
your data comes in and is processed in the fashion that it is entered from the
window. Next, we're going to have a quick comparison of the two different
types of connections. JDBC is in Java-only language, whereas ODBC allows for
multiple language in programming, usually specializing in C and C++. JDBC
can be used on just about any platform, as there are a lot of go-betweens
and translator softwares, allowing you to run off of most OSs, whereas ODBC
is mostly a Windows-only device. When either of these drivers gets a call,
they both respond, but they do so in different ways. JDBC is more
object-oriented, meaning that your data objects are seen as just that,
objects, and have set bundles within their packets, whereas ODBC is more
procedural, meaning that it's looking more for the context within the lists
inside of your calls. The other thing that you need to remember as a touch
point for your exam is the lifecycle of your data lakes. Remember, you are
ingesting and transferring data from wherever your sources may be to gain
that new information to the lake. You are then securing and storing that data
to be able to be processed. We can then transform and use that data for our
analytical purposes, such as dashboards, notebooks, and builds. And once all
of that is set up, we can monitor and analyze our information flowing into our
data lake to make sure that we actually have a stable environment for our
business needs. And with the highlight reel complete, you are one step closer
to being ready for your exam. I hope this refresher helped you out, and I look
forward to seeing you in the next leg of this adventure.
Data Ingestion and Transformation: Exam Tips

Welcome back, everybody. Let's go over some prep questions for your exam
from within the data-ingestion space. One of the things that I want you to
remember when you're taking this exam is to keep things as simple as
possible, and that includes when you're indicating costs. When you're doing
the calculations for costs of resources, as well as through implementation,
just be aware that the cost of code and resources are not going to be the
only part of your calculation, engineering cycles, and the cost of those
cycles, as well as how things are implemented and maintained are also going
to have to factor into your calculations. But let's jump into some of these
questions. So, here's an example question that we can take away from the
blueprint. What AWS service do you utilize for basically real-time batch
analyzation? While there are many services when it comes to data
processing and analytics, the words you need to focus on in this particular
case are near real time and batching. When you see real time or near real
time, remember the Kinesis family of services. And when you see the word
batch is also mentioned within this context, it usually is always referring to
Apache Flink. All right, the next question is, what is the name of the unit of
measurements that Glue utilizes to charge its hourly rate? The name of the
unit is a DPU, or data processing unit, which allows you to process your jobs
and spreading out the demand of those jobs across multiple units. Also
remember the breakdown for your DPUs, as they are the way you measure
the cost per hour and the costs that are associated and potentially change
from region to region. All right, what about the standard application
programming interface, or API, that allows Java programs to access database
management systems? Again, when you are looking in these exam
questions, search for those keywords. The two keywords we need to take
away in this particular case are the API and Java. So, we know, from that,
that we're looking at JDBC, which allows you to connect to your database
management system to communicate only through Java-based programs.
And remember that your ODBC allows you to also connect to database
management systems, but they communicate with applications primarily
using C and C++. All right, and the final question that we're going to go over
in this particular module, which of the following best describes the type of
data found in traditional relational databases? Now, remember, when it says
traditional RDS, or relational databases, think structured data, which is in a
standardized format of columns and rows, which are easily accessed and
processed through AWS. All right, congratulations. You have completed this
module and the first step in your learning path to passing this exam. I am so
excited that you're starting this journey, and I'm really looking forward to you
passing your data engineering associates exam with flying colors.

Data Stores

Introduction to Amazon S3

The shared responsibility model is a found additional security-related

concept in AWS. It defines who's responsible for what. And there are different
tiers of models. Consider responsibility in terms of home ownership. When
you buy a house, the bulk of the responsibility shifts to you. If you've ever
been a homeowner, you know there's a ton of invisible labor that goes
beyond the mortgage payments. From landscaping to trash and recycling,
there's just a ton of maintenance to go around. In contrast, you're assigned
much less responsibility when you rent, and obviously, even less
responsibility when you rent a hotel room. Likewise, infrastructure services
such as EC2 will require plenty of security tasks on your part. Granted, AWS
will take care of the security of the cloud infrastructure itself, such as the
physical data center, the networking and virtualization layer, but beyond
that, you will be in charge of the configuration and maintenance tasks, such
as configuring the firewalls, applying security patches, and installing security
software. In contrast, when you work with container services such as RDS,
the load of the responsibility starts shifting towards AWS. Yes, you're still in
charge of applying updates when necessary, but it's not as labor intensive as
infrastructure services, and when it comes to fully-managed services such as
S3 and DynamoDB, the load of the responsibility shifts drastically towards
AWS, given that the majority of operational tasks get offloaded to AWS. With
S3, your responsibilities as a customer include managing and securing your
customer's data and implementing client-side encryption. While AWS would
be in charge of server-side encryption, network protection and everything
underneath such as the platform. So the service itself, along with securing
the underlying infrastructure such as the operating system, network and
firewall on the physical level, it also takes care of the global infrastructure
including the establishment and management of regions, availability zones
and edge locations. Not to mention the software and all the back end layers
underneath. Lastly, authentication using IAM is a shared responsibility. You're
in charge of configuring your IAM permissions and controlling access of your
users while AWS will ensure that their I roles and permissions are also secure.
Now, from this model, let's segue to the security level whereby the first line
of defense is the network aka the VPC layer, it's the first barrier against
threats. So you can think of it as a thick envelope that's physically protecting
whatever is inside the second layer is authentication and access control,
which is composed of the IAM responsibility, which is shared between you
and AWS. It's like the address and recipient information written on the
envelope to specify who is authorized to open and read the letter inside.
Likewise, I am determines who is allowed to access what within the network.
The third layer is encryption. So if the letter in your envelope had a coded
message of some sort, you would have a third line of defense. This way, if
someone were to open and intercept the envelope, they wouldn't understand
the message without having access to the decryption key. Now let's unpack
each layer starting with the network S3 is accessible over the internet, which
means that you can access your buckets from anywhere using an internet
connection. Basically, you don't need to be within your organization's
network because S3 operates independently of V PCs. Having said that
though you still have the option to configure your buckets to block all public
access. If this option is selected, no one from the public internet would be
able to access your buckets even if they have the correct access control lists
all set up. The only way to access S3 with such a configuration would be
through VPC endpoints. These grant you isolated access by creating a private
connection between your VP C and your buckets. Simply put VPC endpoints
allow you to access restricted buckets directly from within your VP C without
having to traverse the public internet. Next layer was access control. Now,
access control and S3 involves two main components. The first one is
identity based access control. These consist of AWS IAM policies that are
applied to IAM users groups or roles. So they get attached to identity related
elements. Basically, they finally control access based on the identity of the
user or entity accessing the S3 resources. For instance, this IAM policy allows
the IAM user with this particular ARN to retrieve objects from the specified S3
bucket. The second component is resource based access control. This can be
in the form of access control lists. These are represented using XML and
they're used to grant specific permissions to individual AWS accounts or
predefined groups. We say they're resource based because you get to attach
them to objects or buckets within three. So for instance, you can attach an
ASL to a specific object to allow a specific AWS account to read that
particular object. The second resource based option is bucket policies. These
are Jason based policies that get attached to S3 buckets to control access at
the bucket level. S3 bucket policies allow you to specify permissions for a
broader range of users or entities. When compared to IAM policies. For
instance, you can grant public read access to all objects in a bucket as shown
in this policy here. The effect field specifies whether the statement allows or
denies these specified actions. The principal field defines who this policy
applies to. In this case, we have a broad range of users because it's set to
allow everyone. The action here defines that this policy is in regards to
retrieving objects and that resource defines the bucket that this policy is
attached to. So in a sense, S3 bucket policies allow you to define precise
permissions for accessing objects within your bucket. For instance, here we
specified that this permission only applies to the git object action. You can
also share access across accounts by granting access to resources in your
bucket to other AWS accounts. You can deny unencrypted uploads and define
access based on various criteria including IP addresses and IAM users or
roles. Another way to manage access to your S3 data with more fine grained
granularity is through the use of S3 access points. Access points enable you
to define access policies that apply only to the data access. Through that
particular endpoint picture multiple access channels either cause you have
multiple applications or teams that need access to different subsets of data
within the same bucket. So let's say you have a medical application that
serves different teammates within a medical team. Here. A doctor's access
point, we'd allow you to have a bucket that's only accessible by the doctors.
Likewise, a nurse's access point would allow you to have a bucket that's only
accessible by the nurses. And while we're at it, we can also have an analytics
access point that grants you access to both buckets. Now let's talk
encryption, server-side. Encryption takes place within AWS with SSE the
encryption can be managed either by the client with customer provided keys
or by AWS with AWS key management service, client-side encryption takes
place outside of AWS typically within the client's application. So obviously
here, the client is always in charge of the encryption process. AWS only deals
with encrypted data and doesn't have access to the plain text or encryption
keys. server-side encryption is also referred to as at rest encryption. The first
option is offered through customer provided keys where customers provide
their own encryption keys during the upload process. But the encryption and
decryption operations are performed on the server-side within AWS. The
second option is through AWS Key Management Service which gives you
three types of keys. An AWS-managed key, an AWS owned key and a
customer managed key. The AWS-owned key is the default encryption option
here. AWS owns and manages the key and luckily it doesn't incur any extra
costs in the process. The AWS-managed key option stores the key in your
account. So technically you'll own the key while AWS manages it for you. And
unlike the own key option, the managed option will incur KMS charges.
Lastly, if you go with the customer-managed key, you'll be in charge of
everything. This key will be stored in your account and you'll be the one in
charge of creating owning and managing the key and you'll be charged for
this type of key. Lastly, it's worth noting that if needed, you can switch
between these different keys at any given time. Next type of S3 encryption is
in transit encryption. Not much to do here cause Amazon encrypts data in
transit by default, it uses SSL and TLS protocols to secure the connections
between the client and the servers. And for the last form, we've got
client-side encryption, which is fully managed by the client application. So
here your application would encrypted data using its own keys before the
data is even sent to AWS.

Amazon S3 Features and Capabilities

So what is S3 capable of? S3 versioning. This feature gives me peace of mind

knowing that a new version is always created each time an object is either
updated or overwritten. S3 replication allows you to automatically replicate
data from one S3 bucket to another, and S3 event notifications allows you to
get S3 to send notifications when certain events take place within your
buckets. So when I associated S3 versioning with peace of mind, I meant it. I
can't picture ever managing a website that's hosted on S3 without this
feature. Let's say I'm working on this bucket to update an [Link] page. If
I were to reupload the same file again and again with S3 versioning enabled,
S3 would store all versions of my object. This includes both the writing and
deleting operations. So when you delete an object instead of immediately
removing it from the bucket, S3 keeps a marker called a delete marker to
indicate that the object has been deleted. So to restore a deleted object, you
would just delete this delete marker. And there are two options of
configurations. Versioning on the objects level, this is useful for scenarios
where versioning requirements vary between objects within the same
bucket. So for instance, you may want to enable versioning for critical
documents, but not for temporary files or logs. Otherwise, if all documents
are important, you would go with versioning on the bucket level. And it's
probably obvious by now, but let's ask what's the point of versioning? Well,
can you tolerate a system that doesn't have an undo option? I can't. As a
matter of fact, I dislike cooking because it doesn't have an undo option. I just
need to know that I have the option to reverse actions, whether that's
restoring a deleted file or restoring a burnt dish. Now, you might be
wondering why would anyone opt out of this feature? Well, keep in mind that
this feature charges you based on storage usage, so unless you need to use
it, it's probably wise to turn it off, in which case, the files will have the
version set to null, and once enabled, this feature cannot be disabled, only
suspended, meaning that when you suspend the feature, this won't impact
the previous versions, it'll just stop it from versioning going forward. Next
feature was replication which comes in two flavors. We've got cross-region
replication. This is when the source bucket and the replica bucket are in
different regions, and we've got same region replication, which as the name
conveys is when the source and replica buckets are in the same region.
Cross-region replication use cases include disaster recovery. You can use
secondary buckets in different regions to failover to when you experience
outages in the primary region. Compliance and geo redundancy. Some
regulatory requirements mandate storing the data in specific regions, so if
your local users are in a different region, CRR allows you to replicate the data
to the compliant region. And you might be wondering why would anyone
store the data in the local region in addition to the compliant region, which
takes me to the next use case, reduced latency. CRR can improve your read
performance by replicating the data to a bucket that's closer to your end
users. It just helps them fetch the data faster. Similarly, same region
replication also includes disaster recovery. So again, a secondary bucket to
failover to, but this time it will be in the same availability zone, which is still
great for DR as long as you're not faced with an outage on the AZ level.
Same region replication can also serve in testing by syncing production and
test accounts in real-time speed. Lastly, caching. So you can leverage the
replicated data of frequently-accessed content into a caching layer. Now,
let's talk about speed. So how fast is S3 replication? To understand this, you'll
need to know that there are two types of replication, asynchronous
replication which only copies the files from the primary bucket to the replica
once the files have already been written to the primary bucket. As a result,
this option offers near real-time replication. Second type is synchronous
replication which copies the files to the replica bucket while they're being
written to the primary bucket. This approach enables real-time replication.
Can you guess which option is supported by S3? Well, S3 replication operates
asynchronously, meaning that it only supports near real-time replication. To
get started with replication you must enable the versioning feature in both
the source and destination buckets to bypass this error. Only once object
versioning is enabled will you be able to configure replication, and you'll also
need to grant the proper IAM permissions to your S3 bucket, and when it
comes to replication, keep in mind that turning this feature on won't replicate
the existing data, at least not automatically, it will only replicate the
newly-added object. So if you need to replicate existing objects or objects
that fail to replicate, you can use S3 batch replication. Now, going back to
delete markers. Let's say you turn replication on and then you delete a file
which will in turn create a delete marker to flag the file for deletion. By
default, this marker won't get replicated to the replica bucket. So to change
this behavior, you have the option to replicate delete markers from here.
Lastly, S3 event notifications. As the name conveys, this feature allows you
to configure S3 to notify you when certain events take place, such as
creating new objects, removing, restoring, or replicating objects, expired S3
lifecycle events, these are typically used to automate the deletion of objects
after a certain period of time, transitioned S3 lifecycle events, that's when
objects in an S3 bucket are moved from one storage class to another based
on predefined rules. Automatic archival events from S3 intelligent tiering,
this has to do with S3 access tiers and the automation of moving objects
between these tiers, tagging objects, and PUT ACL objects which refers to
the action of setting or modifying the access control permissions of S3
objects. In terms of configuration options, you can filter which objects to be
notified on, let's say by name. So for instance, you might only be interested
in files with the PNG extension. This is commonly used to trigger events
when thumbnails of images are uploaded to S3, and it's good to know that
you can have as many S3 events as needed and that it usually takes a few
seconds for notification to be triggered and sent. In terms of destinations,
you can forward notifications to services such as Amazon SNS, Amazon SQS,
Lambda functions, Event Bridge, which in turn can then send notifications to
various other services.

Amazon S3 Storage Classes

Before we delve into storage classes, let's discuss the different scenarios of
data access because these will set the stage for understanding storage
classes. Scenario number one. You need to frequently access the data, and
the requirements entail low latency and high throughput. So this is the
equivalent of the relationship that I have with my mic. It's mounted on my
desk because I use it on a daily basis. I just need quick access and quick
retrieval. Scenario number two. You need to infrequently access the data,
and you're okay with slightly lower latency, but still require high throughput
once accessed. This scenario reminds me of my relationship with my camera.
I don't usually use it on a daily basis, so I store it in a not-so-close drawer
where it's easily accessible, but out of reach from the kids. This way, I can
quickly retrieve it with no hassle. Scenario number three. You need to archive
the data, but you don't mind the high latency, even if it takes up to several
hours to retrieve those items. Moreover, you also have the bandwidth to
tolerate low throughput, and that's my relationship with my studio gear. I use
these maybe once a year, so they're tucked in the fourth floor of my house in
a super high shelf. So to obtain access, I would need to grab a ladder from
the garage, climb the stairs while carrying the ladder all the way to the
fourth floor, and once there, I would need to set up the ladder, climb the
ladder to retrieve the gear. And for these listed scenarios, S3 offers three
primary storage classes. Amazon S3 standard-general purpose is ideal for
the first scenario where you're frequently accessing the data and need low
latency. Common use cases include websites, mobile and gaming
applications, as well as analytics for big data. Amazon S3
Standard-Infrequent Access is for the second scenario whereby the data is
accessed less frequently, but still requires rapid access once retrieved, and
because it's a downgrade from the general purpose option, it costs less.
Amazon S3 Glacier is great for archiving or backing up data and it costs
much less in terms of storage. There are two types of infrequent access. The
first is Amazon S3 Standard-Infrequent Access or S3 Standard-IA for brevity.
It offers 99.9 availability in multiple zones. Great use cases for this storage
tier include disaster recovery, given that it has a multi-AZ capability and
backups. The second is Amazon S3 One Zone-Infrequent Access or S3 One
Zone-IA for brevity. This option offers 99.5 availability in a single zone. It's
suitable for storing secondary backup copies, but not so much for DR which
relies on having redundancy across multiple availability zones. Now, let's
unpack S3 Glacier types where we've got three retrieval options for
accessing archived data. The first is the instant retrieval option which offers
millisecond retrieval support. This is great for scenarios where you're working
with medical images, news media assets, or data that needs to be accessed
on quarterly basis. The second is flexible retrieval delivery which entails
keeping the stored data for a minimum of 90 days before it can be either
deleted or transitioned to another storage class. Otherwise, you would pay a
fee for moving or deleting the data. It comes in three flavors, expedited
which returns data in a 1 to 5-minute window, standard which returns data
within 3 to 5 hours, and the bulk option which is geared towards scenarios
where retrieval times can be longer and less immediate, so something
between 5 and 12 hours. This option doesn't incur retrieval charges beyond
the storage fees. Lastly, the deep archive option. This one entails keeping
the data here for at least 180 days, so obviously, it's great for long-term
storage of rarely-accessed data. Retrieval options here include standard
mode where retrieval time is 12 hours, this is the default option, and the bulk
mode where retrieval time is 48 hours. Okay, so now that we've covered the
three storage classes, let's talk about S3 intelligent tiering, a unique storage
option that automatically moves your objects between the different access
tiers in exchange of a small monthly fee. So what are these access tiers?
First on the list, there's the frequent access tier. Intelligent tiering will
automatically place objects here by default, and objects remain in this tier as
long as they're being accessed. Infrequent Access Tier. Intelligent tiering will
use this tier to automatically place objects that haven't been accessed for 30
consecutive days. Archive Instant Acess Tier will be automatically used for
objects that haven't been accessed for 90 days. Archive Access Tier can be
optionally used for objects that haven't been accessed for anything between
90 and 730 days. Basically to configure this option, the minimum in activity
is 90 days and the maximum inactivity is 730 days. Lastly, we've got Deep
Archive Access Tier. This is another optional configuration that allows you to
configure intelligent tiering to move data that hasn't been accessed for
anything between 180 days to 730 days. Now, I think I said the word retrieval
at least a dozen times in this lesson, so let's quickly touch on a retrieval
option using what is known as S3 Select or Glacier Select. Basically, you can
use the S3 Select feature in S3 Select enabled storage classes, and most
storage classes support this feature with the exception of S3 Glacier. This is a
CSV file called top-apartments that I uploaded to a S3 bucket, and here is the
option to query my data using S3 Select. When I click on this option, it
prompts me to specify the input settings, so things like the format of the file,
the type of delimiter used to split the columns, the option to exclude the first
line, and whether or not the file has been compressed. The counterpart of S3
Select when it comes to Glacier Storage is Glacier Select. Both options will
allow you to run simple SQL statements to retrieve filtered data. So going
back to my S3 file, if I were to configure the input settings and run the
generated SQL statements, the result would look something like this. This
operation is quick because S3 operates the filtering operation directly on the
server where the data is stored instead of downloading the entire dataset to
the client. This is 400% faster and 80% cheaper compared to client filtering
methods.

Amazon S3 Lifecyle Rules

So by now we know that S3 intelligent tiering automatically moves your

objects between the different axis tiers in exchange of a small monthly fee
and that it does this based on access patterns instead of relying on explicit
rules to make such transitions. So if you have a specific rule that you want
the system to follow when implementing such transitions, you would need to
use S3 lifecycle rules to configure how objects are moved between storage
classes and access tiers based on your own defined criteria. For instance,
you can have a rule to move objects that have a certain prefix in the S3 URL
path to transition to a specific axis tier. A common use case is to configure
objects, especially images, to be moved to the infrequent axis based on file
paths. For instance, we can have a rule to transition all objects with the tag
Cloud Gurus to S3 intelligent tiering. S3 lifecycle rules allow you to configure
specific operations to be performed on objects based on your defined
criteria. You can configure two main types of actions. Transaction actions
allow you to define when and how the objects should be moved from one
storage class or access tier to another. For instance, you can set a rule to
move certain objects to glacier for archiving after 1 year. The second type of
actions is expiration actions where you get to define at what age should
objects expire as in be permanently deleted. For instance, expiration actions
allow you to delete after a specific period of time, and this is usually done for
compliance purposes. And if you have versioning enabled, you can also
configure the expiration action to delete older versions of files after a certain
period of time. For more on this, check out the object lifecycle management
link provided in the Resources section. In terms of costs, as soon as the
object becomes eligible, fully lifecycle transition, or expiration action, billing
gets charged accordingly based on the new access tier or storage class. So if
an object was, let's say, set to expire based on an expiration action and S3
didn't immediately expire the object, the good news is that you won't be
charged for storage after the expiration time, regardless of whether it
expired on time or not. The only exception to this is if you configure a
lifecycle rule to transition to the S3 intelligent tearing storage class. In this
scenario, billing changes only occur once the object has transitioned to S3
intelligent tiering, and that's it for S3 lifecycle rules, so let's switch over to a
couple of use cases. In the first scenario, users upload source images to S3.
These images represent the original high resolution files that obviously
consume a ton of space. So to speed up the application, thumbnail images
are generated, and luckily the process of generating thumbnail images is a
quick straightforward process. Then analysis of access patterns and a few
discussions with the analytics team reveal that the users typically need
immediate access to the source images, but only for 60 days. After that,
users don't mind waiting up to 6 hours when accessing such images.
Similarly, the users need to be able to access the thumbnails for 60 days,
though they access thumbnail images less frequently than the source
images. So given this fact and the fact that it's simple to recreate
thumbnails, users are okay with the possibility of losing thumbnails. So to
address this scenario, you'll need different lifecycle rules for the source and
thumbnail images. For the source images, you'll need to store the images in
the standard access tier for 60 days. Afterwards, our lifecycle rule should be
configured to transition the source images to Glacier after 60 days. As for the
thumbnail images, well, it should be sufficient to store them in the one zone,
infrequent access tier. We don't need multi-AZ given that the users can't
afford to lose the images. And since the images are easy to create and don't
need to be immediately accessed, we can also have a lifecycle rule to expire
the images after 60 days. Next, let's transition to the second scenario. Here,
users need to be granted the ability to immediately recover from rare, but
disastrous accidental deletions. After 30 days and up to 1 year, the users can
afford to wait for the deleted objects for up to 48 hours. Therefore, for
lifecycle rules, you'll need to, first off, enable S3 versioning because this
ensures that the deleted objects are hidden via delete markers and can be
recovered if needed. In terms of lifecycle rules, you'll need to keep the
objects in the Standard access tier for the first 30 days. Afterwards, you can
transition the noncurrent objects, AKA the objects that are flagged with
delete markers, to the standard infrequent access tier. You can keep them
there for 1 year. This allows for quicker retrieval of deleted objects, so you'll
always be able to retrieve the objects in less than 48 hours, and then after 1
year, you can transition those objects to Glacier Deep Archive to save on
costs. And that's it for the scenarios, so let's switch over to Amazon S3
Analytics. Using this feature, you can generate CSV reports with actual
recommendations on how to transition objects based on several factors. This
includes access patterns which can be derived using access frequency, last
time the objects were accessed, and total data scanned. The feature also
provides detailed cost breakdown which allows you to identify areas where
cost savings can be achieved. So with all of this information, you can come
up with lifecycle rules to optimize your costs by transitioning objects
between standard and standard infrequent access using S3 lifecycle rules,
but keep in mind that this feature isn't available for objects that are stored in
the One-Zone Infrequent Axis or Glacier Storage class. And that's Amazon S3
Analytics in a nutshell, so for more on this, check out the link in the
Resources section.

Amazon S3 Security

The shared responsibility model is a foundational security-related concept in

AWS. It defines who's responsible for what, and there are different tiers of
models. Consider responsibility in terms of home ownership. When you buy a
house, the bulk of the responsibility shifts to you. If you've ever been a
homeowner, you know there's a ton of invisible labor that goes beyond the
mortgage payments. From landscaping to trash and recycling, there's just a
ton of maintenance to go around. In contrast, you're assigned much less
responsibility when you rent, and obviously, even less responsibility when
you rent a hotel room. Likewise, infrastructure services, such as EC2, will
require plenty of security tasks on your part. Granted, AWS will take care of
the security of the cloud infrastructure itself, such as the physical data
center, the networking and virtualization layer, but beyond that, you will be
in charge of the configuration and maintenance tasks, such as configuring
the firewalls, applying security patches, and installing security software. In
contrast, when you work with container services such as RDS, the load of the
responsibility starts shifting towards AWS. Yes, you're still in charge of
applying updates when necessary, but it's not as labor intensive as
infrastructure services, and when it comes to fully-managed services such as
S3 and DynamoDB, the load of the responsibility shifts drastically towards
AWS, given that the majority of operational tasks gets offloaded to AWS. With
S3, your responsibilities as a customer include managing and securing your
customer's data and implementing client-side encryption. While AWS would
be in charge of server-side encryption, network protection, and everything
underneath such as the platform, so the service itself, along with securing
the underlying infrastructure such as the operating system, network, and
firewall on the physical level. It also takes care of the global infrastructure
including the establishment and management of regions, availability zones,
and edge locations, not to mention the software and all the back-end layers
underneath. Lastly, authentication using IAM is a shared responsibility. You're
in charge of configuring your IAM permissions and controlling access of your
users while AWS will ensure that their IAM roles and permissions are also
secure. Now, from this model, let's segue to the security level, whereby the
first line of defense is the network, AKA the VPC layer. It's the first barrier
against threats, so you can think of it as a thick envelope that's physically
protecting whatever is inside. The second layer is authentication and access
control, which is composed of the IAM responsibility which is shared between
you and AWS. It's like the address and recipient information written on the
envelope to specify who is authorized to open and read the letter inside.
Likewise, IAM determines who is allowed to access what within the network.
The third layer is encryption. So if the letter in your envelope had a coded
message of some sort, you would have a third line of defense. This way, if
someone were to open and intercept the envelope, they wouldn't understand
the message without having access to the decryption key. Now, let's unpack
each layer starting with the network. S3 is accessible over the internet,
which means that you can access your buckets from anywhere using an
internet connection. Basically, you don't need to be within your
organization's network because S3 operates independently of VPCs. Having
said that though, you still have the option to configure your buckets to block
all public access. If this option is selected, no one from the public internet
would be able to access your buckets, even if they have the correct access
control lists all set up. The only way to access S3 with such a configuration
would be through VPC endpoints. These grant you isolated access by
creating a private connection between your VPC and your buckets. Simply
put, VPC endpoints allow you to access restricted buckets directly from
within your VPC without having to traverse the public internet. Next layer
was access control. Now, access control and S3 involves two main
components. The first one is identity-based access control. These consist of
AWS IAM policies that are applied to IAM users, groups, or roles, so they get
attached to identity-related elements. Basically, they finally control access
based on the identity of the user or entity accessing the S3 resources. For
instance, this IAM policy allows the IAM user with this particular ARN to
retrieve objects from the specified S3 bucket. The second component is
resource-based access control. This can be in the form of access control lists,
these are represented using XML, and they're used to grant specific
permissions to individual AWS accounts or predefined groups. We say they're
resource-based because you get to attach them to objects or buckets within
S3. So for instance, you can attach an ASL to a specific object to allow a
specific AWS account to read that particular object. The second
resource-based option is bucket policies. These are JSON-based policies that
get attached to S3 buckets to control access at the bucket level. S3 bucket
policies allow you to specify permissions for a broader range of users or
entities when compared to IAM policies. For instance, you can grant public
read access to all objects in a bucket as shown in this policy. Here, the effect
field specifies whether the statement allows or denies these specified
actions. The principal field defines who this policy applies to. In this case, we
have a broad range of users because it's set to allow everyone. The action
here defines that this policy is in regards to retrieving objects, and the
resource defines the bucket that this policy is attached to. So in a sense, S3
bucket policies allow you to define precise permissions for accessing objects
within your bucket. For instance, here we specified that this permission only
applies to the GetObject action. You can also share access across accounts
by granting access to resources in your bucket to other AWS accounts. You
can deny unencrypted uploads and define access based on various criteria,
including IP addresses and IAM users or roles. Another way to manage access
to your S3 data with more fine-grained granularity is through the use of S3
access points. Access points enable you to define access policies that apply
only to the data access through that particular endpoint. Picture multiple
access channels, either cause you have multiple applications or teams, that
need access to different subsets of data within the same bucket. So let's say
you have a medical application that serves different teammates within a
medical team. Here, a doctor's access point would allow you to have a
bucket that's only accessible by the doctors. Likewise, a nurse's access point
would allow you to have a bucket that's only accessible by the nurses. And
while we're at it, we can also have an analytics access point that grants you
access to both buckets. Now, let's talk encryption. Server-side encryption
takes place within AWS. With SSE, the encryption can be managed either by
the client with customer-provided keys or by AWS with AWS Key
Management Service. Client-side encryption takes place outside of AWS,
typically, within the client's application. So obviously here, the client is
always in charge of the encryption process. AWS only deals with encrypted
data and doesn't have access to the plain text or encryption keys.
Server-side encryption is also referred to as at rest encryption. The first
option is offered through customer-provided keys where customers provide
their own encryption keys during the upload process, but the encryption and
decryption operations are performed on the server-side within AWS. The
second option is through AWS Key Management Service which gives you
three types of keys, an AWS-managed key, an AWS-owned key, and a
customer-managed key. The AWS-owned key is the default encryption option.
Here, AWS owns and manages the key, and luckily, it doesn't incur any extra
costs in the process. The AWS-managed key option stores the key in your
account, so technically, you'll own the key while AWS manages it for you.
And unlike the owned key option, the managed option will incur KMS
charges. Lastly, if you go with the customer-managed key, you'll be in charge
of everything. This key will be stored in your account, and you'll be the one in
charge of creating, owning, and managing the key, and you'll be charged for
this type of key. Lastly, it's worth noting that, if needed, you can switch
between these different keys at any given time. Next type of S3 encryption is
in transit encryption. Not much to do here cause Amazon encrypts data in
transit by default. It uses SSL and TLS protocols to secure the connections
between the client and the servers. And for the last form, we've got
client-side encryption, which is fully managed by the client application, so
here your application would encrypt the data using its own keys before the
data is even sent to AWS.

Introduction to Amazon EC2

So what comes to your mind when I say real estate or rental to be more
specific? Do you picture a high rise, perhaps your last Airbnb rental, or have
you been in this journey long enough to the point where you picture EC2
instances popping up in the cloud? As you might know by now, the traditional
way of obtaining physical servers is both expensive and time consuming.
Instead, EC2 allows you to rent virtual servers, also known as compute
instances in AWS. The whole process is streamlined and can be achieved on
a short notice because the process of renting such servers is as simple as
launching EC2 instances, either via the AWS console or through APIs. And
when you're done with the instances, you simply terminate them. And so
beyond the fact that it's so much easier, other benefits of EC2 include the
fact that you only pay for what you consume, send commitments, as
opposed to having to invest in long-term and restricting contracts that
usually go anywhere between 3 to 5 years. Moreover, EC2 is pretty flexible.
You can grow and shrink your instances based on your capacity as opposed
to having to calculate your predicted capacity ahead of time in order to
determine the exact number and size of servers needed for your mission.
Lastly, it only takes minutes to provision and launch an EC2 instance.
Whereas, it can take months to select, purchase, and ultimately provision a
physical server. One of the most critical aspects of picking an EC2 instance is
the instance type because the type determines three things. One, how
powerful your virtual machine is going to be. For instance, if you're dealing
with a basic block, you would need to pick a micro instance. However, if
you're working with, let's say, apps that need to handle heavier
computational tasks, you would need to pick a larger instance. Two, how
many virtual CPUs will end up in your application? For instance, an M5 large
instance provides two virtual CPUs, while an instance type like the C5 for
xlarge would offer 16 virtual CPUs. Lastly, the EC2 instance type will
determine how much memory will be provisioned to your application.
Examples of EC2 instance types include general purpose instances. Now,
going back to our three points from the previous slide in terms of power, they
offer a balanced performance of compute, memory, and networking
resources. In terms of virtual CPUs, they also offer a balanced ratio
respective to memory and a balanced ratio of memory to virtual CPUs.
Therefore, these type of instances are great for achieving a more balanced
performance for diverse workloads. Examples include the t3, t3a, and t4
instances. Next, we've got the compute optimized instance, these offer high
computational power, a higher number of virtual CPUs in contrast to
memory, all while providing sufficient memory that would support most
average workloads while still prioritizing CPU, so this balance allows it to
serve applications that demand a lot of computational power. Examples
include c6g, c5a, and c5n. Next, we've got memory-optimized instances
where in terms of power, these offer a ton of RAM while offering a lower
number of virtual CPUs compared to the other instance types. For memory,
as mentioned, these can be configured to meet the highest memory
capacities making them great for memory-intensive applications. Examples
include the r5, r5a, and r5n instances. Lastly, for storage optimized
instances, their power is the high speed storage. They, too, offer less virtual
CPUs while offering moderate to high memory making them ideal for
providing low latency storage requirements in data-intensive workloads.
Examples include l3, l3en, d2, and h1, and although storage optimized does
indeed mean more storage, it's important to note that local storage offered
by any EC2 instance is non-persistent, so don't confuse compute with actual
storage. When we say that storage here is not persistent, this means that
when you terminate an EC2 instance, all data stored on the instance will be
lost permanently. To persist data storage beyond the lifecycle of an EC2
instance, you can leverage additional storage solutions such as EBS or EFS
volumes or even S3 buckets. Now, let's delve into EBS. So EBS belongs to the
Block Storage category where volumes get attached to individual EC2
instances. Such AWS services include the instance store and Elastic Block
Storage. Instance store here is more like cash, so it's mainly meant for
enhancing performance, more so than being a permanent storage solution.
Whereas, an EBS is what you would need if you were mainly after preserving
the data beyond the lifecycle of the EC2 instance. This is usually used for
storing operating system data and application files. On the other hand, file
storage offers shared file storage, so instead of being associated with one
individual instance, it can be mounted by multiple EC2 instances
simultaneously. Such services include EFS for Linux and FSX for Windows.
Use cases here include content sharing. Picture a team of web developers
that are collaborating on the same website, and so they need to access the
same set of static assets. The second use case is distribution of files, so for
instance, distributing large media files across a geographically dispersed
user base. Now, let's talk about the different types of EBS volumes. We've
got the general purpose which is designed for a balance of price and
performance, usually used for small to medium size databases, provisioned
IOPS, which allows you to specify a consistent level of IOPS, so such
applications include transactional processing systems or real-time analytics.
Next, we've got the throughput optimized. It's designed for
frequently-accessed workloads such as big data analytics platforms like
Hadoop or Spark or data warehouses. Next, we've got the cold HDD. These
are designed for infrequent access. Lastly, the magnetic option is designed
for low-cost storage when you've got large amounts of data that you rarely
access. In terms of pricing models, EC2 offers four pricing models, on
demand which allows you to pay by the hour or the second, most instance
types billed by the hour or partial hour, while the ones with higher
performance capacity or specialized hardware billed by the second. Next, the
reserved model entails that you commit to a specific capacity for 1 or 3 years
depending on the contract. And in exchange, you get a 72% discount on the
hourly charge. The spot model allows you to purchase unused capacity at a
discount of up to 90 percent. Lastly, the dedicated model offers physical EC2
servers that are specifically dedicated for you, so obviously, it's the most
expensive option.

Data Formats

Types of Data Formats

So I believe there are three organizational styles. The ultra organizer, take
my mom, for example, she's the Maestro of organization. She likes to align,
color code, and just have things marching in perfect rows. Then there's me.
My soul is always torn between chaos and order. So on good days, my closet
looks more like this. It's not a Pinterest-worthy type of closet, but it's not a
disaster zone either. Then there's me on hectic days. Any time I volunteer to
host an event or engage in a site project, my closet turns into what I refer to
as the Bermuda triangle of closets where everything in there just
mysteriously disappears, and guess what, data formats are just like us. Some
are structured just like my mom. Some are semi-structured like me on good
days, and some are unstructured like me on hectic days. So what exactly is
structured data? This is data that follows a predefined model or schema
through the use of tables, rows, and columns. The relationships between the
data elements are clearly defined, and so consequently, this data is easily
queryable. Simply speaking, just like my mom's closet, you walk in there and
immediately find what you're looking for. Examples include spreadsheets,
SQL databases, CSV files, and relational tables. Semi-structured data, as
mentioned, falls between the categories of structured and unstructured data.
It has some organizational elements, it follows a hierarchy through the use of
tags, keys, and attributes. At the same time, it doesn't adhere to a rigid
schema like structured data, so there is room for change, and you can think
of this using the closet analogy. So my closet is not too rigid, I can change it
over time. Unlike my mom's closet, which has like a ton of customized and
fixed compartments, so it would need to undergo a lot of drastic changes to
change over time. Having said that, hers is just easier to navigate through,
it's easy to go in there and retrieve and find items. So building on this
analogy, we can say that semi-structured data is less queryable than
structured data, but more queryable than unstructured data. Just keep in
mind that you might need specialized querying approaches compared to the
fully-structured data, so basically, just like my closet, it comes with its own
learning curve. Examples include JSON and XML. Lastly, unstructured data is
data that doesn't adhere to a specific schema because it's not organized into
a consistent format. It lacks clear relationships between the elements, and
consequently, is not easily queryable. Like just good luck retrieving anything
from my closet on those hectic days. Therefore, you need to do a bit of
preprocessing to analyze unstructured data. Examples include digital assets
such as images, audio, and video, social media posts, text files that don't
have a specific structure, and word documents. Now, let's explore a few
examples of common data formats starting with CSV, AKA, comma-separated
files that follow a tabular form that's similar to spreadsheets, and by looking
at one of these files, we can clearly tell that it falls under the structured
format umbrella. You can use CSV to exchange data between applications.
For example, you might export data from a database as a CSV file and then
import that CSV into another application. To interchange data in
programming tasks, such as reading and writing data files, so most
programming languages provide libraries to tackle CSV files. For example,
Python has the CSV module to read data from CSV files. Number three, to
perform quick data analysis tasks on small to medium datasets. Applications,
such as Microsoft Excel or Google Sheets, allow you to quickly analyze CSV
data and even turning it into charts or performing calculations. And you can
also use CSV to backup and archive data, especially when the data is
primarily tabular. For instance, you might configure your application to
export CSV files regularly and then have it stored in a secure location as a
backup. Another common format is JSON, which is text-based data that's
organized in a hierarchical fashion using key value pairs. Here is an example
of a JSON record. As you can see, it has key value pairs representing a
student's information. You can use JSON to configure files and settings. It's so
simple and readable. You can easily store these settings inside the keys and
associate the respective values accordingly. And values can have a variety of
data types. They can be strings, numbers, Booleans, arrays, objects, and if
needed, null. The simplicity of JSON also makes it great for exchanging data
between web servers and browsers, which makes it a common choice for
transmitting data in APIs. JSON is language agnostic, meaning that it doesn't
have language-specific dependencies. It works with all sorts of programming
languages, so it's great for applications that require high interoperability.
Lastly, similarly to CSV, JSON can be used for backing up and archiving data.
Next, we've got Avro which represents data in a way that is easy to serialize,
as in convert to binary and deserialize as in convert back to its original
structure. This is an example. Do you notice something? Doesn't it look
exactly like the JSON example? That's cause Avro has a JSON-like syntax. In
addition, Avro is also usually associated with a declarative schema that looks
something like this where as you can see, it defines the names of the fields,
as well as the data types of those fields. So when would you use Avro? One,
to achieve a compact binary format. Avro is designed for efficient
serialization which results in smaller payloads when you compare it to JSON,
which as mentioned is a text-based format. Two, Avro is great for big data
processing frameworks, such as Apache, Hadoop, Spark, and Kafka. For
example, let's say you're using Apache Spark to analyze data, Spark would
be able to easily read and process the data in Avro format because it knows
exactly how the data is structured thanks to the Avro schema that we just
saw in the previous slide. Three, you would use Avro for applications that
undergo regular schema changes. This can be done because Avro allows you
to add new fields and modify existing fields without breaking backward
compatibility. Lastly, you can use Avro when you need high interoperability
by ensuring that the data is consistent between different systems. To achieve
this, Avro supports data validation and strict typing, which means that fields
are constrained to follow specific data types. And lastly on the list, we've got
Parquet, which is an open source storage file format that's designed to store
data in columns. It's pretty similar to Avro, but unlike Avro which had
separate files for this schema and the data, Parquet combines both the
schema and the data into one file. You can use Parquet for tasks that require
data warehousing analysis for instance, analytical or aggregation queries,
given that the column-oriented approach optimizes these queries. Parquet
can be used with Apache Hadoop, Spark, Kafka, as well as Impala and
Redshift spectrum. Parquet is suited for applications that undergo frequent
changes. It's designed to modify without breaking backwards compatibility.
And lastly, it's suited for when you need selective reading. The
column-oriented design allows you to access subsets of columns. Both Avro
and Parquet are popular file formats that are widely used in Apache
ecosystems, given that both are well integrated with Hadoop, Spark, and
Kafka with the main difference being that Avro is supported by Impala, while
Parquet is optimized for Impala. Avro supports row-based storage, while
Parquet supports column-oriented storage. Lastly, Avro uses a JSON-based
schema to define the structure, while Parquet uses its own schema definition
language, which is not JSON-based.
Transforming Data Formats

So what is ETL without the ability to transform data? Which is why there are
several ways to transform data using Glue ETL. You can use Python Shell jobs
for quick manipulation of data. You can use Spark ETL jobs for transforming
data at scale, and PySpark or Scalar jobs for batch processing, as well as
stream processing. Python ETL jobs are suitable for simpler ETL tasks, so
mainly small datasets. It offers prebuilt libraries for transforming data to
other formats, such as from CSV to Parquet and vice versa. They have
libraries for aggregating data which can be used to generate reports and
libraries for reading and writing specific data formats. So for instance, you
can use the CSV module to read from CSV, the JSON module to read from a
JSON file, and so on and so forth. For a use case scenario, picture having a
CSV file containing a small dataset of customer data within an S3 bucket that
you need to read and later convert to Parquet. In this case, you can use the
CSV module to read the file and the PyArrow library to convert the data to
Parquet. If, however, your jobs require higher compute power, you can go
with Spark ETL jobs because Apache Spark is a powerful open source
distributed computing system. It's powerful enough that it can be leveraged
within Glue to perform complex transformations over large scale data. Such
Spark transformations include filtering data, this is when you select specific
records based on certain criteria, aggregating data, whether that's
calculating sums, averages, or counts across different datasets, and joining
data in a sense of combining information from multiple datasets based on
common keys. For this scenario, picture working with a large dataset of
ecommerce transactions in Redshift where you've been asked to extract the
data and then sum the sales across the products to identify the top selling
products. And so because we're dealing with a large warehouse here,
Amazon Redshift, this is where Spark ETL jobs come in. You can use them for
the aggregation piece, and if, let's say, you were also asked to convert the
output to Parquet, you can then leverage Python ETL jobs for the convergent
piece. Lastly, for batch and streaming jobs, you can use PySpark or Scala to
write jobs for both traditional batch processing, which is when you're
processing data in fixed size batches and streaming processing, which is
when you're processing data in micro-batches at regular intervals. For the
traditional scenario, picture owning a retail company where you need to
analyze sales data coming from your stores on a daily basis in order to
identify trends and patterns, and so you ask your store managers to send
you the sales data in CSV format at the end of each day. In this scenario, you
can use PySpark or Scala to read the CSV file into Spark dataframes and then
transform or aggregate the data accordingly. For the streaming scenario,
picture the same retail company, but now you need to analyze the real-time
clickstream data of your clients. Here, you can set up an Amazon Kinesis
Stream to produce regular micro-batches of the data, and then use PySpark
or Scala to ingest and process the streaming data in real time.

Databases

Introduction to Amazon DynamoDB

AWS purpose-built databases come in all shapes and sizes to cater to today's
diverse application needs, and one of the most robust and widely-used
options is DynamoDB. It's a NoSQL fully-managed non-relational database. It
supports both key value pairs, as well as document data, such as JSON,
HTML, and XML, so it allows you to easily adapt your tables as your
requirements change. But to grant you this kind of flexibility, it takes away
things such as the ability to perform joins and analytical queries. Also, access
patterns must be known ahead of table creation. It offers unlimited storage
size, it's super fast, we're talking about single-digit millisecond at any scale
kind of fast, and it offers microsecond latency if you use DynamoDB
Accelerator, also known as DAX. Simply speaking, DynamoDB is limitless in
terms of speed and scaling as long as you're okay with defining your access
patterns and letting go of certain capabilities. DynamoDB's notable features
include DynamoDB global tables. These allow you to replicate your data
across different AWS regions which enables you to provide fast and
responsive access to your users across the globe. Next feature is DynamoDB
Streams. This is a time-ordered sequence that captures all the modifications
you do to your database on an item level. It does this by maintaining a
time-ordered list of all the modifications made on your table. In terms of
scaling and availability, DynamoDB features partitioning capabilities that
enable you to scale your database seamlessly without much manual work on
your end, and it automatically replicates your data across three different
availability zones within a region, so this increases the availability and fault
tolerance of your database. Thanks to all these features, DynamoDB is ideal
for media and metadata stores, so its ability to seamlessly scale with low
latency makes it well-suited for storing and managing photos, videos, and
other media files. Retail and shopping experiences, these applications often
experience high traffic, especially during peak times such as sale events or
holidays, so DynamoDB's ability to handle millions of requests per second
makes it suitable for the high throughput requirements of retail applications.
Similarly, DynamoDB is suited for large-scale gaming platforms given that it
can handle the massive amounts of data in these applications. And beyond
these use cases, you can go for DynamoDB if you're dealing with online
transaction processing workloads, AKA, OLTP. Examples here include financial
transactions and the ecommerce platforms and gaming applications we just
talked about in the previous slide. You can also go with DynamoDB if you
need hierarchical data models since DynamoDB allows you to have a
schemaless structure. Examples here include employee directories,
department structures, and organizational charts. DynamoDB's autoscaling
features also make it great for fluctuating workloads. For instance, social
media platforms that have varying user activity levels throughout the day or
ecommerce sites with flash sales. Lastly, you can use DynamoDB for
mission-critical applications that can't afford any downtime. As mentioned,
DynamoDB is designed to offer high availability and fault tolerance, making
it suitable for applications that require continuous operation such as health
care systems and online banking applications. Now, let's unpack
DynamoDB's building blocks by comparing the NoSQL anatomy to the SQL
anatomy. First one is easy because tables are tables everywhere. Next, we've
got items which are the equivalent of rows or records in SQL. Attributes are
the equivalent of columns or fields in SQL. Pretty much the data elements
that you store in your table make up your attributes, and a unique group of
attributes will give you a single row which is a single item in NoSQL. It's also
worth noting here that a single item can't exceed 400 KB in DynamoDB.
Lastly, we've got primary keys. These can consist of one or two attributes,
they're mandatory, and obviously, they're the equivalent of primary keys in
SQL with the only difference being that primary keys in SQL consist of one or
multiple columns and they are optional. The other difference is that in
DynamoDB, you will be using your primary keys to retrieve the data. This is
why knowing access patterns is essential in our table design because we
need to have enough knowledge to be able to wisely select our primary keys
when creating the table. Now, let's picture an actor directory application that
has a list of actors along with their movies. It has an actor table consisting of
an actor column, movie column, the year a movie was released, and the
genre of the movie. In this example, something like that genre would
represent our key while the field, drama, would represent our value, hence
the naming key value pairs. And since this application revolves around actors
and movies, it makes sense to have a composite key consisting of both the
actor and movie attributes. So if I were to search by, let's say, the actor Will
Smith and the movie King Richard, I would get this single item which consists
of a group of unique attributes. So for example, the values 2021 and drama
are two different attributes that make up a part of our unique item. Now, let's
go back to features. The last feature we've got here is the time to live
feature. From a non-technical perspective, think expiry labels. Manufacturers
add expiry labels on products to inform consumers when to ideally throw
things away. Likewise, if you were to add a time to live attribute to your table
items, it will too act as a timestamp that will be used to tell DynamoDB when
to start getting rid of items. Behind the scenes, TTL compares the current
time to the defined TTL attribute of an item. If the current time is greater
than the item's TTL value, then the item is marked for deletion. And
DynamoDB automatically removes the expired item from the tables, as well
as indexes including local secondary indexes, or LCIs for brevity, and global
secondary indexes, AKA, GSIs for brevity, and it does so within 48 hours of
expiration. Now, you don't need to know a ton about indexes and types of
indexes for the exam, but if you're interested to learn more about local and
global secondary indexes, check out the link in the Resources section. What
you need to focus on though is the feature itself and how it works behind the
scenes, and when we say within 48 hours of expiration, this means that you
might end up querying data that's expired, but hasn't been deleted yet. To
avoid this, use the filter operations to exclude such items from your result
sets. In terms of use cases, you can use TTL as a mechanism to delete
sensitive data. This can be a requirement for a contractual agreement with
your clients or partners, or it could be a compliance requirement. For
instance, you might have an obligation to remove user's data after a year of
inactivity within the application. You can also use TTL to delete sessions or
event logs that are no longer necessary. This will help you reduce your
consumed storage and thus save you costs on the long run. Lastly, you can
also use it for debugging and analytics purposes. For instance, you can use it
to remove temporary logs once they're no longer needed, and for analytics,
you can use them to track usage patterns within specified timeframes.

Amazon DynamoDB: Dealing with Rate Limits and Throttling

DynamoDB's throughput is determined by one of two read and write capacity

modes, the provisioned capacity mode and on-demand capacity mode. The
provisioned option allows you to calculate and provision throughput using
read capacity units for read operations and write capacity units for write
operations. On the other hand, for the on-demand option, you don't need to
specify your throughput, and instead of using capacity units, DynamoDB
charges you in terms of read request units and write request units. Both
capacity modes have a maximum throughput limit of 40,000 read capacity
units or read request units per table and 40,000 write capacity units or write
request units per table. The provisioned mode offers consistent and
predictable performance where pricing is determined by the throughput
capacity that you provision ahead of time, so if you make a mistake with
your predictions, you run the risk of overprovisioning and paying
unnecessary charges or under-provisioning and being subject to throttling.
Therefore, the provisioned mode is usually great for applications with
predictable traffic and capacity needs and traffic that is more consistent in
nature and ramps up gradually over time. In contrast, the on-demand model
automatically scales up or down based on your demand. It follows a
pay-as-you-go pricing model, so it's more expensive per request than the
provisioned option, but at least you don't run the risk of over or
under-provisioning, and it's only subject to throttling if you were to exceed
two times the previous peak within 30 minutes. Therefore, this option is more
suitable for unpredictable traffic or new tables with unknown workloads. In
terms of scaling in the provisioned mode, you can either enable autoscaling
where you would specify the minimum capacity units and maximum capacity
units to control the costs or disable autoscaling and only stick to defining the
provisioned capacity units. And generally speaking, it's recommended to
switch to provisioned mode once your app is steady, which you can do at any
time. On the other hand, with on-demand, no capacity planning is needed on
your end. All you need to do is make API calls. You don't even need to worry
about idle tables because these are only charged for storage and backups,
but not for read and write operations. And you can easily and seamlessly
switch to on-demand when your app is facing a lot of fluctuations. Just keep
in mind that you can switch to on-demand mode once every 24 hours. Now,
let's unpack the risks associated with provisioned mode and ask what would
happen if you under-provisioned throughput capacity. Let's say you send
reads or writes at a higher rate than your provisioned capacity and get to a
point where your consumed capacity is more than your provisioned capacity.
In this scenario, your database will throttle and throw the provisioned
throughput exceeded exception, and this is where burst capacity comes to
the rescue. Burst capacity is one of DynamoDB's unique features. It
occasionally comes to the rescue by providing occasional bursts or spikes. As
a result, read or write requests that would typically throttle and fail end up
succeeding. And you're probably asking, where does it provide this capacity
from? Well, the answer is that when your throughput is not fully used in a
given partition, DynamoDB retains up to 5 minutes, so 300 seconds worth of
that unused capacity. You can say it acts like a piggy bank collecting unused
capacity to use at a later time. Having said that though, think of this as your
backup plan, so don't rely on it entirely. A well-thought database will need
more reliable solutions to avoid throttling in the first place. Such solutions
include changing the table capacity mode to on-demand mode, otherwise
increasing your read and/or write capacity units. You can also consider using
AWS application autoscaling. This will automatically adjust the provisioned
throughput and allow you to meet surges in traffic. You should also look into
implementing retry logic to better manage failures. So if something is going
to fail a couple of times, you should at least give it a breather between the
failures. In DynamoDB, exponential backoff is a specific strategy used in retry
logic. It allows you to determine the time delay between retrying the failed
operations. You can also refine your query or scan operation to ensure that
you're retrieving the data efficiently because if your query retrieves
unnecessary data or irrelevant attributes, then you're wasting your capacity
units. So to avoid querying unnecessary data, you can use WHERE clauses,
and if you're interested in specific attributes, for instance, let's say you're
only interested in retrieving the top scores attribute from this table, you
would go with the projection expression feature to only retrieve the top score
attribute. Lastly, to resolve throttles, you can look into examining your
access patterns and your partition key to make sure you're not dealing with a
hot partition, which is when one of your partitions is receiving a
disproportionately high volume of read or write requests compared to the
other partitions within the same table.

Amazon DynamoDB: PartiQL

One of the coolest things about accessing DynamoDB through the console is
the ability to interact with data right there on the spot without having to
install any client applications on your local machine. PartiQL Editor is one of
the tools that allow you to query DynamoDB through the console. It allows
you to use SQL-like syntax to interact with DynamoDB tables. And since most
developers are familiar with SQL, PartiQL simplifies the learning curve of
transitioning from relational to NoSQL databases. It supports common SQL
statements, such as insert, update, select, and delete, and as you can see at
the query results section of the screen, you have the option to obtain the
output in either a table view format or JSON view format. And aside from
using the DynamoDB console to access PartiQL, you can also access it
through the AWS command-line interface, through the DynamoDB APIs, and
even through NoSQL Workbench, which is a standalone graphical tool that
you can download and install on your local machine. So in a sense, this is
pretty magical given that all you need to do is type in simple SQL-like
language without having to worry about translating your queries into
DynamoDB operations because DynamoDB takes care of this translation
behind the scenes, except it's not 100% magical because your queries don't
always get translated to efficient DynamoDB API operations. To understand
this, let's take a step back and understand scans and queries. Scans are the
equivalent of doing a SELECT * from a table in SQL. The specific command in
DynamoDB is aws dynamodb scan --table-name. These are super expensive
because even if you use filter expressions to limit the number of returned
items, it still would introduce the consumption of your capacity units because
all it does is discard the filter data after having consumed it, and if performed
on a large table, it can use up your entire provisioned throughput in a single
operation, which is why it's wiser to use queries instead. Those operations
allow you to retrieve specific items in a table using the primary key. It uses a
similar syntax to the scan operation, but instead of the scan command, it
uses the query command. And so because the query command works on
targeting specific items, it consumes less throughput which then makes it
less expensive than scans. Now, let's switch back to PartiQL. How does one
prevent PartiQL statements from getting unnecessarily translated to scan
operations? Well, you have a few options, such as utilizing AWS IAM to deny
the scan operation altogether. You do this in the identity that's being used to
run the PartiQL statements. Another option is to create secondary indexes
and write queries that use these indexes. Lastly, it's a good practice to
regularly monitor the performance of your queries and analyze any full scans
that occur to catch issues early in the process.

Amazon Redshift Distribution Styles

Redshift distribution styles. These determine how the data is distributed

across the compute nodes within a cluster. We've got four styles, the even
style. Here, the leader node tries to spread all the data evenly across all the
compute nodes regardless of the values in any particular column. In contrast,
for an uneven distribution, we have the key option. Here, the data is
distributed based on values whereby identical key values are placed on the
same compute node. Next, we have the all option whereby the entire table is
replicated in each node, so basically all the nodes store a copy off the table's
data. Therefore, it's not ideal for frequently-written tables. Lastly, if you don't
specify a distribution style, Redshift will use auto-distribution where it
automatically identifies the best-suited style based on the size of the table.
For instance, it might initially assign the all style to a small table and then
change it to an even style as the table grows. So when should you use each
style? It's recommended that you go with the even style if your table doesn't
participate in JOIN operations because the even style is well suited for
balanced workloads while JOIN operations can cause data skew. In contrast,
go for the key style if your queries are heavy with JOIN operations, as long as
you have a clear choice for a suitable distribution key. If however, you don't
have a key that can be used, you can go with the all style. Just keep in mind
that this one is more suited for small, fact/static tables that don't undergo a
lot of frequent changes because each time we change anything on this table,
the write operation would have to take place on each node in our cluster.
Lastly, if your table is unpredictable or you're simply not sure which style to
go with, opt in for the auto option because it's well suited for tables that are
likely to change in size over time.

Amazon RedShift Workload Management (WLM)

Now, let's talk about the Redshift workload management feature. So picture
this, you're a cashier in a large retail store overseeing a single queue. While
the setup seems straightforward, it's not free of challenges. Occasionally,
customers with just a few items ask to pass ahead of those with giant carts,
and this causes a lot of delays, not to mention that it can get pretty dramatic
when a toddler at the back of the line starts having a meltdown when they
find out they won't be taking any of those tempting impulse items, and if this
wasn't frustrating enough, imagine discovering that not just one, but two of
your colleagues are sitting idle at their empty registers because none of the
customers know that additional checkout lines are open. To optimize this,
your manager comes up with a new system. So now, picture a similar setup
with a different configuration. Your colleague gets assigned the express
queue to handle transactions of 10 items or less, and your most patient
colleague is tasked with assisting families with young kids. This clears up
your queue and leaves you with customers that have more than 10 items.
Sounds pretty magical, doesn't it? Well, this is where Redshift workload
management comes in. Thanks to this robust feature, you can design a
system that prioritizes queries by categorizing them into distinct queues. You
can create a queue for long-running queries, one for short and fast running
queries, and WLM also allows you to categorize queues based on user roles.
For instance, you can have a queue that's specific for the data analytics
team. Lastly, by default, WLM includes a super user queue that caters to
critical system operations such as administrative tasks, system maintenance,
and troubleshooting activities. So the main purpose of WLM is to prevent
long queries from holding up short, fast-running queries or prevent ad hoc
exploratory queries from holding up, let's say, high-critical queries. To get
started with WLM, you would need to set it up through parameter groups.
AWS uses these database configuration parameters to control the behavior
and performance of your databases. AWS uses these parameters to allow
you to configure the behavior of your databases. Here is my Redshift
console. I'm using a custom parameter group called WLM parameter group to
configure and prioritize my queues. As you can see, the second tab here has
the workload management feature. This feature allows you to have up to
eight queues with each queue having its own concurrency level to determine
the number of concurrent queries. So for instance, if the concurrency level is
set to one, only one query will be able to run at any given time. If the level is
set to five, then five queries will be able to run at the same time, and so on
and so forth. There are two types of WLM modes. We've got the automatic
mode. If you're working with the default parameter group, you will get this
mode by default. If you want to switch to the manual mode, you would need
to create a custom parameter group. And so because I'm using a custom
parameter group over here, I see the option to switch to the manual mode
over here. However, it's worth noting that AWS recommends creating a
separate custom parameter group, even for your automatic WLM
configurations. Other differences between these two modes include that in
the automatic option, Richard takes care of managing the concurrency level
of each queue. It determines resource allocation, such as memory, based on
the workload. So if you have a demanding query, for instance, a query with
multiple hash joins between large tables, Redshift will lower the concurrency
value so that fewer queries run simultaneously. You also have the option to
customize what's known as the priority values to identify the relative
importance of each queue. You have six options to choose from. There's the
critical, highest, high, normal, low, and lowest. In contrast, in the manual
option, you need to manage the concurrency level yourself, whereby you can
have a maximum concurrency value of 50. This means that within a specific
queue, up to 50 queries can be executed concurrently at any given time.
Likewise, the sum of all concurrency levels for all user-defined queues also
have to be within 50, so this means that no more than 50 queries can run
concurrently within all the queues. And in addition to being able to create
your own queues, by default, Redshift will create two queues, one queues
that will have a concurrency level of five, thus enabling up to five queries to
run concurrently, and a predefined superuser queue with concurrency levels
set to one, so this means that only one query from the super user queue can
be executed at any given time.

Dense Compute versus Dense Storage clusters

So Redshift clusters are technically composed of a collection of compute

resources known as nodes, and there are two types of clusters. Single node
clusters that consist of a single node that combines the functionality of both
leader and compute nodes. You don't need to know the roles of
leader/compute functionality, but on a high level, leader nodes coordinate
the overall operation of the cluster while compute nodes take care of
processing the data and executing the queries. The second type is the
multi-node cluster which consists of one leader node and one or more
compute nodes, and there are three node types in Redshift, dense compute,
dense storage, and RA3. Let's start with unpacking dense compute. As the
name conveys, these nodes are optimized for computational performance.
They offer high CPU and memory resources. To do so, they prioritize compute
over storage, hence, why they offer smaller storage capacity and charge a
higher cost per terabyte of storage. On the other hand, dense storage nodes
are optimized for storage, so they offer a large amount of storage capacity
all while charging a lower cost in comparison to dense compute. So as you've
probably concluded by now, you should use dense compute if you require
intensive compute resources. Basically, go with dense compute if you require
high query processing, so this is likely to be a requirement if you're regularly
working with complex analytical queries, or if you need to run concurrent
queries or real-time analytics. And you can also go with dense compute if
you're dealing with memory-intensive workloads, given that these nodes
typically provide more memory per compute unit. On the other hand, dense
storage would be more suitable if you're dealing with terabytes or petabytes
worth of data, if you need a balance between compute and storage
resources, given that dense storage can provide sufficient computational
performance, along with ample storage capacity all for a lower cost per
terabyte of storage. Lastly, use dense storage if you don't mind a slower
performance, and to clarify this one here, dense storage is not inherently
slower than dense compute; however, they may exhibit slightly lower
computational performance in comparison to dense compute nodes,
especially if you come across workloads that heavily rely on intensive
computational processing or complex transformations. As for our third node
type, it's distinguished by its ability to scale, compute, and storage
independently, as well as the ability to automatically offload data to S3 if a
node happens to grow beyond the capacity of a local SSD. And generally
speaking, AWS recommends using this one over dense storage nodes. You're
only recommended to go with dense storage nodes if your workload is
relatively predictable and you want an easier way to track costs. To clarify
what I mean by this, let's take a step back and discuss the pricing structure
in these nodes. So first dense storage couples the charges of compute and
storage resources together, and so the cost structure is rather
straightforward. On the other hand, RA3 charges compute and storage
separately. As a result, you need to keep track of the costs of both the
compute node and the data stored in S3. Hence, why tracking the expenses
in dense storage might be easier than RA3.

Amazon RedShift Spectrum and Materialized Views

In terms of scaling Redshift, one option is to add Redshift clusters, also

referred to as concurrency scaling. This feature automatically adds more
compute power temporarily to meet the sudden spikes in concurrent read
requests. This is possible thanks to the fact that Redshift supports running
queries in parallel. You can also resize clusters by updating the nodes within
your cluster. For horizontal scaling, you would scale in and out by adding or
removing nodes from the cluster, and for vertical scaling you would scale up
or down by changing your node types. Another indirect scaling option is to
query data via S3 using the Redshift spectrum feature. This enables you to
query large volumes of data in S3 without having to wait for extract,
transform, and load jobs to obtain access to your data, so technically it
spares you from having to store that data in Redshift. Redshift spectrum
enables you to query exabytes of data from S3. Cool, right? But where it gets
really special is that it allows you to do so without having to load the data
into Redshift, so you're literally just querying data directly off S3. Moreover,
you don't need to worry about scaling. Redshift spectrum handles it all
behind the scenes for you. So does this mean that you don't need to use
Redshift at all in the process? Well, not really because you'll still need a
Redshift cluster to interface with Redshift spectrum. Also note that your
cluster must be in the same region as your S3 bucket as it will be sending
your query to thousands of spectrum nodes. Lastly, you can actually have
more than one Redshift cluster querying the same data in S3 at the same
time. Behind the scenes, this is how it works. Your Redshift cluster
communicates with Redshift spectrum, which in turn uses an IAM role to
connect to a data store that has information about your data. The data store
then sends the data back to your cluster to enable you to query the data in
S3. So basically, in addition to having a Redshift cluster, you'll also need an
external read-only table for each dataset you're trying to query. Instead of
physically moving or loading the data into Redshift, you'll create an external
table that references the data location in S3. These tables can support select
and insert operation, but they won't support update or delete operations.
Given that the purpose of these tables isn't to import data, but rather to
specify the data format, file location, column structure, and the information
needed for querying the data in S3, and as per the diagram, your external
tables will need to reside in a data store. For instance, you can use a Glue
Data Catalog in AWS Glue or you can use Amazon Athena or create an EMR
cluster that integrates with an external Apache Hive Metastore, and if you're
still wondering where is S3 in this architectural diagram? Well, remember it's
part of the data store because you'll need to provide the S3 path you're
trying to query when setting up any of these data stores. Now, what if you
want to combine the data from S3 with data from other external database
services? Well, this is where Redshift federated query comes into play as it
allows you to query data across various databases, warehouses, and data
lakes. You can use it to perform join queries to combine data from Redshift
with a variety of external databases such as S3, RDS, including both
PostgreSQL and MySQL engines, as well as Aurora, and again, including both
engines. Moreover, in addition to performing complex joins, you can also
apply quick transformation on the data, and thus, eliminating the need to
build an ETL pipeline to transform the data. Don't you wish your office had
such a view? Materialized views in Redshift aren't as breathtaking, but
they're cool nonetheless. To understand them, let's take a step back and
describe regular views which are virtual tables consisting of saved queries
that pull data from one or more tables. The results of these tables aren't
stored in the view per se, but each time you query the view, it will use the
saved query and retrieve the data from the underlying table. Think of your
Instagram feed or newsfeed, for instance. Each time you go to the timeline,
the app uses a saved query to retrieve the latest data from the underlying
tables, which is why your feed is always getting refreshed each time the
underlying tables are updated. Materialized views, on the other hand, create
snapshots, so they use the saved query to query the data once and then
store a physical copy of the result set in the view itself. You still query them
the same way you would query a table or a regular view, but they serve a
different purpose. Think of your saved posts for instance or bookmarked
articles. You retrieve it the same way you retrieve anything else, but it's not
the same as a streaming feed that gets refreshed around the clock. It's more
of a collage of pinned items in a sense, and so materialized views are more
suited for querying predictable and recurring queries. For instance, querying
your end of quarter reports would be a good use case for materialized views.
Now, let's use two tables, an employee table and a department table to
create a materialized view for employee details. To do so, we will simply type
CREATE MATERIALIZED VIEW, insert the view name AS, and then do a select
statement to select the data that we're after from the employee table and
the department table, which is done through the use of joins. Refreshing
views. So to keep the views in sync with the source table, you can either turn
AUTO REFRESH on to have the view updated each time the source data in
the underlying table is updated. The other option is to manually issue the
REFRESH command. The syntax here is REFRESH MATERIALIZED VIEW and
you insert the name of your view.

Migrating Data

Migration versus transfer. Migration is more like moving houses, the entire
house is getting packed and moved. Transfer, however, is more like moving a
box or a parcel from one place to another. AWS DataSync is a transfer service
that transfers files and objects between on-premise and storage services,
such as S3. One of the key distinguishing aspects of the AWS DataSync
service is that by design, the data between the source and target is
constantly synced. This is great for ensuring data consistency and availability
in real time. The second transfer service is the AWS Transfer Family, which is
a set of fully-managed file transfer services that are suited for standard file
transfer protocols, such as SFTP, FTPS, or FTP. One of the key advantages of
this service is its ability to integrate seamlessly with existing authentication
systems, such as Active Directory, or LDAP. This ability enables users to
leverage their existing user directories and authentication mechanisms, and
that's it in terms of transfers, so let's segue to migration, starting with the
planning process. You can plan migration in AWS through the AWS
Application Discovery Service, which is a component of the AWS Migration
Hub. It's mainly designed to assist you in discovering your on-prem
environment. It does this by scanning your on-prem applications to gather
comprehensive information about your applications. This includes their
configurations, interdependencies, and resource utilization, which you can
then use to determine the optimal migration approach. Application Discovery
Service, AKA ADS, offers two types of assessments, agentless discovery,
which is remote. Remember when all schools turn to virtual learning during
COVID? I didn't think I could possibly appreciate school teachers anymore,
and the year of teachers orchestrating classrooms virtually took my
appreciation to a whole new level. Likewise, with agentless discovery, you
don't need to install anything on your machine. Instead, it uses the AWS
Agentless Discovery Connector that scans your network and infrastructure to
gather information about your servers, applications, configurations, and
resource utilization, such as CPU, memory, and disk usage. So this form of
assessment is great for situations where the installation of additional
software AKA agents is not possible, whether that's due to security or time
restrictions, given that it can take a bit of time to install agents manually. The
second form of ADS assessment is agent-based discovery, which deploys
lightweight software agents referred to as discovery agents in your system.
This is more like teachers being physically present in a classroom, so just like
on-site teachers, these agents are able to continuously monitor and collect
more granular and more real-time data, and just like in-person education,
this is great for scenarios when more control and customization are needed.
Now, let's head over to migration services where we've got the Application
Migration Service. It focuses on application-level migrations, so it's ideal for
re-hosting, AKA lifting and shifting applications running on physical servers,
virtual machines, or other cloud environments into AWS, and you can
achieve this either by migrating applications to AWS or by replicating data
from source to target. Either way, it offers minimized downtime. The stages
of the migration lifecycle here include discovery which involves discovering
the applications in the existing environment, and then once the applications
are discovered and you identify the scope and complexity of the migration
project, the planning stage kicks in. This involves developing a migration
road map, and then before performing the final migration, it's essential to
validate and test the migration process in AWS. You achieve this by setting
up a separate test environment that closely mirrors the target AWS
environment where the applications will be migrated. This process allows you
to simulate the migration process and validate the migrated applications
without affecting the production environment, and the best part is that you
get to do this before doing the final cutover. Next migration option is the
snow family. This family is composed of physical devices that specialize in
migrating large data volumes. When you have a significant amount of data
to migrate, transferring it over the internet would be time consuming if not
completely impractical due to limited network bandwidth, and that's where
this family comes in. It includes the Snowball device which is great if you
have at least 10 TB to migrate. The other member of this family is the
Snowball Edge device, which is also designed for when you have more than
10 TB worth of data. It provides onboard compute resources that enable you
to process your data. This means that the Snowball Edge device is not just a
storage device, it also has the power to perform computations on the data it
holds. This can include tasks such as data transformation, analysis, or other
processing tasks, and therefore, Snowball Edge is suited for complex use
cases where you need both large storage and compute capabilities. Last, but
not least, is the Snowmobile device which is great if you have more than 10
PB worth of data.

Database Migration Service (DMS)

Database migration service is an AWS fully-managed moving service for your

data. It's meant for migrating on-prem databases to AWS. Picture a flatbed,
carrying a heavy workload from your source database and into your target
database, and you can say that it's a smart flatbed because you can also use
it to discover eligible source databases and then migrate them from the
source to the target database or consolidate a bunch of source databases to
a single target database, and either way, the source database typically
remains available during the migration. Most AWS migration services do this
to minimize downtime. Possible migrations that can take place within DMS
include migrating into the cloud, whether that's into AWS or another provider
such as Azure, from the AWS cloud to on-premises, and between on-prem
instances through an AWS cloud setup. To serve these scenarios, DMS offers
two primary methods for migrating data between databases or data
warehouses, one-time migration and continuous replication. In a one-time
migration, DMS moves the data from the source database to the target
database in one operation, and once the migration is complete, you can stop
DMS and your target database will become the new primary database. This
method is typically used when you need to migrate an existing database to a
new environment, such as migrating an on-prem database to the cloud. On
the other hand, with continuous replication, which is also known as CDC, or
change data capture, the changes are constantly synchronized between the
source and target databases in a near real-time fashion. To better
understand when to use such methods, let's take a step back and explore
types of migrations. One, full load. Here, you migrate the entire database
from source to target. It's typically used for one-time migrations, given that
you're unlikely to migrate the entire dataset more than once, unless you
come across issues or discrepancies and need to redo the process, and this
option can result in some downtime. Therefore, full load is suited for initial
migrations and/or when the target table doesn't exist, it needs to be
completely overwritten. Next type is CDC-only, which stands for change data
capture. With this approach, you're only concerned with capturing changes
found in the source database and replicating them in the target database. It
typically yields in less downtime, and it's suited for syncing the target
database with the source without needing to do a full load. So you would go
with CDC if you've already migrated the data via a native tool, such as
pg_dump if you're working with PostgreSQL or if you've restored from a
backup snapshot. Otherwise, if you need CDC and haven't performed a full
load yet, you would go with the third migration type, which is a full load plus
CDC. As the name conveys, this is a hybrid approach where you perform a
full load for the initial migration and then replicate the ongoing changes via
CDC. And while we're on the topic of CDC, let's segue into the available
options for streaming CDC changes into S3. There are two options for
capturing and delivering these changes in real time. Option one is to stream
CDC data via Kinesis Data Streams. So first, the data is captured from the
on-prem database and into DMS via the CDC capabilities, and then the CDC
data is ingested directly into an S3 bucket where it's stored in Parquet
format. Pretty straightforward. The second option, however, includes the first
step as option one, but for ingestion, instead of directly ingesting into S3, the
data is ingested into Amazon Kinesis Data Streams where it's then streamed
into Amazon Kinesis Data Firehose. From here, you can apply more
sophisticated data transformation using Kinesis Data Firehose, and
ultimately, store the data in S3. And regardless of what migration method
you're working with, there are two migration types, homogeneous migration
where no schema conversion is needed, you only migrate the data because
the migration is between two compatible engines. For instance, if your
source database is an on-prem EC2 self-managed MySQL database, and your
target database is an RDS MySQL database. This would be an example of a
homogeneous migration. The second type is heterogeneous migrations
where you're dealing with different engines, so the migration ends up being
split into two phases. There is a schema convergence stage and a data
migration stage. For instance, if your source database is an on-prem Oracle
database, and your target is an RDS Postgres database, then you're dealing
with a heterogeneous migration, and you would need to use the schema
conversion tool offered by AWS to convert your source database schema and
make it compatible with the target database. The AWS schema conversion
tool is a standalone GUI application that you'll need to use to resolve
compatibility issues between your source and target engines. Only then will
you be able to go to the second stage and perform the actual migration from
source to target. So how does it all work? Behind the scenes, DMS is a server
that runs a replication software. It's an actual EC2 instance referred to as the
replication instance which runs within a VPC. On your end, you're in charge
of creating the replication instance, and put a pin in this one cause we'll
unpack it shortly. You're also in charge of configuring the source and target
connections to tell it where to load the data from and where to load it to.
Next, you'll need to create the target tables manually, unless you're using
the schema conversion tool to create some or all of the target tables, or
indexes, or views, or triggers, and so on and so forth. You'll also need a
database migration task, and to grant DMS access to the sources that DMS
will need to do its job, you'll need to create a specific IAM role with specific
permissions known as a service access role. Now, let's revisit the point that
we pinned earlier. Working with DMS revolves around creating replication
instances. As mentioned, the replication instance is merely in charge of
migrating the data using an EC2 instance, so this means that these instances
will need some sort of way to identify the source and target databases, which
is where endpoints come in. There is a source endpoint for the source
database and a target endpoint to connect to the target database. So if this
is our replication instance, it would be composed of a source endpoint and a
target endpoint, and from here, a replication task would be used to identify
the data that needs to be migrated, replicated by replicating the changes
from source to target, or synchronized by syncing the data in both source
and target. Now, let's talk about table mappings and transformation rules.
Table mappings help specify the relationships between columns from the
source tables to columns in the target tables. For example, you might have
email listed under email_address in the source database, and under
contact_email in the target database. So that's where table mappings come
in, they map such columns. Transformation rules, on the other hand, are
mapping rules that, in addition to specifying such relationships, allow you to
transform the mapped columns. Such transformation can include converting
data types, so you might have a string in the source database that needs to
be converted into a number in the target database, performing calculations,
so you can consolidate numbers from different columns in the source
database and then send them to compute a new column in the target
database. Altering strings, this can be achieved through a variety of ways,
including removing, adding, or replacing characters within a string. You can
also use transformation rules to combine strings. For instance, you can take
the first name and last name columns and then combine them to derive the
full name column in the target database, and that's it for table mappings and
transformation rules.

Data Cataloging Systems

Components of a Data Catalog

Data cataloging systems make a big portion of the exam. So what's a data
catalog system, and what is it used for? Well, just like the conventional
library, data catalog involves the systematic organization of books and helps
you discover, understand, and categorize library data easily. In compute
terms, data cataloging involves the systematic or organization of your data
assets, so in a sense, it's like a well-organized library for your metadata. It
provides all sorts of insights about the data such as where is the data stored,
so the location or path of the data, what's in the data? For instance, it would
tell you that a particular table is about customer sales and includes, let's say
columns, X, Y, and Z, who uses the data, which can give you insights on who
regularly accesses the information. It can even give you access patterns. For
instance, it can inform you that the sales department usually accesses the
customer's table, while the marketing team, let's say usually accesses the
campaign's table. How good is the data? This is usually referred to as data
quality. High quality indicates that the table undergoes regular validation,
has few missing values, and minimal duplicates or inconsistencies. How does
a piece of data relate to other data? This is also known as data lineage,
which helps you visualize where is the data being sourced from, and what
kind of transformational stages does it go through before hitting the final
destination? Examples of data catalog systems include the AWS Glue Data
Catalog and the Hive Metastore. So what is this catalog, this centralized
platform that organizes data built of? To answer this, let's list the main
components of a data catalog, one, metadata repository, two, search and
discovery features, three, data lineage, four, data asset descriptions, and
five, access and security. And we shall unpack all of these components
shortly, but first, let's discuss Apache Hive and the Hive Metastore. Apache
Hive is an open source data warehousing and SQL-like query language tool. It
allows you to create data warehouses on top of large-scale distributed
storage systems such as Hadoop distributed file system or Amazon S3. The
SQL-like language that allows you to query and analyze these distributed
environments is known as HiveQL. Hive integrates with EMR so you can
install and run Hive on EMR clusters. In terms of the data catalog
components, the Hive Metastore is the Hives Data Catalog and it stores
metadata about the tables, also known as Hive tables, the schema which
mainly consists of information about the columns and their data types,
partitioning information, and the storage location of the data associated with
each Hive table. Alternatively, if you don't want to rely on Hive's built-in
Metastore and prefer a cloud native catalog, you can leverage a Glue Data
Catalog to use as a Hive Metastore, which takes us back to data catalog
components. So let's discuss the data catalog components using the Hive
Metastore as an example. First component was the metadata repository. So
the Hive Metastore in this example, or any data catalog for that matter, acts
as a centralized repo that stores metadata for each dataset. Search and
discovery. Whether you're using the Hive Metastore or any other data
catalog, you should have the capability to search using database names,
table names, columns, and other attributes such as keywords. Tags and
annotations can also be used to provide additional context for search
capabilities. For data lineage, the Hive Metastore shows the flow of data from
its source and transformational stages all the way up to the final destination.
It also shows the relationships between data. For instance, the foreign key
relationships between columns in different tables. Next component was the
data asset descriptions. All tables in the Hive Metastore have descriptive
information, including the table's purpose, owner, and creation date. The last
component had to do with access and security, so basically information
about implementing access controls that ensure that only authorized users
can view, modify, or interact with the assets. Such information can include
access permissions both on the table and column level and security labels
that are used to categorize data based on confidentiality, all of which can be
queued and analyzed through the Hive Metastore. Now, let's segue to Elastic
MapReduce. EMR is designed for processing massive amounts of data,
commonly referred to as big data, which can be structured, semi-structured,
or unstructured. It's a fully-managed service, meaning that AWS takes care of
the underlying infrastructure, including provisioning, configuring clusters,
and managing resources. This enables users to focus on writing and joining
their processing applications to process large-scale data, given that they
don't have to worry about the operational details. In terms of scaling, EMR
allows you to scale the number of compute resources, also known as
compute nodes up or down based on your processing requirements where
you're only charged for the provisioned resources. And the magic of EMR lies
within its ability to use frameworks that can distribute and parallelize data
processing tasks for faster and more efficient processing. Such frameworks
include Apache Hadoop and Apache Spark. To better understand this,
visualize your EMR cluster in the form of a large warehouse that consists of a
team of workers coming together to process the inventory. The equivalent of
these workers is EMR's compute nodes. Each EMR cluster is made up of EC2
compute nodes that do the bulk of the work to process the inventory that's
received from vendors, and in the case of EMR, the data is sourced from a
storage system, whereby, this data gets transported to EMR clusters via the
Elastic MapReduce file system. Lastly, as mentioned before, the data catalog
system is compromised of the Hive Metastore or the AWS Glue Catalog if you
wish to go with that option.

Let's Look at Metadata

Metadata, simply put, is data about data. It provides essential contextual

information about the main data by providing characteristics and properties
such as its location, schema, including data types, and the relationships
between the data elements which is often referred to as data lineage.
Metadata plays a crucial role in data management and governance. Think of
it as the library data catalog that helps keep a library together. Just like a
data catalog helps you take care of managing the books and ensuring that
their quality is preserved, metadata preserves the quality and integrity of the
data. For instance, it specifies the schema and data types which can then be
leveraged to ensure that the fields across the dataset adhere to the correct
data types. Moreover, metadata also supports version control which allows
you to audit changes and ultimately enhance the integrity and reliability of
your data. Types of metadata include structural, which describes how the
data is organized. For example, in a relational database, structural metadata
may include information about the relationships between the tables which
you can then use to show data lineage. And generally speaking, structural
data also includes information about the table names, column names, data
types, and constraints. Descriptive metadata provides information about the
content and purpose of the data. It helps users understand what is the data
about and how can it be used. This includes attributes such as table
descriptions, table comments, and annotations that describe the type of
content, purpose, and usage of the data. All of these details enable effective
search and discovery capabilities. Administrative metadata focuses on the
management and administrative aspect of that data, so it captures details
such as ownership, access permissions, as in who has permission to read and
write to your dataset, who has permissions to read only, and so on and so
forth. It also captures versioning which assigns a version number to each
dataset and documents changes in the version history. Technical metadata
provides technical details, such as the storage format, so for instance, is it a
Parquet file, is it AVRO, JSON, or CSV? It captures details such as the usage of
serialization and deserialization libraries that handle the conversion between
these data formats, as well as indexing which tells you what type of indexes
are used and which columns are being indexed. Now, let's talk about the
uses of metadata starting with data lineage. Have you ever been bombarded
with time-sensitive events that rely on purchased items being present? And if
so, do you end up obsessively tracking the journey of your packages as they
get dispatched from the source of origin and all the way to their final
destination on your doorstep? The lineage of your shipment here is similar to
data lineage where source information pertains the origin of the data,
whereby sources can be databases, applications, or any external system.
Processes and transformation. So just like packages, data goes through
intermediate stages where it undergoes changes, such as data cleaning,
enrichment, or filtering. Storage locations. These can be databases or
warehouses that temporarily store the data throughout its journey. Lastly,
destination refers to the final storage where data is ultimately outputted to,
whether that's databases, warehouses, or applications. Now, let's explore
this concept visually. So let's say our data starts in an on-prem location in the
form of different file formats, such as TXT, CSV, and XML. And all these files
get extracted into our landing zone where it becomes ready for ingestion into
our data lake. Afterwards, the data is transformed and loaded to a target
warehouse such as Redshift, and ultimately, it's published into digestible
reports. Capturing this type of lineage helps you understand data flow, it
provides a clear understanding of the end-to-end flow of the data, it enables
impact analysis by illustrating how changes in one part of the data
ecosystem may affect downstream processes and reports. This way, when
your business user calls frantically to complain about a malformed column in
one of the reports, you can easily trace it all the way back to the on-prem
file. Compliance and auditing. Given that providing this detailed history of
how the data is handled across each step of the process helps with
transparency and accountability. Lastly, data lineage streamlines
troubleshooting and debugging data-related issues by identifying the specific
stages where problems may arise. The second purpose has to do with data
quality metrics, so these are quantitative measures to evaluate the overall
health and reliability of the information. Examples of the most fundamental
metrics include accuracy, which measures the percentage of correct data
values within a dataset. These indicate the degree to which the recorded
values align with the true or expected values. For example, if let's say
student A scores 90%, accurate data would show 90%. Inaccurate data
would have any other number or value besides 90%. Next metric is
completeness which measures the extent to which data is complete and
contains all the necessary values, attributes, or records. Basically, it
measures to what extent are the required fields being populated. So back to
our example, if certain grades are missing, this would drop the completeness
score. Consistency measures coherence of data values across different
sources, systems, or time periods. So if student A is shown to have 90% in
one system, but 80% in another system, this would drop the consistency
score. Validity refers to the percentage of valid data values that conform to
predefined rules. For instance, if the valid values of grades are only supposed
to be within 0 and let's say 100 and one of the grades is stored as A, this
would be an example of invalid data and would ultimately drop this score.
Lastly, uniqueness measures the percentage of unique metadata entries as
in the percentage of unique identifiers. For instance, if let's say you have the
same students with the exact same ID being listed to have had two different
grades on the exact same exam, this would be a violation of uniqueness and
would drop this metric. Benefits of these metrics include decision-making
because what would be the point of having data-driven decisions if they
aren't being drawn from accurate and reliable data? Two, saving costs. Data
quality metrics minimize the expenses that are typically associated with
errors, having to rework things, and just the inefficiencies caused by poor
data quality. Compliance. All these metrics ensure that you actually meet
regulatory requirements and ultimately reduce the risks associated with the
non-compliance, which takes me to the next point, and that is collaborating
with other teams to manage metadata. Tool number one for collaboration is
through adding annotations and comments to your metadata which allows
users to provide context and explain their content to other teams, such as
the data engineering team who might not be as familiar with the data, and
most AWS services, such as AWS Glue, support such capabilities. For
instance, you can add descriptions or tags to AWS Glue Data Catalog objects.
Second tool is change tracking which allows you to keep track of the changes
made to the metadata. For instance, you can use AWS CloudTrail to monitor
who is making metadata updates and when. Lastly, notifications. So of
course, for collaboration, you need to automate the process of alerting
specific teams when relevant metadata is updated, and you can configure
CloudWatch event rules to monitor when such changes take place in your
Glue Data Catalog or S3 Metadata buckets and then trigger notifications via
SMS or other notification services.

Demo: Creating a Data Catalog

It's demo time, so let's create a data catalog which, as you already know by
now, is a collection of virtual databases, tables, and other metadata entries.
To create one, you'll need at least one database, one table which you can
either create automatically via Glue crawlers or manually via the AWS Glue
Console or API, and regardless of which option you choose, you'll need a data
source to feed the table. We will use Amazon S3 for the sake of this demo.
Then once everything is created, you can retrieve the data via Amazon
Athena, Redshift Spectrum, or AWS Glue. We will use Athena for this demo.
And I want you to picture a hypothetical housing analytics application where
we will be creating a database for Manhattan's properties, and the source
data will be partitioned by neighborhoods, so let's switch to the source file.
This is what the source file that I'll be using looks like. It has six columns
pertaining apartments in Manhattan, price, the number of bedrooms, the
number of bathrooms, square footage, status, and address. This data is
partitioned by neighborhood, so I have two other files that are similar to this
one, whereby each one of these files represents a specific neighborhood in
Manhattan, and this particular file over here belongs to the East Harlem
neighborhood. Now let's head over to the console and jump over to Amazon
S3. I just used the service so it shows up over here, but you can also search
for it using the search bar, then click on it from the search results. Ahead of
the demo, I created the bucket manhattan-housing-analytics and created the
subfolders athena-output-results to host Athena's output results and
source-datafeed to store the source files. As mentioned, the data is
partitioned by neighborhoods, so I created three subfolders inside this folder
where I've got neighborhood=EastHarlem for the first partition,
neighborhood=Harlem for the second partition, and the
neighborhood=LittleItaly for the third partition, and I uploaded the respective
files to each one of these folders. If you wish to follow along, you can grab
these files from the link provided in the Resources section. The next step is
to navigate to AWS Glue. So using the search bar, I'll search for Glue and
click on it from the search results. First things first, let's start with building
the database. So over here, click on Databases, Add database. For the name,
I'll go with manhattan_insights_db and click on Create database. Here is our
database, so I'll click on it. Then we need to create our table. To do so, let's
try out AWS Glue crawler, so click on Add tables using crawler. For the name,
I'll go with manhattan_crawler, click on Next. It's asking if the data has
already been mapped to Glue tables. That's not the case, so I'll leave not yet
selected. To configure our data sources, click on Add a data source. The data
source is S3, so I'll keep S3 selected over here. To set the S3 path, click on
Browse S3 and navigate to our source-datafeed folder. Click on Choose. Now
it's complaining about the S3 path. I believe this is an error in the console
because all it takes is to just delete the forward slash, then type it back in
again for it to work. For the subsequent crawler once configuration, we will
only run this one time so it doesn't matter in our case, but you have the
option to crawl all folders with every subsequent crawl or to only crawl new
subfolders. It doesn't matter in our case, so I'll leave this as-is and click on
Add an S3 data source. The Classifier section helps us define how the files
are formatted, so click on Add new classifier. For the name, I'll go with
manhattan-csv-classifier, and for the type, I'll select CSV. Leave everything
else as-is and click on Create. The classifier has been successfully created, so
I'll go back to our crawler, hit Refresh, select the newly-created classifier, and
click on Next. For the IAM role, I need to create a new one, so click on Create
new IAM role. I'll call it Properties-Analyst, click on Create, I'll skip the
security configuration options, and click on Next. For the target database, I'll
select the database that we created earlier, and for frequency, I only want to
run this on demand, so I'll keep this as-is. Just keep in mind that minor costs
are associated with crawlers, so make sure not to set this frequency any
more than necessary. Click on Next. Review the summary page, which looks
about right, so I'll go ahead and click on Create crawler. The crawler was
successfully created, so I'll go ahead and click on Run crawler. This usually
takes a few minutes, so I'll pause and come back once the crawler is done
running. Alright, so the crawler is done running, the status has changed to
completed, so let's head to our table to make sure that it was created
correctly. Click on Tables, hit Refresh. Here is our table source_datafeed, so
as you can see, you use the same name as the folder in our S3 bucket, so
click on the table. It has the correct database, the correct location is set to
our S3 path, the classification is set to CSV, and the schema looks good. And
as you can see, we've got the neighborhood column set to be our partition
key. So everything looks good, and we're ready for the final step, and that is
retrieving data. So click on Actions, View data which will transfer us over to
Athena, so click on Proceed, and now in Athena, we see that the AWS Data
Catalog is selected underneath the data source, manhattan_insights_db is
our database, and here is our table with the correct schema and the partition
column. Inside the Query Editor, we've got the select query to retrieve the
data from the table, but before we can run this query, we need to set up the
path for the output results. It's also warning us here that no output location
has been provided. So to fix this, click on Edit settings, Browse S3, navigate
through our S3 folders, select athena-output-results, and click on Choose. Hit
Save, then go back to the Query Editor. Hit Run, and voila, here is our output
result, it has the correct data, correct schema, and the partition key set to
neighborhood. With this, we've come to the end of our demo. Don't hesitate
to reach out if you face any issues following along.

AWS Glue versus Apache Hive

Both AWS Glue and Apache Hive are data processing frameworks that
provide capabilities for processing and analyzing big data in data lakes. AWS
Glue is a data processing framework that offers fully-managed ETL services.
They simplify processing from the ingestion stage and all the way up to the
loading stage. You can use AWS Glue to crawl the data which infers the
schema and creates the AWS Glue Data Catalog which you can then use to
transform and load the data to your target database, and all of this can be
done via defining ETL jobs either through the console or custom Python
scripts to get Glue to ingest the data from various sources. Afterwards, the
data cataloging piece comes in. This enables Glue to crawl the data to create
and organize metadata that's then used to facilitate the extract, transform,
and load process, whereby, the transform stage can be achieved through
either built-in functions or custom code that transforms the data and then
loads it into the target data stores and analytical platforms. Likewise Apache
Hive is engineered for data processing and analytics. It facilitates data
transformations through a SQL-like language. You can use Hive-QL to write
SQL queries to process and transform data stored in either the Hadoop
distributed file system or S3. In terms of transformation, Hive-QL allows you
to perform operations such as filtering, aggregation, joining, and many other
operations that you can use to transform large datasets, and just like AWS
Glue, for data cataloging, Apache Hive incorporates a robust centralized
metadata store, and in the case of Apache Hive, it's referred to as the Hive
Metastore. And though both platforms serve similar purposes in the context
of data processing and analytics, they differ in their architectures and
deployment models. For instance, AWS Glue runs in AWS as a fully-managed
ETL service, while Apache Hive runs on Hadoop clusters. Glue handles
everything from resource provisioning to scaling and maintenance, so you
don't need to worry about infrastructure when working with Glue. In contrast
setting up and configuring Hadoop clusters requires significant effort and
expertise. So here, you're in charge of managing and maintaining the
infrastructure. In terms of processing, Glue supports both batch and
streaming processing. This flexibility allows you to process data in real time,
as well as in batch mode. On the other hand, Apache primarily supports
batch processing only, making it less suitable for real-time analytics
scenarios. In terms of ETL job translation, Glue translates your ETL jobs to
extract, transform, and load the data in AWS. On the other hand, Hive
translates Hive-QL into MapReduce jobs that are then executed on Hadoop
clusters. Glue integrates with various AWS services such as S3, Athena, and
RDS, so this integration allows you to leverage the full capabilities of the
AWS ecosystem. On the other hand, Hive integrates with other components
of the Hadoop ecosystem, such as HDFS, YARN, and Hadoop MapReduce. In
summary, while both AWS Glue and Apache Hive are valuable tools for both
data processing and analytics, their differences in architecture, deployment
model, and features make them suitable for different use cases and
environments.

Self Discovering Schemas in AWS Glue

A data lake is a centralized repository in enterprises mainly designed to

store, process, and secure large amounts of data in the raw format, so in
their native format. It acts as a single source of truth, so instead of having
multiple silos, you get to combine various forms of data from multiple data
sources into one single consistent data store. To achieve this, we use some
sort of a data integration process. An example of such a process is ETL,
which stands for extract, transform, and load. This is one of the common
processes that data engineers use to extract data from different sources,
transform the data into a usable and trusted resource, and then load that
data into the system that end users can then access downstream to solve
business problems. This is how it would look like. Data gets extracted from
different data sources into the lake where it's validated and transformed to
meet the company's data quality rules. Afterwards, it's loaded to the target
database where it gets published into APIs and used to create reports that
business users can easily consume. Now, using the same diagram, let's
explore how Glue fits in within the ETL process, and it all starts with S3 being
used as the storage layer for the data lake. From there, AWS Glue is
configured to crawl the data in S3 in order to infer the schema and create an
AWS Glue Data Catalog. Then using Glue ETL scripts, you can transform and
load the data into the target database. As you can see, the entire pipeline
here relies on the Glue Data Catalog, which begs the question, how does
AWS Glue populate the data catalog? Basically, when you configure your
Glue crawler, you have one of two options. You can either enable automatic
schema discovery, which allows the Glue crawler to analyze the data in your
sources, this was S3 in our previous example, and from there, Glue would
automatically identify the column names, data types, and other metadata
attributes without any manual intervention on your part, making this a
well-suited option for scenarios where your data is likely to change
frequently. Alternatively, you can choose to define the schema manually.
With this option, you get to specify the column names, data types, and other
metadata attributes explicitly and according to your requirements.
Obviously, this approach gives you more control over the schema definition,
so it's ideal for scenarios where your data doesn't change frequently or
where you have specific schema requirements that need to be enforced.

Optimization Techniques for Improving Query Performance

Three common techniques for optimizing your query performance, indexing,

partitioning, and compression. Indexing, as the name conveys, is similar to
conventional indexes. For instance, table of contents, phone books, or the
typical back of the book type of index. Indexes help you quickly find specific
information without scanning the entire book, and so similarly, when it
comes to tables, indexing enables you to avoid full table scans which are
both costly and time-consuming. Examples of applications that leverage
indexing include spreadsheets where rows and column identifiers are used to
reference specific cells within the spreadsheet. SQL databases. Here, the
indexes get created on columns within the tables. Lastly, relational tables.
Here, primary keys and unique constraints are examples of indexes that help
with facilitating faster retrieval. Next, we had partitioning which has to do
with dividing a large dataset into smaller subsets known as partitions. Again,
this also simplifies the process of finding items by reducing the amount of
data scanned, and so having more manageable chunks that can be
processed concurrently, which in turn enables parallel processing, which
obviously is faster than let's say processing tasks in a sequential manner.
Examples include partitioning sales data based on date ranges. For instance,
you might partition sales data by months, quarters, or years. Each partition
would then contain sale records that fall within a specific timeframe. And so
this way, if let's say you're interested in last year's fourth quarter, you would
go straight to that segment without wasting time and money scanning all
four quarters. Another example could be partitioning customer data
alphabetically. For instance, partition one could be for customers with names
starting with A, B, or C, partition two for names starting with D, E, or F, and
so on and so forth. Lastly, compression, this reduces the space needed to
store data. To do so, it encodes the data in a more compact fashion that can
later be decompressed when needed. Both compression and decompression
consume CPU resources, so just weigh the tradeoffs. Only opt-in for
compression if you've got ample CPU resources and the benefits of reduced
storage outweigh the overhead of the associated operations. Examples of
compression techniques include GZIP, ZIP, GZ, and RAR, also Redshift, which
is a fully-managed AWS warehousing solution, has its own compression
algorithms, namely LZOP, BZIP2, and GZIP.

Schema Evolution and Updating Data Catalogs

Change in databases is like the weather, you can try to predict it, but
sometimes you just have to roll with the storm. Similarly, when schemas
evolve, data catalogs need to roll with the changes by having corresponding
updates to their metadata. Therefore, there are different mechanisms for
updating data catalogs in AWS. The manual approach involves updating the
catalog using the Glue Console or by manually triggering the Glue crawlers
to rescan the data sources and update the catalog. The programmatic
approach can be achieved via AWS Glue APIs, AWS SDKs, such as Boto3 in
Python and ETL scripts such as the pandas or PySpark libraries in Python.
The automated approach is done by scheduling AWS Glue crawlers to rerun
periodically to ensure that your catalog is always up to date with your
schema, and you can also use AWS CloudFormation to automate the
deployment and update of data catalogs alongside other infrastructure
components. The most obvious way for evolving schemas is through
updating schemas. You can directly modify the schema by editing the table
while preserving the underlying data. Schemas also evolve through
partitions, so you can add or remove partitions by altering key value pairs. If
you're adding or updating a partition, you would need to update the
partition's metadata pertaining its location, format, or compression. For
instance, if you're adding a new partition based on the year key for, let's say,
the 2022 value, you'll need to specify the directory path where the data for
that partition will be stored, so this is usually an S3 path. Another way
schemas evolve is through updating tables, for instance, creating and
deleting tables, updating table properties and changing your table's
ownership, which refers to the AWS account or IAM role that has
administrative control over a particular table, so any of these changes would
alter your schema and entail a data catalog update. Lastly, indexes, which
are typically used to improve query performance. Playing with indexes will
alter your schema and will need to be reflected in data catalog. Now, let's
unpack the impact of schema changes. Let's say you delete columns. In this
case, you'll no longer be able to query those columns despite the fact that
the underlying data won't get deleted. In contrast, if you add columns, you'll
be able to query the new column in the query results, and to deal with
previously-extracted data, you can either populate the new columns with null
values, specify a default value, or populate the column using existing value.
Lastly, if you change data types, AWS will end up interpreting the column
according to its new data type which may result in data type conversion
errors or inconsistencies if the existing data cannot be cast to the new data
type, so it will be on you to clean such errors up, which takes us to the next
point, and that is the transformation piece. You can transform data via ETL
scripts. One form is custom transformations, think fancy custom closets that
are tailored to your needs. Custom transformations are user-defined
functions that you can create from scratch using Python or Scala. Then you
would package your script in the library, upload it to S3, and create an AWS
Glue job that references the S3 path of your package. These functions are
mainly meant for complex data transformations and for format conversion,
for instance, converting from JSON to AVRO and vice versa. The second form
of scripts is bundle transformations which are predefined functions, think
typical prebuilt out of the box type of closets, not the fanciest thing, but they
definitely do the job. You can configure these built-in operations through the
AWS Glue Console or Glue APIs, and these functions are typically used to
streamline common data processing tasks such as mapping, joining, filtering,
and aggregation. Examples of such bundle transformation operations, the
DropFields operation which allows you to remove specific columns from your
dataset. The DropNullFields operation, this one allows you to remove
columns from your dataset if all the values in those fields are null. Filter
allows you to filter rows based on specified criteria. Join allows you to
combine datasets based on common keys, so it's similar to SQL Join
operations. Map enables you to transform each record in your dataset using
a custom function, so you can use it to add, delete, or modify fields, as well
as perform external lookups to enrich the data. Lastly, we've got
ResolveChoice. This operation resolves ambiguities in dynamic frames, and
we'll visit dynamic frames shortly, but for now, you can conceptualize this by
knowing that it resolves issues when data inconsistencies arise. It starts by
identifying such ambiguities and allowing you to specify rules on how to
handle such inconsistencies. For example, picture having a price field that
doesn't always have the same data type. Sometimes it's a string, while other
times it's a float. Using ResolveChoice would allow you to provide guidance
across the board on whether the field should be converted to a string or a
float. Now going back to dynamic frames. So data frames and dynamic
frames are data structures that you're likely to work with when creating ETL
scripts. A data frame is a data structure that's composed of records as in
rows and columns. They're commonly used for batch processing tasks and
are mainly designed for structured data with defined schemas, so more like
relational tables and spreadsheets, but they can also handle semi structured
data and unstructured data to an extent. And in either way, in the case of
any schema changes, you would need to manually update the schema. To
work with data frames, you can use APIs that are provided by data
processing frameworks that support data frames such as Apache Spark.
Likewise, a dynamic frame is also a data structure, but it's composed of
dynamic records, so it's able to extend the capabilities of data frames.
They're specifically designed to handle semi-structured data. They support
nested data format such as JSON and XML, and they support schema
evolution without requiring manual intervention on your part. And given that
dynamic frames were introduced by AWS Glue, you can actually transform
dynamic frames using AWS Glue's built-in scripts. Lastly, another form of ETL
scripts is machine learning transformations. These are AWS Glue's
specialized operations that enable you to leverage machine learning
algorithms to perform various data processing tasks, such as deduplication,
records linkage, which refers to linking related records from different data
sources that refer to the same entity, and data quality enhancement. An
example of a machine learning transformation is the FindMatches operation.
It facilitates deduplication by identifying duplicate records. It does this by
determining the likelihood of two records representing the same entity
despite having variations in their attribute.

Conclusion
Summary

Congratulations. You've made it to the end of this course on data store

management as it pertains to the data engineer associate exam, so let's do a
quick recap of the main takeaways. In this course, we peeled the data pillars
of data store management by unpacking data stores, data formats,
databases, and data catalogs. Remember that data store management takes
up 26% of your total score, which is why it's crucial to know this portion
inside and out, both to pass the exam and to implement data solutions in the
real world. The first thing we talked about was data stores where we did a
deep dive into Amazon S3 and introduced EC2. For the S3 portion, we
unpacked various features, such as S3 versioning, S3 replication, and S3
event notifications. We also looked into S3 storage classes, namely Amazon
S3 standard for frequently-accessed data, Amazon S3 standard infrequent
access for data that's accessed less frequently, but still requires rapid access
once retrieved, and Amazon S3 Glacier for archiving or backing up data. We
also unpacked different subcategories of these storage classes, such as S3
Standard-IA and S3 One Zone-IA, and for S3 Glacier, we unpacked three
subcategories, namely the instant retrieval option, flexible retrieval, and
deep archive, and we saw how these options offered different retrieval
options. So for instance, the instant retrieval option offers millisecond
retrieval, flexible retrieval offers an expedited option, a standard option, and
a bulk option. Lastly, we saw how deep archive offers two similar options as
well. We explored S3 intelligent tiering, a unique storage class that helps you
optimize costs by automatically moving objects between the frequent,
infrequent, and archive instant access tiers, and remember that objects can
also be configured to transition between the archive and deep archive access
tiers. On a similar note, we looked into S3 lifecycle rules for moving objects
between storage classes and access tiers based on explicitly-defined criteria.
Next, for data formats, we explored when to use the different types and
explored how to transform between these formats using Python Shell jobs,
Spark ETL jobs, and batch and streaming ETL jobs through the PySpark or
Scala languages. For the database pillar, we explored the notable features of
various databases. DynamoDB's notable features included DynamoDB global
tables for distributed applications, DynamoDB Streams for capturing
item-level modifications, partitioning, and the built-in replication feature
which increases availability and fault tolerance. We also explored
DynamoDB's time to live feature, which is used to instruct DynamoDB when
to start getting rid of certain items pretty similar to expiry labels on regular
everyday items. For Redshift, we did a deep dive on Redshift's workload
management feature and saw how it can be used to prevent long-running
queries from holding up short and fast running queries. We discussed
Amazon Redshift spectrum, which allows you to query exabytes of data in S3
without having to load the data into Redshift. We said the Redshift
materialized views provide a snapshot of the result set of a specific
predefined query and that Amazon Redshift federated query is used to
retrieve data from Redshift while combining it with data from other external
databases. Then for transferring data from on-prem to AWS, we explored
transfer services such as AWS DataSync which transfers files and objects
between on-prem and storage services such as S3. We focused on
distinguishing factors to get into the habit of figuring out when to use each
transfer or migration tool. So for instance, we identified that AWS DataSync
is great for scenarios when continuous syncing is needed. Another transfer
service that we looked into was the AWS Transfer Family Service, which is a
set of fully-managed file transfer services. It's suited for standard file transfer
protocols and for scenarios when you need to leverage your organization's
existing user directory and authentication mechanism. For migration, we
looked into the AWS Application Discovery Service which is great for planning
migrations, the AWS Application Migration Service for migrating full
applications, and the Database Migration Service, DMS, and Schema
Conversion Tool, SCT, for migrating on-prem databases and for converting
your schema. Lastly, for the data catalogs pillar, we mainly focused on AWS
Glue and Apache Hive processing frameworks. We did a deep dive on the
similarities and differences of these frameworks to figure out when to use
each option with the biggest difference being that AWS Glue is fully managed
and overall requires less development effort. Whereas an Apache Hive
requires setup configuration and management of the underlying Hadoop
clusters. And with this, you're now one step, or shall I say, one lesson closer
to sitting and passing the AWS Certified Data Engineer Associate exam. Best
of luck with keeping up this momentum, and if possible, please let me know
when you hit that certification milestone.

Data Store Management: Exam Tips

Nothing beats practicing exam questions to prepare for this exam. I highly
recommend that you go through a couple of practice exams. So to wrap up
this course, let's go over three sample exam questions pertaining the data
store management portion of the exam. The first sample question is about
data catalogs. It says, a data engineering team has been tasked with
creating and maintaining a central metadata repository to be accessed via
Amazon EMR and Amazon Athena queries, and the repository must also
include the schema and properties for numerous tables. Some metadata is
currently stored in Apache Hive, and a team needs to import this data into
the central repository. The main ask is which solution minimizes development
effort. The provided choices are as follows. A, deploy a Hive metastore on an
EMR cluster, B, utilize Amazon EMR and Apache Ranger, C, employ the AWS
Glue Data Catalog, or D, implement a custom metadata import solution with
AWS Lambda and Amazon S3. Now there are different strategies to go about
solving this question. One way is to exclude the choices that come with
development overhead. For instance, it's fair to assume that the Apache
Ranger plus Amazon EMR, Hive metastore plus Amazon EMR, or AWS Lambda
plus Amazon S3 would come with a lot of development overhead. Therefore,
it's fair to exclude the choices A, B, and D. Thus, concluding that AWS Glue
Data Catalog is the correct choice. It's a fully-managed service that offers
automatic metadata discovery, and it seamlessly integrates with both EMR
and Athena, thus, meeting all the requirements provided in the question. The
next question is about optimizing queries. The question reads, you've been
tasked with optimizing the performance of Amazon Athena queries. You
observe that the data files used by Athena queries are presently stored in
uncompressed .csv format. Additionally, you know that users predominantly
execute analytical queries and aggregations by filtering data based on
specific categories. Which approach will most effectively improve the
performance of Athena queries? The choices are A, convert the data from
CSV to JSON format and utilize Snappy compression, B, apply Snappy
compression to the existing CSV files, C, switch the data format from CSV to
Apache Parquet and utilize Snappy compression, and D, employ GZIP
compression on the existing CSV files. Here, we can utilize a different
strategy by identifying the only approach that can meet the needed
requirements. This happens to be Apache Parquet, along with Snappy
compression in this scenario. First off, Apache Parquet is great for analytical
queries, aggregations, and data warehousing tasks. Moreover, Apache is a
columnar storage format that supports the ability to select certain columns,
which is ideal given that the aggregations are done by filtering data based on
specific categories. Lastly, this choice includes applying Snappy compression
to the Parquet files which further reduces the storage costs and improves the
efficiency of the queries. So with this, we can right off the bat conclude that
C is the correct answer. The next question is about purpose-built databases.
A business utilizes an on-premise Microsoft SQL Server database to manage
financial transaction data. Monthly, the company transfers this transactional
data from the on-premises database to AWS. Recently, the company has
observed a rise in the cost of migrating data from the on-premises database
to on Amazon RDS for SQL Server database. Seeking a cost-effective solution
for data migration to AWS, the company aims to minimize disruption to
applications accessing the database. Which AWS service should the company
employ to fulfill these criteria? The choices are, A, AWS Direct Connect, B,
AWS Database Migration Service, also known as DMS, C, AWS Snowball, or D,
AWS Transfer Family. Again, let's approach this one by flagging the wrong
tools starting with the AWS Transfer Family. This is a fully-managed service
that's meant for file transfers, so we can exclude it given that it's not
designed for database migration tasks. Next, we can exclude AWS Snowball.
This is a physical AWS device that's primarily designed for scenarios where
transferring large volumes of data over the internet would be inefficient or
downright impossible. Lastly, it's fair to exclude Direct Connect, given that it
doesn't directly address the tasks of migrating transactional data. So
excluding A, C, and D leaves us with the AWS DMS option, which indeed
meets the criteria. It offers a cost-effective solution for migrating data from
an on-premise Microsoft SQL Server database to Amazon RDS for SQL Server.
It minimizes the disruption by its ability to continuously replicate from source
to target. This guarantees that the transactional data changes will be
captured and migrated in a near real-time fashion. Lastly, AWS DMS is
capable of transferring transaction data from on-prem to AWS on a monthly
basis, which was also one of the key requirements. And with this, we've
come to the end of our course. You've come a long way in your learning path,
and you should be proud of yourself. Until next time, remember, all gurus get
there one lesson at a time.

Data Operations and Support

Introduction

Hello, everyone. My name is David Blocher, and welcome to my course, AWS

Certified Data Engineer-Associate Data Operations and Support. I'm a cloud
author here at Pluralsight. Managing data pipelines on AWS from ingestion to
analysis can be a complicated process involving many services and
technologies. In this course, we're going to learn how to automate data
processing, monitor our data pipelines, and ensure the quality of our data
along the way to analysis on AWS. Some of the major topics that we'll cover
include automating data processing with AWS services, data analytics on
AWS, and data pipeline monitoring and maintenance. By the end of this
course, you'll have everything you need to excel in the data operations and
support domain of the AWS Certified Data Engineer Associate exam. Before
beginning this course, you should be familiar with the basics of AWS
technology and services and general data storage and ETL concepts. I hope
you'll join me on this journey to learn data operations and support on AWS
with the AWS Certified Data Engineer- Associate Data Operations and
Support course at Pluralsight.

Automate Data Processing by Using AWS Services

Data APIs

Welcome to this module on automating data processing using AWS services.

When collecting data for processing, you will often have to interact with data
sources that are external to AWS. These could be another cloud provider,
on-premises data sources, or applications. And these data sources need to
find a way to interface with AWS services. Whenever one of your data
services is interfacing with an external data source, they're going to be
collecting data via a data API. Data APIs can allow data to flow freely and
securely between AWS services and external data sources. One example of a
data API is JDBC or ODBC APIs, that's Java Database Connector or Open
Database Connector APIs. These AWS services, Redshift, Athena, and Glue,
allow you to leverage JDBC or ODBC drivers in order to connect directly with
external data sources. Now, this really only works with relational databases.
Now, sometimes we may want to connect to external data sources that are
not traditional databases, and AWS provides a service that allows us to
connect to third-party Software as a Service platforms. People often want to
process and analyze data from Salesforce or SAP or Slack. To import this data
to your AWS data lake or data warehouse, AWS provides Amazon AppFlow.
This is a service specifically designed to integrate with these third-party
Software as a Service platforms and import that data into AWS. Appflow is a
fully managed integration service and it transfers data from Software as a
Service applications. Not only can you transfer this data, but you can filter,
enrich, and validate it to make sure that the data you're getting is in the form
that you want. Additional integrations include Google Analytics, Facebook
ads, ServiceNow, but there are literally tens and tens of these built-in
integrations that you can leverage right out of the box. So if you see an
exam question about analyzing data from Software as a Service applications,
or SaaS applications, make sure you remember Amazon AppFlow. And
another service that may show up on the exam is AWS Data Exchange. AWS
Data Exchange is a third-party data API marketplace. Data providers can host
their data APIs on Data Exchange, and then data consumers can subscribe to
and use the data APIs. You really only need to know Data Exchange at a high
level, it's really only likely to show up as a distractor on the exam. But if
you're training machine learning models or otherwise need high volumes of
quality pre-vetted data, Data Exchange is a great place to look. That's all for
data APIs. When you're ready, please join me in the next lesson.

Intro to Amazon EMR

Amazon EMR is a service that is primarily used to extract, transform, and

load petabytes of data. The name EMR comes from Elastic MapReduce. And a
map production is the process of taking a huge chunk of data, splitting it into
separate chunks of data for parallel processing, then each chunk of data is
transformed using its own source of compute power. The output of this
transformation is then reassembled to become a transformed version of the
original data source. The act of distributing and transforming the data is
called mapping, and then reassembling it is called reducing, thus, Elastic
MapReduce. In order to orchestrate this distributed processing, ERM
leverages Apache Hadoop. Hadoop is an open-source big data framework
that is specifically designed to facilitate parallel processing for big data
analytics. Hadoop leverages open-source tools for data preparation, as well
as data analysis and processing. Apache Hive is a tool for data warehousing
at scale where you can centrally interact with distributed data using an
SQL-like interface. Apache Spark is used to define data transformations at
scale. You can also analyze distributed data with Spark machine learning
library to define and train machine learning models on your data. And Presto
is an optimized big data query engine. Now, these are only a few of the many
open-source tools that integrate with Hadoop, but one of the hardest parts of
leveraging these powerful open-source services is managing the underlying
infrastructure, and that's where Amazon EMR comes into play. Amazon EMR
allows you to leverage the Hadoop ecosystem, but manages the cluster of
EC2 instances for running those big data operations. This way, the only thing
you have to worry about is defining your transformations and choosing the
size of the ERM cluster you'd like to use. Once you've configured your EMR
cluster, AWS will provision the underlying architecture for you. Let's take a
look at that underlying architecture. For the purpose of performance, EMR
clusters always exist in a single availability zone. Now, this may seem odd
because high availability best practices dictate that you usually want to
distribute things across multiple AZs, but in the case of EMR, performance is
so important that a single availability zone deployment is necessary to make
sure that all of the nodes are physically close to each other as to optimize
performance. However, you can achieve some degree of high availability.
When you configure your EMR cluster, you can choose to have a single
primary node or three primary nodes. Now the extra primary nodes are on
standby, they don't actually do anything unless there's a failure of the
primary node. You can think of the primary node as the conductor. It's
responsible for distributing data and tasks to the core nodes. Core nodes
host the distributed file system, and they run tasks on the distributed data.
So the primary node distributes data to the core nodes and then coordinates
task delegation to the core nodes as well. There is a third type of node that
can be used purely to amplify your processing power. These are called task
nodes. Task nodes do not take place in the distributed file system, but they
can be leveraged by the core nodes to help process the data. And task nodes
are really great for scaling in and out to optimize processing performance.
Now, there are a couple of different file systems that are available for EMR.
By default, the file system you'll be using is HDFS. This is the native Hadoop
open-source distributed file system where the primary and core nodes will
coordinate to store distributed data. Next, there's EMRFS, which is an
implementation of HDFS, but this integrates with S3. So while any data
stored in HDFS will go away when the cluster is terminated, data stored in
EMRFS can be persistent. And the third type of storage available when you're
running jobs on EMR is the local file system. This is really only used for
storing temporary data while jobs are running, so you may have caches or
buffers that you just want to store in the local file system, and this data will
never really be saved and stored with your other distributed data. Now, if the
configuration of an EMR cluster is too much operational overhead, they have
also recently introduced EMR serverless. So if you're not sure what instance
sizes or how many nodes you need to optimize the performance of your EMR
cluster, EMR serverless may be a really great option. You simply define the
runtime and jobs and AWS will size and provision workers when your jobs
execute. Another thing to consider when you're provisioning an EMR cluster
is whether you want the cluster to be ephemeral or long standing. An
ephemeral cluster will spin up when a job is initiated and then it will tear
down as soon as the job is complete. This can lead to huge cost savings
when compared to long-standing clusters. Long-standing clusters should
really only be used for jobs that involve indefinite streams of data or if
they're going to just be in close to constant use. Another situation where it
may be important is if you need the absolute best performance. You don't
have to wait for it to spin up. As soon as a job is initiated, a long-standing
cluster can begin processing that job. Another way you can think of this is an
ephemeral cluster is like renting a machine for limited use, whereas, a long
standing cluster is like building a factory that's constantly in use. of these
has their strengths and drawbacks, and you're going to be tasked with
choosing between ephemeral and long-standing clusters. You're also going to
need to understand the EMR Hive metastore. This is a central metadata
repository that stores data about your data. It keeps track of table schemas,
partitions, and data types across your distributed data. If other services or
applications need to interact with this Hive metastore, you're going to need
to create an external Hive metastore. The easiest way to do this is through
an integration with AWS Glue Data Catalog. EMR has a built-in feature that
allows you to use Glue Data Catalog as your external Hive metastore. You
can also use a database service like Amazon Aurora or RDS for your external
Hive metastore. AWS Glue Data Catalog is a great choice if you want to
natively integrate with other AWS services such as Amazon Athena or
Redshift Spectrum. While using Amazon Aurora or RDS for your external Hive
metastore might be the best choice if you want to expose it to other
third-party open-source applications such as Apache Ranger or Apache Atlas.
AWS Glue Data Catalog is also fully managed and highly available by default,
where you may need to do some extra configuration to make sure your data
stored on your RDS instance is replicated. So in many ways, EMR is used
much like AWS Glue is used for the extraction, transformation, and loading of
large sets of data. These two have some in common, but really there's a lot
that separates these two services. Let's start with what they have in
common. They both use Apache Spark to define jobs that transform data.
However, with EMR, you can leverage a whole host of open-source tools. So if
you already have a lot of experience with open-source big data processing,
EMR might be the right choice. If you want more built-in features and less
configuration management, Glue may be the better choice. Another huge
differentiator is that EMR is much more price performant than Glue. Meaning
if you have a really huge set of data or very intense processing that has to
happen, EMR might be a lot cheaper. So if you have to optimize for cost, EMR
is likely the correct choice. However, if you want to optimize for operational
efficiency and not have to worry about the underlying infrastructure at all,
AWS Glue is probably the right choice. If you're interested in using EMR, I
highly recommend you check out the relatively new EMR Studio. This is a
fully integrated development environment that helps you define and develop
jobs to run in EMR. This is not likely to come up on the exam, but it's a good
thing to know about. That's all for this clip on EMR. This has been David
Blocher, and when you're ready, I'll see you in the next lesson.

Intro to AWS Glue DataBrew

Once you've collected your data, it will reside in a data lake or a data
warehouse. And often to process a subset of this data, you'll want to migrate
some of it to an S3 bucket or some other data store for further analysis.
We've already seen that extracting, transforming, and loading this data and
preparing it for analysis is a difficult technical challenge. And often before
this data goes for analysis, you'll need to ensure its quality. So how can we
ensure the quality of the data? And what if we want data analysts to be able
to apply no code or low-code transformations without having to write full ETL
jobs? Fortunately, AWS has a solution for this. It's a service called AWS Glue
DataBrew. Glue DataBrew is a visual data preparation tool which is designed
to enable non-technical data analysts to filter, validate, and transform
incoming data. DataBrew provides no-code solutions for data validation and
detecting anomalies in the source data, and it even has over 250 prebuilt
transformations that you can use to clean and normalize the source data. You
can also use DataBrew to visually map your data's lineage. Let's start by
looking at how you can validate data quality with Glue DataBrew. Glue
DataBrew will sit between your data lake or data warehouse and the S3
bucket where you want to stage your data. A data analyst will leverage Glue
DataBrew to define data rules. These rules will continuously be applied to
incoming data and will not allow that data to transfer to the staging bucket if
it doesn't meet the data quality rules. DataBrew will then alert the analyst
about the malformed data and the analyst can remediate the source data so
that it passes the data quality check. Similarly, your analyst can define and
reuse transformations. So if you need to normalize a data column or
potentially split a column into multiple columns, an analyst can use no-code
and low-code tools in DataBrew to apply those transformations. Once
transformed, that data will be transported to the staging S3 bucket. Let's
take a look at the most common types of data transformations in DataBrew.
You can remove or replace missing values. You can combine different
datasets, you can create new columns. Potentially you want to split a time
stamp into multiple year, month, day columns. You can filter data, select only
a subset of data that you want to analyze. You can do label mapping, which
involves mapping each of several categories to a numerical value, for
example. And you can aggregate data. The biggest thing to take away for
the exam is that any time a scenario mentions the need to perform and
automate data transformations or validation rules without writing code, you
should think about AWS Glue DataBrew. That's all for this clip on DataBrew.
When you're ready, please join me in the next one.

Apache Offerings

If you're taking this course, it's likely you fall into one of two camps. You're
either experienced in data analysis and processing with open-source tools
and you're interested in implementing these on AWS or you're experienced in
AWS and relatively new to these open-source tools. This is a lesson to help
you understand how different AWS services implement open-source Apache
applications. We'll take a look at the Apache application and then any
services that leverage that application or act as alternatives to that
application. Let's start with Apache Flink. Flink is a super popular data
analytics application for streaming data. And AWS has Amazon Managed
Service for Apache Flink. So if you're streaming data using Amazon Kinesis,
for example, you can stream that data to Amazon Managed Service for
Apache Flink for a live analysis before it reaches its final destination. As an
example, perhaps you have a fleet of IoT devices that are streaming data to
Amazon Kinesis Data Streams. This data can be streamed to Amazon
Managed Service for Apache Flink, which can provide live analysis or
automated events before storing that data in S3, for example. Apache Flink
is capable of doing streaming ETL, applying minor transformations to data
before it's stored. It can give you continuous metric generation, real-time
analytics, and interactive analysis of data streams. Another popular Apache
offering is Apache Airflow. Apache Airflow is a workflow management
platform. You can use it to orchestrate and monitor really complex workflows.
If you want to migrate workflows that are already configured for Apache
Airflow, you can use Amazon Managed Workflows for Apache Airflow.
However, for more robust support of AWS services and less overall
operational overhead, most of the time AWS would prefer if you used AWS
Step Functions. While Step Functions are ideal for tightly sequenced
workflows, there are other alternatives for Apache Airflow for specific use
cases. For example, AWS Glue workflows can be used specifically for defining
ETL data pipelines. And Amazon EventBridge can be used for event
processing. EventBridge can receive events from many sources and then
push events to many targets. You can also use EventBridge to schedule
events. As mentioned, Glue workflows are specifically for defining ETL
pipelines where you have multiple transformations and staging steps.
Apache Kafka is an open-source distributed event streaming platform that is
optimized for real-time data streams. Once again, if you're migrating existing
Kafka applications to the AWS Cloud, AWS provides the Amazon Managed
Streaming Service for Apache Kafka. However, if you are building something
from scratch on AWS, it may be much simpler to use the Amazon Kinesis
family of services. Apache Hive is built on top of Hadoop for data
warehousing. And if you want to leverage Hive, you'll be using Amazon EMR.
Hive helps you interact with petabytes of data by allowing users to read,
write, and manage data using SQL. Apache Spark, on the other hand, is used
to define transformations. Spark is an engine for large scale data processing.
It can be used to process big data in parallel using Amazon EMR or AWS
Glue. For AWS Glue, Apache Spark is used to define ETL jobs. In this example,
we crawl an S3 bucket to create an AWS Glue Data Catalog. We use AWS
Glue to transform data that is stored there. We then output this to a new S3
bucket, create a data catalog for that bucket, which can then be analyzed
with services such as Amazon Athena. Similarly, with EMR, we can intake
data, which can then be defined and managed using Apache Hive. Not only
can we intake data from AWS, but we can also intake data from external data
sources using AWS DataSync. Once our tables are defined, we can execute
jobs using Apache Spark. So I hope this clarified how different Apache
offerings are leveraged within the AWS ecosystem. I'll leave you with this
exam tip. AWS will tend to prefer their fully managed and integrated
offerings. So if there is a specific requirement for using open source or if
you're migrating existing open-source applications, you may want to use the
managed Apache services. However, in cases where you are building things
from scratch, using the fully integrated AWS services is more likely to be the
correct answer. Take a minute to review, and when you're ready, please join
me in the next lesson.

Intro to Amazon EventBridge

Amazon EventBridge is an AWS service that's used for processing and

scheduling asynchronous events all across the AWS ecosystem. It can
natively ingest and deliver events to and from many AWS services, and it can
also be used to schedule automated actions. Events can be published from
AWS services, as well as API calls. Then you can use EventBridge to receive
these events and define rules for when to trigger automated actions with
other services and applications. In EventBridge, you'll define an event bus
and one or more event rules. When you receive an event from your source,
you can then route that event to one or more targets. As mentioned, you can
also use Amazon EventBridge to schedule events. That way, even without an
event source, you can generate an event to trigger scheduled events such as
triggering a Glue workflow or a Lambda function. Now, Amazon EventBridge
can be implemented in almost any event-driven architecture or pipeline.
However, there is often a more simple solution. Especially in the context of
this exam, be careful not to overcomplicate things with EventBridge. If you
need to schedule events or handle events from many sources, EventBridge
may be the correct choice. However, if you have a simpler workflow, such as
when a CSV file enters an S3 bucket, you can often cut EventBridge out of
the picture. For example, you can define an S3 event, which directly triggers
a Lambda function. Generally speaking, unless asked otherwise, you want to
look for the simplest solution that meets your needs. That's all for this clip on
EventBridge. When you're ready, please join me in the next one.

Analytics

Intro to Amazon Athena

Welcome to this module on data analytics in the context of the Data

Engineer- Associate AWS certification. We're going to start things off by
talking about an analytic service called Amazon Athena. Once your data has
been prepared and staged, you can then begin to analyze that data. One of
the most straightforward ways to do this is using Amazon Athena. Amazon
Athena allows you to directly query data that's held in S3 buckets using an
SQL interface. Amazon Athena is a fully managed, serverless, interactive
query service, meaning you never have to worry about the underlying
infrastructure. Athena is great for running ad hoc queries on petabytes of
data. Under the hood, it's using an open-source query engine called Trino.
You can also create tables and schemas using the Apache Hive Data
Definition Language. And optionally, you can run ad hoc Apache Spark
applications. This can be a good option for testing your Apache Spark
applications, but when implementing them, you almost always want to be
using Glue or EMR. So let's talk about the supported data formats that you
can query with Athena. Most commonly, you'll be seeing CSV files, JSON
blobs, or Apache Parquet. Apache Avro, and Apache ORC are also supported.
Now, when you're creating queries for petabytes of data, they can take hours
or even days to complete. That's why it's really important to think about
query optimization. There are many ways you can optimize your Athena
queries, but here are the main three. One is you can partition your data to
optimize it for analysis. You can compress your data, and you can use
columnar file formats. When you first create a table with Athena, you can
define partitions. Partitions help you group related data together based on
values like date, country, and region. By partitioning your data, you only
need to load the data that's needed for your query instead of loading all of
the data for every query. This one will appear a lot on the exam. One thing to
note is that when you define your partitions, data will not automatically
populate. For Hive style partitions, you'll have to perform the MSCK REPAIR
TABLE command. And for non-Hive or custom partitions, you'll have to run
the ALTER TABLE ADD PARTITION command. Hive style partitioning has a
key-value format with an equal sign between, and non-Hives style
partitioning can take a lot of different forms, and here's one such example.
Apart from partitioning your data, you may consider compressing your data.
Athena supports common compression formats. Most commonly, you'll see
gzip or snappy. JSON and CSV files, however, are not splittable, so you can't
compress these and then process them in parallel. Compressed Parquet and
ORC files are always splittable due to segmentation and metadata. Another
advantage of Parquet and ORC is that they are columnar file formats, which
are optimized for querying. So if you need to improve the performance of an
Athena query, consider first transforming your data into one of these
columnar data formats. That's all for Amazon Athena. When you're ready,
please join me in the next one.

Publishing Data Using Amazon QuickSight

Once you have isolated the data that you want to analyze, the next common
step is to create visualizations for that data. Amazon QuickSight is a service
that allows you to create and share visualizations from many different data
sources. When you provision Amazon QuickSight, you create an entire
serverless application where you can add users and share visualizations. So
your AWS account will have IAM users or IAM roles that give humans access
to your AWS resources. QuickSight will have an entirely different pool of
users which you can assign QuickSight permissions to. So in AWS, you may
have your architects and engineers, and then in QuickSight, you can give
permissions to administrators, data analysts, and business leaders to view
dashboards. Dashboards are a collection of visualizations, and you can give
different permissions to users to view different dashboards. In this example,
we have a car company that sells both gas cars and electric vehicles, so you
can create separate dashboards for each of these business units and give
permissions to view these dashboards to particular business leaders. Now so
far, we've only mentioned Athena and S3 as data sources for QuickSight, but
you can use many, many other data sources. Pretty much any relational data
source you can think of on AWS and even third-party data sources like
GitHub, Snowflake, and Jira can directly integrate with QuickSight for data
visualization. Now, in order for QuickSight to have access to your AWS
resources, it has to be granted an IAM role to be able to visualize that data.
One important nuance here is that when you want to grant QuickSight the
ability to visualize data from Amazon Athena, you will also have to apply a
role to grant that QuickSight application access to any underlying S3
buckets. This is a really common troubleshooting scenario with QuickSight
and Amazon Athena. Another common scenario you may encounter on the
exam is having to choose between Amazon QuickSight and Amazon
CloudWatch for data visualization. These have some things in common, but
they're really quite different if you know what to look for. So, Amazon
QuickSight allows you to create dashboards of visualizations with highly
interactive charts and diagrams. Amazon CloudWatch also allows you to
create dashboards and the visualizations are also interactive, although, not
quite to the same degree. QuickSight is optimized for business intelligence
data visualization, whereas, Amazon CloudWatch is optimized for cloud
resource monitoring and automation. QuickSight is AWS's business
intelligence service, and the tools provided by QuickSight are biased toward
making really visually appealing diagrams to be shared with business
decision-makers. QuickSight has a huge selection of visualizations, whereas,
Amazon CloudWatch has a very limited selection of visualizations. Another
big difference is that QuickSight is a separate application with a separate
user pool, whereas, with Amazon CloudWatch, you have to manage the AWS
permissions for the human users. Now that you understand the basics of
Amazon. QuickSight, please join me in the next one.

Visualizing Data

AWS provides many services to help you visualize data, and these services
range from collecting and processing data to storing and analyzing that data.
You can think of the services shown here as existing on a spectrum from data
preparation to data analysis. First, let's take a look at AWS Glue Studio. Glue
Studio is a visual interface to help you filter and transform data directly from
the AWS console. Using the visual editor, you can define your Glue pipeline.
Amazon Athena also has a visualization tool called Athena visual query
analysis. With visual query analysis, you can dig in to your query
performance. You can see each logical step of the query and how much time
it's taken. If your query needs optimization, you can use visual query analysis
to find the most impactful place to improve your Athena query performance.
When it comes to creating data analysis visualizations of your data, one tool
you can use is Amazon Redshift Query Editor. With version 2.0, you can
create visualizations directly from the query editor itself. This way, you can
create simple visualizations without having to provision an entire QuickSight
application. If you simply want an ad hoc visual or just a test to see that your
query is getting the results that you want, you can visualize the data in
simple visualizations such as line charts and bar charts. For more advanced
analysis and visualization, you'll want to use Amazon QuickSight. QuickSight
can connect to many data sources and create visualizations that are ideal for
sharing across your organization. On the exam, you may also be tasked with
choosing between different types of visualizations, and it's important to
understand the differences between common types of visualizations. First,
there are line charts, which are ideal for tracking trends over time. Bar charts
split your data into categories and show comparisons between categories.
Pie charts can help illustrate percentages or parts of a whole. And scatter
plots or bubble charts compare two or three dimensions, respectively. Scatter
plots just show dots on an xy grid, while bubble charts also change the size
or color of the dart to represent some other dimension. Funnel charts are
used to show stages through a process. A good example of this would be
tracking rates of customers visiting your site, putting an item in their
shopping cart, and then actually going through with purchasing the item.
Histograms show the distribution of values for a given dimension. And
gauges display a single metric like a key performance indicator. That's all for
visualizing data. This has been David Blocher. When you're ready, please join
me in the next one.

Maintaining and Monitoring Data Pipelines

Intro to Amazon Macie

Welcome to this module on maintaining and monitoring data pipelines in the

context of the AWS Data Engineer-Associate certification. In this lesson,
we're going to introduce you to a service called Amazon Macie. Data
pipelines can get messy. They can pull from multiple sources. They can have
multiple stages with multiple staging buckets, and they can split and join and
fork in all kinds of complicated ways. Even with fairly simple data pipelines,
you'll be thinking this is a lot of buckets, and it can be overwhelming to
imagine monitoring all of these buckets for sensitive data. Amazon Macie is a
service that's designed to simplify this monitoring process. You can activate
Amazon Macie in your account, and it will continuously scan all of your S3
buckets for sensitive data such as PII, or personally identifiable information.
You can also set it up to scan for things like specific intellectual property that
you don't want residing in your S3 buckets. Amazon Macie will scan your S3
buckets and then send you an alert or trigger automated action if it finds any
sensitive data. Amazon Macie is also great for multi-account environments.
Even if your S3 buckets for data pipelines are spread across many accounts
within an AWS organization, Macie can be activated within each account and
then centrally managed from a single account. Just like in a single account, it
will aggregate findings and you can set up automated actions based on
these findings. Macie is a great service if you need to inspect the contents of
your S3 buckets with very little operational overhead. That's all for this short
clip on Amazon Macie. When you're ready, please join me in the next one.

Intro to Amazon CloudWatch logs

Another service you can use to monitor your data pipelines and your AWS
services in general is Amazon CloudWatch. CloudWatch can collect logs from
many different sources like Amazon EC2, CloudTrail, or even custom
application logs. Logs from each source are then placed in a log group. By
default, these log groups are retained indefinitely, and you can search them
for patterns or anomalies. A lot of AWS services natively support CloudWatch
logging. All you have to do is enable it on the service and those CloudWatch
logs will start to be sent to CloudWatch. You can also send CloudWatch logs
via the CloudWatch agent, which can be installed on EC2 instances or
on-premises servers. You can also programmatically send CloudWatch logs
with the AWS CLI or AWS APIs such as with the AWS software development
kit, or SDK. Once those logs are received by CloudWatch, you can analyze
them in many different ways. You can simply enable anomaly detection on a
log group. You can use CloudWatch Logs Insights to interactively query log
data. You can periodically export your logs to Amazon S3 for analysis with
services such as Amazon Athena, and you can stream your logs to Amazon
Kinesis or Amazon OpenSearch. First, let's talk about anomaly detection. You
can create one log anomaly detector for each log group. Anomaly detection
uses machine learning and pattern recognition to create baselines and
detect outliers within your log group. It will then dynamically assign priority
to detected anomalies based on how far it deviates from that recorded
baseline. So this is a great way to catch novel errors as they occur. If you
need to inspect your logs and perhaps investigate a specific event, you can
use CloudWatch Logs Insights to define and execute queries on your log
groups. You can either create CloudWatch Logs Insights queries or you can
use a natural language query which leverages AI to translate your query into
the proprietary format. These queries can be executed on up to 50 different
log groups and the results can be downloaded and further analyzed. Another
option for a more advanced analysis of CloudWatch Logs is exporting your
logs to Amazon S3. To export your logs to S3, you have to manually or
programmatically export those logs on a periodic basis. From there, you can
use services like Athena and QuickSight to query and visualize the logs
you've exported. Exporting to S3 is a great option if you need to do some
sort of periodic analysis like a once-monthly report. S3 is also a cheaper
option for storing your logs long term than keeping them in CloudWatch
forever. If you need to react to the contents of your logs in real time, you'll
want to take advantage of log streaming. If you want to search and visualize
your logs in real time, you'll want to integrate CloudWatch Logs with Amazon
OpenSearch Service. This is the best option for near real-time monitoring and
searchability of your CloudWatch Logs. There's another option if you simply
need to react to certain patterns in your logs in near real time, and that is
subscription filters. You can create a subscription filter that triggers an event
whenever a certain pattern is met within a log group. That event can trigger
automated action, it can stream to Amazon Kinesis, or directly trigger an
AWS Lambda function. So while OpenSearch is great for general purpose,
real-time analysis, subscription filters are a great option to react to specific
log patterns in near real time. That's all for this clip on CloudWatch Logs.
Take a moment to grab some water or tea, and when you're ready, please
join me in the next one.

Conclusion

Summary

Congratulations. You're almost through this section of the AWS Certified Data
Engineer-Associate exam prep learning path. In this clip, we're just going to
quickly summarize everything that we covered in this course. First, we took a
look at Amazon AppFlow and how it can help you integrate your data
services with third-party data providers. Then we had a tour of Amazon EMR,
or Elastic MapReduce. EMR is a managed cluster for running big data
operations, and it takes care of a lot of the underlying infrastructure
management and security. You also have the option of using EMR Serverless,
where you don't have to define the number of nodes or the size of the
instances of those nodes. This allows you to eliminate cluster configuration
altogether, and all you have to do is define your runtime and any jobs that
you want to run. Sticking with EMR, we also learned the difference between
ephemeral and long-standing clusters. If your jobs are only running
periodically, you'll want an ephemeral cluster. Long-standing clusters should
really be reserved for jobs that involve streaming data or clusters that are
getting close to constant use or scenarios where super high performance and
immediate start up are a must. We learned about the EMR Hive metastore
and how you can leverage AWS Glue Data Catalog or services like Amazon
Aurora or RDS to create an external Hive metastore. We talked about Glue
DataBrew as a low and no-code solution to help data analysts filter and
validate data coming from your data lake or warehouse. On the exam, look
out for Glue DataBrew in scenarios where you need to perform and automate
data transformations in a low or no-code solution. We also talked about the
plethora of Apache open-source applications and their AWS counterparts.
While AWS offers managed services for Apache applications such as Airflow
and Kafka, they'll tend to prefer their AWS integrated offerings unless there's
a requirement for the use of open source. We also talked about EventBridge
and how it's used in event-driven architecture. It can also be used to
schedule automated actions to take place. But we also cautioned that
EventBridge can overcomplicate some architectures. You should always look
for the simplest solution to help you solve the problem at hand. We spent
some time talking about Amazon Athena, which is a serverless interactive
query service for analyzing data in S3 buckets. On the exam, you'll be tasked
with understanding how to optimize Athena queries. You should understand
how to partition your data, compress your data, and use columnar file
formats which are optimized for analysis. You'll need to understand that
QuickSight can ingest data from many different data sources and share data
visualizations across your organization. A common troubleshooting problem
involves giving QuickSight the roles necessary to access the underlying data
for an Amazon Athena query. In this case, QuickSight needs an IAM role both
to access Amazon Athena and the associated S3 buckets. We also talked
about different ways to visualize your data processing and analysis. AWS
Glue Studio is a visual editor in the console where you can view the flow of
your data transformations. Athena visual query analysis helps you visualize
and troubleshoot optimization of your Athena queries. The Redshift Query
Editor takes it a step further by allowing you to visualize query results
directly from the query editor. This is great for creating simple visualizations
without having to provision an entire QuickSight application. If you have
experience in data analytics, you probably are already very familiar with
these common types of visualizations. But if you're not, I highly recommend
you review each type of visualization and review their use cases. There are
many more types of visualizations, but these are the ones you're most likely
to encounter on the exam. We talked about Amazon Macie and protecting
sensitive data in S3. Macie continuously scans your S3 buckets and can alert
you if it finds sensitive data such as PII, or personally identifiable information.
And finally, we talked about CloudWatch Logs and the different ways you can
monitor and analyze CloudWatch Logs. CloudWatch can ingest logs from your
AWS services or custom application logs and sort these into log groups. You
can then enable anomaly detection on the log groups. You can dig into the
logs with CloudWatch Logs Insights or if you need long-term storage or more
in-depth analysis, you can export those logs to Amazon S3. This is great if
you need to do periodic analysis with tools like Amazon Athena. It's also a
more cost-effective option if you want to store your logs long term. For
real-time log streaming, you should consider Amazon OpenSearch Service.
This is the best option for real-time monitoring, analysis, and searchability of
CloudWatch logs. If you only need to respond to particular log patterns, you
can also use a subscription filter. A subscription filter can send an event to
Amazon Kinesis or AWS Lambda, for example, and help you respond to
events in near real time. Great work on making it this far toward your AWS
Certified Data Engineer-Associate certification. You're one step closer now,
and I have no doubt if you keep applying yourself, you can achieve this
coveted designation. So make sure you review any topics where you feel less
confident, and when you're ready, please join me in the next one.

Data Operations and Support: Exam Tips

If you've made it this far, it's probably time to start thinking about scheduling
that exam with AWS. But to make sure you're prepared, let's walk through a
few scenarios that you're likely to encounter when taking the exam. First,
how can you create a persistent, central data catalog for an EMR Apache
Hive metadata store with the least operational overhead? For this question,
your mind should immediately go to creating an external Hive metastore in
AWS Glue Data Catalog or Amazon Aurora or RDS. Generally speaking, AWS
Glue Data Catalog is the simpler and more effective solution here. Your
metastore remains managed by EMR for your Hadoop workloads, but Glue
provides a centralized catalog that's really easy to integrate with other AWS
data sources. Next, let's take a look at this one. You need to scan S3 buckets
across your data pipeline to make sure they don't contain PII. How can this
be achieved? The correct answer involves enabling Amazon Macie in your
AWS account. This is a great service to know for the exam as it appears as
the correct answer in a couple of scenarios and also can appear fairly often
as a distractor. So it's really important to understand the function of Macie.
Next, you're tasked to enable a group of non-technical data analysts
invalidating and enriching incoming data. The process should be easy to
automate once it's defined. What service can help accomplish this? For this
one, we're looking for a low or no-code solution because our data analysts
are non-technical and we need to be able to both validate and enrich
incoming data. Here, we're looking for AWS Glue DataBrew, which is
specifically designed for this scenario. Data analysts can use DataBrew to
define repeatable data validations and data transformations. Generally
speaking, anytime a scenario requests a low or no-code solution, you should
see if Glue DataBrew is a good fit. Here's the last one. Your company has
several applications sending custom CloudWatch logs to several log groups.
You need to aggregate data, monitor application logs, and set up anomaly
detection for these logs as they arrive in real time. How can this be
achieved? Here, the solution is going to involve Amazon OpenSearch.
OpenSearch is the best option for near real-time monitoring and searchability
of CloudWatch logs. I hope you enjoyed this course on data operations and
support for the Data Engineer-Associate certification. I wish you luck in the
rest of your learning journey, and I look forward to seeing you next time.

Data Security and Governance

Introduction

Introduction

Hello, everyone! My name is David Blocher, and welcome to my course, AWS

Certified Data Engineer - Associate: Data Security and Governance. I'm a
Cloud Author here at Pluralsight. Data drive value at companies around the
world, and controlling access to that data is more important and more
challenging than ever. This course is designed to help you learn everything
you need to know for the data security and governance domain of the AWS
Certified Data Engineer Associate exam, as well as prepare you for securing
data on AWS in the real world. Some of the major topics we'll cover include
controlling access to AWS data services, encrypting or masking data, and
governing complex data ecosystems on AWS. By the end of this course, you'll
have all of the skills necessary to confidently pass the data operations and
support domain of the AWS Certified Data Engineer Associate exam. Before
beginning this course, you should be familiar with the basics of AWS
technology and services and general data storage and security concepts. I
hope you'll join me on this journey to learn data security and governance on
AWS with the AWS Certified Data Security and Governance course at
Pluralsight.

Limiting Access to Data

IAM Foundations

Hello, my name is David Bocher, and I'll be leading you through this course
on data security and governance for the Data Engineer Associate
Certification on AWS. In this section, we're going to be talking about the tools
and services that can help you limit access to data. Now, what do we mean
by access to data? Well, you may have a database, and there are many
different types of principles that may want to access the data in the
database. You could have AWS services, applications, or human users that
want to achieve access to your database. Now, I'm using RDS as an example
here, but you can really think of this as any data service on AWS. Throughout
this course, we'll be peeling apart the layers of security and governance that
you can use to help protect your data. Now, it's important to understand for
the Data Engineer Associate Certification, you need to understand how to
access data from different perspectives. You may need to give some users
very limited access to data, while other users need to be data
administrators. As an AWS Data Engineer, you'll need to understand how to
achieve either perspective. So let's start by taking a look at IAM, or Identity
and Access Management. This is the cornerstone to achieving access to any
AWS service. The resource, application, or user that's trying to access your
AWS services is called the principal, and these principals will be granted
permissions to access resources. The service that allows you to grant
principals access to resources is IAM. And it's important to note that
principals can access resources in different ways. Human users can use the
AWS Management Console or the AWS CLI, and applications and AWS
services will be using the AWS API to access resources. The way principals
are granted access to resources is through IAM Policies. Without an IAM
policy, principals cannot reach resources. An IAM policy can either be an
identity-based policy attached to the principal, or it can be a resource-based
policy attached to the resource. If either of these policies grants access to
the principal, and neither of them denies access to the principal, then the
principal will be allowed access to the resource. Let's take a look at how IAM
policies are evaluated. First, IAM will check does any policy, the
identity-based policy or the resource-based policy, have a deny rule? If either
policy denies access, then access will be denied. Then it will evaluate if
either policy has an allow rule, and if at least one policy has an allow rule for
the action taking place, then access will be allowed. IAM policies are stored
in the JSON file format, and they look something like this. Take a moment to
look at this policy and think about what it might do. In this case, it allows
three different actions. You can either put objects into, get objects out, of or
get the object versions of objects in our example bucket. And this is an
identity-based policy, which means it would be added to an IAM user or role.
Let's take a moment to explore IAM users. IAM users are a way to map
permission sets to human users. In this case, we may have a data engineer
permission set, and we'll apply it to the IAM user, which is a data engineer.
We'll have a separate permission set for data administrators, which will be
applied to the datastore administrator. Now, if you only have two people
working in AWS, this may work, but most organizations have many, many
more. So, they might use IAM groups. Instead of applying permission sets to
individual IAM users, you can apply permission sets to IAM groups and assign
users to those groups. That way you can change the permission sets for a
group of users all at once without having to do them one by one. Now, if
you're just using IAM users and groups, you may run into a problem, and that
problem involves giving programmatic access to users. You see, when you
give AWS CLI access to users, you create a static access key and secret
access key that that user has to store on their local machine. If those access
keys are leaked or somehow acquired by bad actors, then the bad guys
would have all the same access as the data engineer in this scenario. This is
how a lot of data breaches happen on AWS. The solution here is using IAM
Identity Center. With Identity Center, you don't map permission sets directly
to users or user groups. Instead, you map permission sets to IAM roles. And
through single sign-on, the data engineer will be able to temporarily access
this data engineer IAM role anytime they log in. They can generate
temporary access keys for the length of their session for CLI access. Avoiding
long-lasting access keys is always a good idea, and IAM Identity Center
simplifies this process quite a lot. Now, you may already be familiar with IAM
roles. IAM roles are used by AWS services and applications to temporarily
assume privileges to access AWS resources. So not only can human users
use IAM roles through IAM Identity Center, but AWS services and applications
will do the same to access those resources. I hope you'll join me as we
continue to peel back the layers of data security on AWS. This has been
David Blocher, and when you're ready, please join me in the next lesson.

Accessing Private Data Stores

The next thing to consider when granting principles access to data is

whether or not they have the network connectivity to reach that data. Let's
look at some networking scenarios you may need to consider for your exam.
You are likely to see some questions around AWS, VPCs, and private subnets.
You can have a private database, such as RDS in private subnets, and you
can also have private compute services, like EC2 hosted in your VPCs. You
not only need to consider how can you grant access to your private
datastores, grant your private EC2 instances to things like data stored in S3
buckets, you'll also need to consider this from the perspective of AWS
Lambda. How can you give your Lambda functions access to private
resources? Let's start by taking a look at how to connect EC2 instances to
private RDS instances. So, these are private resources, which means there's
no public route to the internet through an internet gateway. RDS instances,
as well as EC2 instances will be assigned a security group. And at the subnet
level, you may encounter network access control lists, or NACLs. Both
security groups and NACLs have inbound and outbound rules that can be
applied to them. And these will determine the types of traffic that can enter
and exit the security group and the subnet. They're fairly similar and easy to
confuse, so let's start by talking about the differences between security
groups and NACLs. Security groups are applied at the instance level. They're
like a firewall just for the instance itself. Network access control lists are
applied at the subnet level. For both security groups and network access
control lists, you can define inbound and outbound rules. For security groups,
there are only allow rules, and any traffic not allowed is implicitly denied.
With network access control lists, you can also apply deny rules. With
security groups, your rules are all applied at once and the order of the rules
doesn't matter. Whereas with network access control lists, the rules are
evaluated in order. Another big difference between security groups and
network access control lists, security groups are stateful. So if the instance
initiates allowed traffic, it will always allow the response traffic back in,
whereas NACLs are stateless. Even if traffic is allowed out, the return traffic
will not be allowed if the NACL doesn't allow that type of inbound traffic. So
let's break this down a little further. You have your EC2 instance and it's in a
security group. You define some inbound rules. In this case, you're allowing
port 22 access from a specific IP range. This is to give SSH access to your
EC2 instances within that range. You can think of this like a wall or a
blockade that has a tiny little access point in it. Only traffic from specific IP
addresses will be allowed in, and only if they're using SSH over port 22. Any
other protocol traffic coming from this source will not be allowed because it's
not TCP port 22. And furthermore, any traffic from the public internet, no
matter what protocol is being used, will not be allowed. Now, let's say we
have an outbound rule that allows all traffic to any destination. The EC2
instance can reach the public internet, and the public internet can respond.
Even though there's not an inbound rule because security groups are
stateful, it remembers that the instance initiated this communication, and so
it will allow the response to this initiation. That is a stateful connection. Now
let's take a look at network access control lists. Again, these are applied at
the subnet level. Here's what a set of NACL rules might look like. As you can
see, they're numbered and they're evaluated from lowest to highest. This
network access control list allows SSH from a specific IP range, and it allows
HTTPS from any source. So this might be an example of like a web server.
Again, if you have traffic coming from within that IP range, you can SSH to
the instance. However, in this case, our request times out, and we never get
a response from this EC2 instance. Why is that? Well, if we take a look at the
outbound rules, currently we only have an outbound for HTTPS allowing all
traffic from port 433. In order to receive response traffic from SSH, you're
going to need an outbound rule that allows outbound traffic from your
ephemeral ports. As you can see, our destination is that same IP range, and
we're allowing SSH traffic out from the EC2 instance. So in this case, it
doesn't matter who is initiating a connection. If the rule doesn't exist to allow
outbound or inbound traffic, the NACL will prevent that connection from
happening. Now, if this was a little overwhelming, don't worry, this is more in
depth than you'll need to know for the Data Engineer Associate exam. What
you will need to know is how to troubleshoot connectivity between private
instances and private data sources, such as a private RDS instance. So we
want to connect our instance to RDS. The first thing that you'll need to check
is whether or not there is a route that exists to route traffic from your EC2
instance to RDS. Next, you'll need to make sure there's not a network access
control list disallowing the connection from the EC2 instance to the database.
The protocols for different types of databases are different, and you don't
need to memorize them all. But you may need to know that the network
access control list could be denying traffic if all other things are correct. The
third thing you want to check is whether there is an inbound rule on the
database security group that allows traffic from the instance security group.
Your instance has the IAM permissions and the route to connect to the
database, but still, it's not able to connect. The most likely reason is that
your security group is missing a rule to allow traffic. Likewise, you could be
missing an outbound rule to allow instance traffic from the instance security
group to the database security group. So if there's one thing you take away
from this, you should really understand how to set up security groups to
allow traffic across private resources within a VPC. Next, let's talk about how
you might give a private compute instance access to S3. So you have your
VPC and your private subnet, and you want to give it access to an S3 bucket.
Well, because it's a private subnet, you're not going to be able to route to the
public internet and then to the S3 bucket, and you probably wouldn't want to
do this anyway. It's going to be slow and it's going to be expensive. You pay
for all the outbound traffic on AWS, so you'd be paying for the request itself,
and then when your S3 bucket has to send the object, you'd be paying for
that object to travel into the public internet as well. Not to mention this is a
lot less secure. You can achieve this same request without any traffic
traveling on the public internet. So, how should we achieve that? Well, here
you want to know about gateway endpoints. Gateway endpoints are private
endpoints for S3 or DynamoDB, and they're a great way to give private
resources access to S3 where the request traffic never goes across the public
internet. These only work within the same region, but they're a simple way to
keep your subnets super secure, and they'll give you relatively inexpensive
access to data stored in your S3 buckets. Now, let's talk about if you want a
Lambda function to access your private RDS instances. This is a little bit
different than giving an EC2 access because you don't have control over the
VPC that your Lambda is in. By default, Lambda functions will not have
access to your VPC, and they cannot connect to your private instances. What
you can do when provisioning your Lambda functions is configure them to
have access to one of your private VPCs. The Lambda functions still reside in
an AWS-managed subnet, but you'll be given an elastic network interface
through which you can route private requests from your Lambda functions to
your data sources. At that point, you basically follow the same
troubleshooting steps as you would for an EC2 instance. It's important to
note here that deploying your Lambda functions to have access to your VPC
will in effect make them private themselves. They won't have access from
the public internet or a public API or something like that. But this is the way
to grant Lambda functions network access to your private RDS instances.
Finally, let's take a look at how Lambda functions can access S3 buckets.
Well, in this case, you don't have to do any networking magic. As long as the
Lambda function has IAM permissions to access the bucket, either from an
identity-based policy or a resource-based policy, then you should be able to
achieve network communication between the two. So that does it for our
networking scenarios that you may encounter on the Data Engineer exam.
This has been David Blocher, and whenever you're ready, please join me in
the next lesson.

Intro to Secrets Manager and Parameter Store

The next thing to consider when granting principals access to your

datastores is authentication. Now, oftentimes IAM takes care of
authentication, but in the case of databases like RDS, you may have a
separate authentication that needs to happen in order to grant principals
access to your database. In the case of RDS, there are three authentication
options. In some cases, you can delegate authentication back to IAM. You can
also use Kerberos authentication, which leverages AD, or Authenticated
Directory, from Microsoft. And the third option is to use password
authentication. Now, on the Data Engineer Associate Exam, you are unlikely
to see Kerberos authentication, so we're going to focus on these other two
options. First, let's break down IAM authentication. The first important thing
to note is that this form of authentication is only available for a few different
database engines on RDS. These include MariaDB, MySQL, and PostgreSQL.
An important caveat here is that IAM authentication may limit your maximum
connections per second. This limitation depends on the size of your instance,
but suffice it to say, if you're receiving hundreds or thousands of connections
per second to your RDS instance, IAM authentication may not be optimal,
and it might slow down your reads and writes. The other form of RDS
authentication is password authentication. When you create an RDS
database, you'll set up a master username and password, and these will be
needed to take certain actions on your RDS instance. Now, there may be
times you want to use credentials programmatically. Perhaps you want to
allow a Lambda function to authenticate with your RDS database. Now, one
thing you could do is store your username and password in plaintext in your
Lambda code. As you might have guessed, this is a really bad idea, and you
might consider putting it in an environment variable or something like that,
but that's still not a great idea, as you still risk giving access to plaintext
username and password. Fortunately, AWS provides a couple of services that
help you keep your secrets safe. First, there is Parameter Store, and this is
where you can store, encrypt, and programmatically access key-value
parameters. This is great for application configurations, custom environment
variables, product keys, and in some cases, credentials. We also have
Secrets Manager, which is always encrypted and can automatically rotate
your key-value secrets. This is really great for API keys and passwords. On
the Data Engineer exam, you may be tasked with choosing between storing
credentials in Parameter Store or Secrets Manager. Let's take a look at their
differences. Both of them store key-value pairs and allow you to
programmatically access them. Parameter Store optionally allows encryption,
whereas Secrets Manager secrets are always encrypted. If you want to
change the secrets stored in Parameter Store, you must manually change
those secrets, whereas Secrets Manager grants you the option to
automatically rotate the secrets. One big difference is that Parameter Store
gives you 10,000 parameters for free, while Secrets Manager is always 40
cents per secret per month. Parameter Store is really great for centralizing
variables that may exist across many code bases, whereas Secrets Manager
always provides the highest security for your secrets. So, when it comes to
RDS password authentication, Secrets Manager is almost always the correct
answer. RDS has a direct integration with Secrets Manager to update and
rotate those credentials on the RDS instance. So for any secret that requires
top security and especially rotation, Secrets Manager is king. So what would
it look like to programmatically access your secrets from Secrets Manager?
Let's take a look. Here's another example, Lambda function, and this Lambda
function would have IAM access to the secrets stored in Secrets Manager.
First, you would define your secrets name and region during initialization.
Then you would initialize your session and client. This would allow your code
to interact with AWS services. Finally, at runtime, you would use that client to
retrieve your secrets. In this case, you're using the get_secret_value function
to retrieve your secret based on its secret ID. Now you won't encounter code
like this on the exam itself, but I think this is useful to look at to get an idea
of how you would retrieve secrets in a real-life scenario. This way, you never
have to store those secrets in plaintext. In fact, no one even has to know the
username and password; they're all being rotated behind the scenes and
managed by IAM permissions. So that's it for this lesson on authentication.
Take a moment to grab a coffee or tea and whenever you're ready, please
join me in the next lesson.

Securing Data Using Lake Formation

So far, we've been focusing on giving principals access to datastores where

your datastore is a single resource where you can grant IAM permissions,
network access, and authentication. But permissions can get tricky when it
comes to using data lakes. AWS simplifies this with Lake Formation, and this
service can help you centralize IAM permissions, granular permissions, and
monitoring and logging for your data sources. Let me show you what I mean.
So far we've been giving our resources direct access to our datastores. But in
most practical use cases, this will not be the case. In many practical cases,
principals will be accessing data stored in a data lake. This data can be
aggregated from many data sources filtered and then stored in a data lake.
Now, this could be as simple as just granting your principals access to the
data lake, but this can also get a lot more complicated. You see, when it
comes to your data lake, it's not just the data itself. You also have data
catalogs and data analytics, and you may want to grant different permissions
to different principals for each of these resources. In order to grant access for
your principal, you'll have to apply a policy that grants access to all of these
services. And for each different role that needs access to these different
resources that compose your data lake, you'll need individual IAM
permissions to access each of the resources that compose your data lake.
Well, Lake Formation creates a single entry point through which users can
access all of the resources that compose your data lake. So in practical
terms, you only need to determine which actions these principals can take
and the specific data they should have access to. And instead of having to
define these per service, they will apply across all services that comprise
your AWS data lake. So how do Lake Formation permissions work? Well, you'll
be using IAM to define the actions that can be taken, and then you'll be using
Lake Formation to determine exactly what data those actions can be taken
on, and no matter which service you're leveraging within the data lake, this
user will only be able to access the data that's allowed by the Lake Formation
permissions. So when they search S3 using Athena, they'll only be able to
query data for which they have access. Lake Formation achieves its
fine-grained permissions using RDBMS. This stands for Relational Database
Management System, and it's a common syntax used to restrict access to
databases by database, table, or column. So let's take a look at how we
would grant permissions to our data analyst. This is a data analyst, and she
only needs access to the food item sales table. For her persona permissions
through IAM, you may grant her the managed policy, Amazon Athena full
access, and add an inline policy for Glue and Lake Formation read
permissions. This would allow her to query and analyze all data within our
data lake. With fine-grained permissions using RDBMS, we can restrict access
to only the food items sales table. Let's take a look at a different persona.
Here we have our data lake administrator. Their persona permissions from
IAM will include permissions to access all the data in the data lake, as well as
grant access to other users. So by defining other users, our lake
administrator can grant access to other users depending on their role. Now,
not only can access be granted to your data lake to users within your
account, but you can also give cross account access. You can grant data lake
permissions to principals in different AWS accounts. Perhaps you're in an
organization with many accounts, but you control a centralized data lake. You
may need to grant permissions to the data lake to principals around your
organization. Lake Formation also centralizes event logging. CloudTrail will
track all the API actions taken within your data lake, and Lake Formation
makes it a lot easier to tell when data actions and permissions granting take
place with your data lake. That's all for this lesson on securing data using
Lake Formation. This has been David Blocher, and I'll see you in the next
lesson.

Using Lake Formation Granular Permissions

Let's take a closer look at how Lake Formation can help you achieve super
fine-grained permissions. When you set up a data lake with Lake Formation,
you create a data catalog. This helps you pull data from multiple sources and
break it into databases and tables that you can access with analytic services
and further transformations. With Lake Formation, you can give principles
access to particular tables within your data catalog. For even more
fine-grained access control with Lake Formation, you can add filters. There
are a few different types of filters that you can use in Lake Formation. Filters
are defined at the table level and allow you to give custom fine-grained
access to different users. Let's see what this looks like. One type of filter you
can do is a column masking filter. Say you have a table of item orders and
you want to mask the shipping address for certain users. You can add a filter
to this table, which excludes the column shipping address. Similarly, you can
use an expression to filter out certain rows. So say you have an analyst
whose job it is to analyze all of the orders that are placed from the US. Well,
you can do a row filter expression to only allow access to the rows that
contain that value in the country column. These types of filters can be
combined to achieve cell-level security. You can conditionally mask individual
cells from certain users. So if there's any case where you need to aggregate
data and achieve cell-level security Lake Formation is likely part of the
answer. Now, one thing I want to make really clear here is that filters are
applied at the table level, and then when you give users access to a table,
you can apply one or more filters to those users. For instance, you may use
the US sales filter to grant access to a US analyst. Perhaps there is a Canada
filter to give an analyst access to only data for Canadian orders. These can
be granted to individual users or groups of users. That's all for this lesson on
Lake Formation granular permissions. This has been David blocker and I'll
see you in the next one.

Ensuring Data Encryption and Masking

Protecting PII and Masking Columns

Welcome to this module on ensuring data encryption and masking. When it

comes to protecting PII, or personally identifiable information, and masking
columns of data, questions on the Data Engineer Associate exam are mostly
going to revolve around two services. First, we have S3, where we'll be
creating data lakes, and we'll need to know how to identify and protect PII
across many S3 buckets. Then there's Redshift, which is an enterprise-scale
data warehouse, which can contain PII that we don't want to expose to
analysts and data engineers. First, we'll take a look at data lakes in S3. When
it comes to protecting personally identifiable information in S3, we need to
consider not only S3 buckets in a single account, but S3 buckets across many
accounts. One thing that's important to know is that you can provision
Amazon Macie at the organizational level. Amazon Macie can be configured
to look for certain forms of PII across all S3 buckets across your organization.
It will continually scan your S3 buckets and provide alerts or generate
automated action when PII is identified in your S3 buckets. Next, let's take a
look at Redshift and how we can protect PII from being exposed in our data
warehouses. Redshift allows you to mask whole columns and rows, and it
does this in slightly different ways. To mask columns, you'll be doing this
directly with the grants that you provide to users. So when you're granting
select and update privileges, you can do this not only by table, but also by
individual column. To mask specific rows in a Redshift cluster, you'll need to
create an RLS policy. This is called a Row Level Security policy, and this will
limit row access, and you can apply this to particular users or rows. Here's an
example of what an RLS policy might look like. After creating the policy, you
would apply to one or more users or roles. Now, not only can you mask entire
columns or rows in Redshift, you can also do dynamic data masking on
Redshift. You can mask part or all of a particular column of data depending
on the user or role accessing this data. And what's really cool about this is
that the masking occurs at query time, so it doesn't take up any extra space
in your table, and it doesn't have any impact on queries that don't involve
this masked data. A common use case might be a credit card number, where
you don't want to expose the whole number to almost anybody, but
sometimes you may want to be able to expose the last four digits of the
credit card number. So let's review masking data in Redshift. You have a few
options. You can do column-level access control, which you define at the
same time as any other access control. You can do row-level security policies,
which can be applied to users or roles to restrict access from certain columns
in your data warehouse. If you don't want to mask the entire column, you
can programmatically define if you want no mask, a partial mask, or a full
mask for a particular column of data, and again, you can apply specific
dynamic data masking policies to different users and roles. A user can have
multiple layers of security applied to their user or role. And when they query
your Redshift database, the results will dynamically mask data according to
their permissions. That's all for this lesson on masking data in Redshift. This
has been David blocker. Go ahead and grab a coffee or a tea and when
you're ready, please join me in the next lesson.

Data Encryption Options

Another way that we can protect our data in AWS is through the use of
encryption. To put it simply, encryption is a way that you can scramble your
data such that only particular entities can access the unscrambled secret
data. When data is encrypted, one or more encryption keys is produced, and
only entities with access to the correct encryption keys can unencrypt the
data. Now, there are a couple different types of encryption that you should
be aware of. The first thing I want to talk about is data at rest versus data in
transit. If data is sitting in an S3 bucket or on an EBS drive in EC2, that data
is considered at rest. The same goes for any data on the hard drive on the
computer or device you're watching this on right now. However, we also have
to be concerned about the encryption of data in transit. Data in transit is
almost universally encrypted using TLS, or transport layer security. That's the
protocol that's used whenever you access a web resource using HTTPS. Now,
when it comes to encrypting data at rest, there are a couple different ways
you can achieve this. You can use client-side encryption or server-side
encryption. If you use client-side encryption, the application that is sending
the data will apply encryption before the data is sent. The encrypted data
travels across the internet and ends up in S3. This is really secure because
unencrypted data never even leaves the application where it originated from.
However, the application is still responsible for managing the encryption key,
which is a bit more complicated. Another option is to use server-side
encryption, where data is sent across the internet in its unencrypted form,
and once it reaches S3, it will be encrypted with an AWS managed key. This
is a solution that requires a lot less operational overhead. And as long as
you're encrypting that data in transit, it should never be exposed to the
public internet. Now, when it comes to S3, you have a couple different
options for server-side encryption. You can use S3 managed keys, which is
enabled by default. You can use KMS keys if you need more granular access
to key permissions. Or you can use customer-provided keys. You might need
to use this option if all of your encryption keys must be managed
on-premises. Let's take a more detailed look at these different types of
encryption. First, S3 managed keys. Again, these are enabled by default, and
it's super secure because each object in the S3 bucket is encrypted with a
unique key. This uses 256-bit advanced encryption standard, and when
putting objects to an S3 bucket, you'll need to include a header that looks
like this. You might choose to encrypt your S3 objects with KMS keys. If
you're using KMS keys, you can view and edit the key policies for individual
keys. KMS also makes it a bit easier to follow individual key usage in
CloudTrail. If you're using KMS to put an object in S3, you'll need to include
one header that defines KMS as the type of server-side encryption being
used, and you'll also need to supply the key ID for that particular object.
Using customer-provided keys is the most complicated option and should
really only be chosen if it's a security requirement for your application.
Customer-provided keys give you full control of encryption keys, and these
keys can be managed on-premises, on an application hosted in AWS, or a
different cloud provider. When you're creating an object by API using
customer-provided keys, you'll include the encryption algorithm being used,
as well as the encryption key itself in base64. It's important to note that if
you ever send this header over an unsecured network, you should consider
your key to be compromised. Next, let's talk about encrypting data in
Redshift. With Redshift, you'll likely be encrypting your data using KMS, and
you can use AWS-managed keys or KMS customer-managed keys. One really
important detail to note is that you can't encrypt a Redshift database without
affecting its performance. You either have to choose to encrypt it upon
launch, or you migrate it to an encrypted cluster by changing the encrypted
database option. AWS still manages this migration, but you may experience
some downtime as the migration takes place. For AWS Glue, you have a
couple different considerations for encryption. First, as you receive data from
your data sources, you're going to want to make sure that these are
encrypted in transit. You can configure AWS Glue to only access data stores
through a trusted TLS protocol. Then there's all the data at rest, including
your Glue data catalog and data stored in S3 buckets. These will be
encrypted using KMS or S3-managed keys. When it comes to granting access
to encrypted data, it's important to know that the principles that wish to
access your encrypted data at rest not only need the permissions necessary
to access the data, but they're also going to need the permissions to encrypt
and decrypt data from those data sources. That's all for this lesson on data
encryption. This has been David blocker and when you're ready, please join
me in the next lesson.

Data Privacy and Governance

Preparing Logs for Audit: Tracking API Calls via CloudTrail

Welcome to this module on data privacy and governance. In addition to

securing our data by controlling access to the data, we also need to make
sure we can audit that access to our data. We can achieve this by tracking
API calls with CloudTrail. Now, you should be somewhat familiar with
CloudTrail already, but for a quick refresher, when any principal accesses
your AWS account or resources within your AWS account, an API call is made,
and CloudTrail records all of these API trails. In other words, no matter who is
taking any action in AWS, CloudTrail always leaves a trail. This trail can later
be used to audit activities within your AWS accounts. Once you have
collected CloudTrail logs, you may want to analyze those logs to audit your
activity. You have a couple of different options here. You can store your
CloudTrail logs in an S3 bucket and then use analytics tools such as Amazon
Athena to query the contents of that bucket. If you need to find particular
types of activity or analyze activity in a certain time period, this is a great
option. If you want a more managed solution, you can also store your logs in
a CloudTrail Lake. CloudTrail Lake makes it easier to store and save your
queries, and you can filter your CloudTrail logs by certain types of events. So
far, we've only talked about capturing CloudTrail logs in a single account. But
the reality is in most cases, you're going to have multiple accounts with
different teams working on different resources in AWS. One option is to have
CloudTrail active in each of these accounts, and each account storing its own
log data. However, in this case, you have to restrict access to the logs
bucket, and you can't query log activity across your entire organization. AWS
considers it a best practice to have an account where you can aggregate logs
and centrally store, secure, and analyze those logs. That way you don't have
to be so careful about managing S3 permissions in those other accounts. If
you store all of your logs in a centralized account, you can have really
restricted access to those logs and make sure they aren't tampered with or
accidentally deleted. Once those logs are centrally stored, you can grant
access to certain analytic services, such as Amazon Athena. That's all for this
lesson on preparing logs for audit. This has been David blocker and when
you're ready, please join me in the next lesson.

CloudWatch Alarms and Logs

Another tool we'll be using to analyze activity on our data services is

CloudWatch. While CloudTrail can monitor API activity on the resources
themselves, we'll need to leverage CloudWatch to monitor resource
performance and application log. For example, with RDS and Redshift, you
can export the database logs to CloudWatch. And with this data, you can set
up alarms when certain activity occurs and take automated remedial action.
So, how does this work? Well, you'll be collecting CloudWatch Logs in log
groups, and you can subscribe certain services to log events that take place
from certain log groups. You can also subscribe services to events that take
place across an entire account, and you can either stream the whole log
group or you can filter it to only capture certain types of events. One fairly
common use case you may encounter on the exam is analyzing your
CloudWatch Logs in near real time using Amazon OpenSearch. You can also
attach metric filters to your log groups, and you can define alarms using
these metric filters. So the idea is if a metric reaches or exceeds a certain
defined threshold for a certain period of time, you can trigger an alarm, and
this can be used to send an SNS notification to your security team, for
example, or trigger automated action. For example, if you wanted to receive
an email notification whenever DDL, or Data Definition Language, commands
are issued in Redshift, you could filter the Redshift Logs long group for that
metric and set up a CloudWatch alarm on that metric filter. At this point,
we've covered both CloudTrail and CloudWatch, so I wanted to take a
moment just to disambiguate these because they're easy to confuse.
CloudTrail records all of the API activity in your account, while CloudWatch
monitors the performance of your resources and can capture application
logs. The main use case for CloudTrail Logs logs is tracking and auditing
actions across your AWS accounts, where CloudWatch Logs is meant to
monitor the resources themselves. With CloudTrail Logs, you can store logs in
S3 and monitor events with CloudWatch. While you can get real-time analysis
of your CloudWatch Logs that are stored in CloudWatch, keeping your
CloudWatch logs in CloudWatch can get quite expensive over time. One thing
you can do is back up your CloudWatch Logs with Amazon S3. While you lose
out on some of the features of CloudWatch, you can save quite a bit of
money, and you can still analyze these archived CloudWatch Logs with
services such as Athena. It's important to note that by default, CloudWatch
Logs are stored indefinitely. So unless you really need your logs to be stored
forever, I highly recommend you change this default setting whenever you
set up a CloudWatch Log group. This exact scenario isn't super likely to come
up on the exam, but I wanted to mention it because it could save you a lot of
money and pretty much everyone uses CloudWatch. That's all for this lesson
on CloudWatch Logs and alarms. This has been David Blocher, and I'll see
you in the next lesson.

Sharing Data Across Redshift Clusters

One scenario you may encounter around data security and governance on
the Data Engineer Associate exam is sharing data across Redshift clusters.
These Redshift clusters can exist within the same AWS account, or you could
potentially need to share data across accounts. In either case, you're going
to be using a Redshift datashare in order to share this data. A datashare can
consist of one or more database, table, user-defined function, materialized
view, or schema. First, you define which Redshift objects will be shared, and
then you share it with a particular Redshift cluster. When deciding which
Redshift cluster to share your data share with, it's good to know that you can
share across cluster types, you can share across availability zones or regions,
and you can even share datashares across AWS accounts. Let's take a look at
what this would look like. So here we have some commands issued in the
Redshift console. You would create and name your datashare. Here, we add
the public schema, which is the default schema of any cluster, and then we
add the sales table. Finally, once we've added objects to our datashare, we
can grant usage of the datashare to a particular Redshift cluster. Once you've
granted usage, Cluster B will have read only access to all objects placed in
the datashare. And there's no need to refresh the datashare to give
up-to-date data; it will automatically read the live data from Redshift Cluster
A. That's all for this short lesson on sharing data across Redshift clusters.
This has been David Blocher, and when you're ready, please join me in the
next lesson.
Conclusion

Summary

All right, congratulations! You've made it to the end of this course on Data
Security and Governance as it pertains to the Data Engineer Associate exam.
Let's take a moment to review the major subjects that we covered
throughout this course. In this course, we peeled apart the layers of data
security and governance on AWS. We talked about how to implement and
troubleshoot the different layers of data security. Remember that data
security and governance takes up 18% of your total score on the Data
Engineer Associate exam. While this is the smallest domain on the exam
guide, it's still crucial to understand data security and governance inside and
out for implementing data solutions in the real world. The first thing we
talked about was IAM, or Identity and Access Management. This is the
backbone of all access management on AWS. With IAM, you grant principals
access to resources, and you do this either with identity-based policies, or in
some cases, resource-based policies. When you grant people access to your
AWS resources, they can take actions using the AWS Management Console,
the AWS CLI, or programmatically using AWS APIs, or the AWS Software
Development Kit. Next, we talked about network security. Specifically, we
talked about how to grant network access to private datastores and how to
give private compute access to datastores. We talked about how to
troubleshoot connection between a private database and a private instance.
You need to make sure that there is a route in the route table. Then you need
to make sure the NACL, or the network access control list, does not restrict
this traffic. Then you need to make sure each security group has a rule
allowing traffic from the other. In the case of the database, you need to make
sure that the security group allows inbound traffic from the instance, and the
instance security group allows outbound traffic to the database security
group. If you're certain you've provisioned the correct IAM rules and there's a
route to your private datastore, there's a good chance that security group
rules are the reason your connection is timing out. After we went over
network security, we talked about authentication. Sometimes you need more
than just IAM to authenticate with a data source. For RDS, you can use IAM
authentication, Keberoes authentication, or password authentication. You
aren't likely to run into Kerberos authentication on the exam, but for
password authentication, you are definitely going to need to know about
Secrets Manager. Secrets Manager can help you encrypt, and, most
importantly, automatically rotate database credentials that you store. You
may be tasked with choosing between Secrets Manager and Systems
Manager Parameter Store. While Parameter Store is a bit cheaper, Secrets
Manager tends to be the most secure. It directly integrates with RDS and
should be considered the best option for encrypting and programmatically
accessing secrets. We also talked about encryption and the different services
you can use to encrypt S3 and other datastores. For S3, you can use
S3-managed keys. You can use KMS keys, which are either AWS or
customer-managed, or you can bring your own keys. For Redshift, you'll be
using KMS, either AWS-managed or customer-managed keys. The important
thing to know here is you either have to encrypt your Redshift cluster upon
launch, or you can migrate it to an encrypted cluster. AWS handles this
migration, but you may experience some downtime as the migration takes
place. We also talked about fine-grained security, row-level and column-level
security. This can be achieved in data lake using row masking. You can also
do column masking and even cell-level masking. And in Redshift, you can
mask whole columns when granting select and update privileges, and you
can mask whole rows by applying RLS policies to individual users or groups.
Redshift also allows you to do dynamic data masking of columns where you
can transform or partially mask the data found in a column. For monitoring
and logging, we talked about CloudTrail and CloudWatch. For CloudTrail, it's
considered a best practice to consolidate your CloudTrail logs in a single
security account. This allows you to really limit access to that account so
those buckets can't be tampered with or the data accidentally deleted. And
you can still analyze these logs centrally using analytic services like Amazon
Athena. For CloudWatch, you can do near real-time analysis of application
logs by sending those logs to log groups and then streaming events from
those log groups to different AWS services. You can subscribe services to log
events at the account level or the log group level, and you can filter the
events from these log groups up to two filters for an event log group and one
filter if you're doing it at the account level. You're now one step closer to
sitting and passing the AWS Certified Data Engineer Associate exam. Keep
the momentum going, and please reach out and let us know when you reach
that shiny new certification.

Data Security and Governance: Exam Tips

Before you move on from this course, I wanted to go over a few topics you
might see on the Data Engineer Associate exam. I've prepared a few
challenges to help you gauge if you're ready to move forward, or maybe if
you need to go back and review some lessons and documentation from
earlier in the course. Let's start with this scenario. You are trying to connect
an EC2 instance to a private RDS instance. You verified that the routing and
IAM permissions are correct. What is the likely issue? Knowing how to
troubleshoot routing to a private RDS instance can earn you quite a few
points on the exam. You need to verify that both the security group and the
NACL allow database traffic across your network. The likeliest culprit is the
security group. The database security group needs to allow inbound traffic
from the EC2 security group, and the EC2 security group needs to allow
outbound traffic to the RDS security group. Next, your Redshift database
contains personally identifiable information that should not be exposed to
analysts, but it can't be removed from the data warehouse. How can you
give analysts access to the database? Most likely you'll mask columns when
granting those analysts access to the database. You also have the option of
row-level security using RLS policies. And in some cases, you'll be using
dynamic data masking, depending on whether or not you need some portion
or a transformed version of the data to be visible. You are required to encrypt
data at rest in your organization's S3 bucket. However, you are also required
to have control over encryption keys and their associated key policies. How
can this be achieved? So, we know the options for S3 are S3 managed keys,
KMS keys, and customer-provided keys. S3 managed keys don't give you
access to the key policies, so you can meet the requirements of the scenario
using KMS or customer-provided keys. However, customer-provided keys are
going to be more complicated, as you don't get to offload some of that
operational complexity to AWS. KMS is a great solution in this scenario
because you get access to those key policies, but you don't have to fully
manage the encryption keys. Finally, data engineers that manage a Redshift
cluster in another AWS account need access to a materialized view from a
Redshift cluster that you manage. How can you share that materialized view
without giving those data engineers access to your cluster? This is a perfect
scenario for datashares. You can create a datashare, add the materialized
view to the datashare, and then provide that read-only access to the other
Redshift cluster, even across accounts, regions, or different cluster types.
This will give them live, up-to-date data just as it appears in your Redshift
cluster. I hope you found this exam review useful. And with that, we've
reached the end of this Data Security and governance course. You should be
really proud of yourself for making it this far, and I hope you're excited for
whatever is next on your learning journey.

Practice Exam Questions

Practice Exam Questions: Data Ingestion and Transformation

One of the best ways to prepare for certification exams is by doing as many
practice exam questions as possible. For the data engineer exam, you can
either do relevant exam questions after completing each domain, or you can
go over questions from all the domains once you're done studying. Our first
domain is data ingestion and transformation. But before we dive into the
sample question, let's warm up by going over rather trivial, but super
essential questions, starting with this question. What's the best AWS service
to utilize when you need new real-time batch analytics, Amazon EMR,
Amazon Redshift, AWS Glue or Amazon Managed Service for Apache Flink?
The key words here are near real time and batch. Usually when you see real
time, you're likely to think of services within the Kinesis family. And though
Kinesis wasn't explicitly listed as one of the choices, Amazon Managed
Service for Apache Flink is part of the Kinesis family. Therefore, the correct
answer is D because Amazon Managed Service for Apache Flink meets the
needs for both near real time and batch processing. Apache Flink is primarily
designed for real time or near real-time stream processing with low latency,
making it suitable for applications that require immediate insights from
streaming data. It integrates with Kinesis Data Streams. But as mentioned, it
can also handle batch processing. It's also suited for complex event
processing. It can handle large volumes of data efficiently, and it's scalable
and fault tolerant, so it can scale and handle failures gracefully. The next
question is which of the following AWS services can be used to run Apache
Spark clusters for data processing? Is it Amazon Kinesis, Amazon Redshift,
AWS Glue, and/or Amazon EMR? So let's review each one of these services.
Amazon Kinesis is a platform for streaming data on AWS. It's used for
collecting, processing, and analyzing real-time streaming data. While it is an
excellent choice for real-time data processing, it's not designed for running
batch processing frameworks such as Apache Spark. Amazon Redshift is
simply a data warehouse service, so it's not designed for running Apache
Spark clusters. And AWS Glue is a fully managed ETL service that supports
Apache Spark. It allows you to create and run Spark jobs for data
transformation and processing tasks. However, it's not primarily designed to
run Spark clusters directly. To do so, you would need Amazon EMR to run big
data frameworks, including Apache Spark. Therefore, depending on the
complexity of your scenario and other requirements, you would either go
with AWS Glue or EMR. With this, we're done with our warmup, and we're
ready to unpack a sample exam question. The question reads, "A retail
company collects data from its store sensors to optimize customer
experience. They use Amazon Kinesis Data Streams to publish the collected
data, which is then transferred to an Amazon S3 bucket via Kinesis Data
Firehose. The company requires a real-time display of customer activity on a
large screen in the store. Which solution will achieve this with the lowest
latency?" The options are A, Use Amazon Managed Service for Apache Flink
(previously known as Amazon Kinesis Data Analytics) to process the sensor
data. Use a connector for Apache Flink to write data to an Amazon
Timestream database. Use the Timestream database as a source to create a
Grafana dashboard. B, Use Amazon Managed Service for Apache Flink
(previously known as Amazon Kinesis Data Analytics) to process the sensor
data. Create a new Data Firehose delivery stream to publish data directly to
an Amazon Timestream database. Use the Timestream database as a source
to create an Amazon QuickSight dashboard. C, Configure the S3 bucket to
send a notification to an AWS Lambda function when any new object is
created. Use the Lambda function to publish the data to Amazon Aurora. Use
Aurora as a source to create an Amazon QuickSight dashboard. Lastly, D, Use
AWS Glue bookmarks to read sensor data from the S3 bucket in real time.
Publish the data to an Amazon RDS database. Use the RDS database as a
source to create a Tableau dashboard. Now, if you're thinking what I'm
thinking, you'll say it's fair to exclude C and D. C involves multiple steps and
services. We've got S3, Lambda, Aurora, so tons of latency, not to mention
how operationally complex the solution is. You need to manage notifications,
lambda functions, and database writes to name a few. Besides, Aurora is a
relational database, which means it's more suited for transactional workloads
rather than real-time analytics. Similarly, we can exclude AWS Glue for the
same reasons. It's designed for batch processing and not streaming data.
And just like C, it doesn't meet the low latency requirements nor are these
services suitable for real-time analytics and dashboarding. Now this leaves
us with A and B as potential solutions. A is more direct because it uses
Apache's native connector, and it uses Grafana for dashboarding. B, on the
other hand, has an added layer. It uses Data Firehose to deliver the data to
Timestream. Also, Amazon Kinesis Data Firehose doesn't natively support
Amazon Timestream as a delivery destination because you cannot create a
Data Firehose delivery stream that publishes data directly to Timestream. In
addition, in terms of the analysis tool, QuickSight is typically slower than
Grafana for real-time visualization. And so for all of the above, given that the
key requirement is low latency, it's fair to say that A is the optimal choice for
this particular question.

Practice Exam Questions: Data Store Management

The second domain for the data engineering certification is data store
management. And once again, let's warm up before diving into the sample
exam question. The first question is, which of the following services is a
general purpose data storage on AWS, Amazon EC2, Amazon S3, Amazon
RDS or AWS Lambda? Amazon EC2 provides resizable compute capacity in
the cloud. From a storage perspective, while EC2 instances can have
attached storage such as EBS volumes and the instance store, their primary
purpose isn't for general purpose data storage, but rather for running
applications and processing data. RDS is a fully managed relational
database. In terms of storage, while RDS manages relational databases, it's
mainly optimized for transactional and relational data rather than being a
general purpose storage. Lambda is a serverless compute service. It allows
you to run code without provisioning or managing servers. In terms of
storage, it doesn't provide persistent storage for general purpose data. In
contrast, Amazon S3 is indeed a general object storage service. It's designed
for storing and retrieving any amount of data from anywhere on the web. So
going back to our question, Amazon S3 is the correct answer. The next
question is, which of the following file formats is not typically used for storing
data in S3? Is it JSON, XML, YAML, EXE or Parquet? Questions on different file
types usually come up in the exam, and it's one of those things that you can
easily score if you understand the different uses of these types. To answer
this question, let's review the file formats. JSON files are lightweight files that
are easy for humans to read and write. They're also easy for machines to
parse and generate. Therefore, they're usually used for APIs and
configuration files. XML is a markup language that defines a set of rules for
encoding documents in a format that is both human and machine-readable.
So similar to JSON, but a bit more verbose. YAML is also human-readable. It's
a data serialization standard often used for configuration files. YAML enables
you to create complex data structures with simple syntax. And as you might
already know, EXE stands for executable files. If you've ever worked with a
Windows machine, you'll probably be familiar with the process of
double-clicking an executable to install a program on your machine. Lastly,
Parquet is a columnar storage file format designed for efficient data storage
and retrieval. It's a binary file format, so it's not human-readable. And all of
these files are likely to be stored in S3, except for executable files. Of course,
this is a super trivial question, but it goes to say that you need to understand
the ins and outs of each file type to determine which file to use, depending
on any given scenario. Next question is which of the following databases is a
fully managed NoSQL database in AWS? Is it Amazon Aurora, Amazon
Redshift, RDS or DynamoDB? To determine the fully managed NoSQL
database, let's quickly review each service. Aurora is a fully managed
relational database service that's compatible with both MySQL and
PostgreSQL. So it's not a NoSQL database. If anything, it uses SQL to manage
and query relational data. Amazon Redshift is a fully managed data
warehouse designed for large-scale data analytics, so definitely not intended
for NoSQL workloads. RDS, like Aurora, is designed for SQL databases. And
technically speaking, Aurora is one of the engines offered by RDS. This
leaves us with DynamoDB, which indeed is a fully managed NoSQL database
service. So when a question has a NoSQL requirement or requires some
schema flexibility, this is your cue to go with DynamoDB. And so here, the
correct answer is Amazon DynamoDB. Which S3 storage class is the most
cost effective for infrequently accessed data, Standard S3, S3
Intelligent-Tiering, S3 Standard-IA or S3 Glacier? To answer this, let's review
access tiers. Standard S3 offers high durability and availability with no
retrieval fees. So it's ideal for frequently accessed data. However, it does
offer higher storage costs compared to S3 Standard and frequent access.
Therefore, it's less cost effective for infrequently accessed data. S3
Intelligent-Tiering automatically moves data between the infrequent and
frequent access tiers to optimize costs based on the changing access
patterns. It's a great feature, but it does incur an additional fee for
monitoring and automation. So unless you're working with an unpredictable
workload, you're better off selecting a specific access tier. S3 Standard-IA
provides low storage costs and quick access for infrequently accessed data.
It does this while maintaining high durability and availability. Just keep in
mind that it charges retrieval fees and requires a minimum storage duration,
making it less suitable for frequently accessed data. Lastly, S3 Glacier
provides extremely low storage costs for long-term data archiving with high
durability. However, given that retrieval times can range from minutes to
hours. it's not suitable for scenarios that require quick access. Therefore, for
infrequently accessed data, the correct answer is the S3 Standard-IA tier.
Next question, which S3 feature helps prevent accidental data loss by
keeping multiple versions of an object? Is it S3 Transfer Acceleration, S3
Replication, S3 Versioning or S3 Lifecycle Policies? S3 Transfer Acceleration is
designed to speed up the transfer of files to and from Amazon S3 using
CloudFront's globally distributed edge locations. And it doesn't have any
functionality related to versioning or preventing data loss. S3 replication
allows you to automatically replicate objects and their metadata across
different S3 buckets. You can do this within the same AWS region or across
different AWS regions. And while it enhances data durability and availability,
it doesn't keep multiple versions of an object to prevent accidental data loss.
In contrast, S3 versioning helps prevent accidental data loss by keeping
multiple versions of an object in the same bucket. When this feature is
enabled, S3 preserves existing versions of objects whenever they're
overwritten or deleted. Lastly, S3 Lifecycle Policies manage the lifecycle of
objects in S3. They allow you to transition objects between different storage
classes or to delete them after a specified period. And while useful for cost
management and data retention, they don't keep multiple versions of
objects. Therefore, the correct answer is S3 Versioning, given that it enables
you to recover from unintended changes. Next question, which of the
following does not support transaction processing? Is it Amazon Redshift,
Amazon RDS, Amazon Aurora or Amazon DynamoDB? In other words, which
one of these is not designed for OLTP, i.e. online transaction processing, and
which ones are designed for OLAP, aka online analytical processing? So let's
go over these services. Amazon Redshift is a data warehouse service
designed for large-scale data analytics and reporting. While it can handle
SQL queries and support some transactional capabilities, it's primarily
optimized for read-heavy analytical workloads rather than transaction
processing. In contrast, Amazon RDS supports transaction processing, as it
can manage various relational database engines such as MySQL, PostgreSQL,
Oracle or SQL Server. All of these engines are designed for traditional online
transaction processing workloads, or OLTP, for brevity. Likewise, Amazon
Aurora is a MySQL and PostgreSQL-compatible relational database that was
built specifically for the cloud. As a matter of fact, it's one of the engines
offered by RDS, so it too supports transaction processing. Lastly, Amazon
DynamoDB, as mentioned earlier, is a fully managed NoSQL database
service that supports transaction processing. Therefore, from this list, the
only OLAP service is Amazon Redshift. Now let's switch over from warmup to
an actual practice exam question. The question reads, "A data management
team is tasked with handling a project that includes continuously updated
datasets in Amazon S3 and transactional data stored in a DynamoDB table.
To adhere to data retention policies and reduce storage expenses, the team
needs to establish a method that keeps only the necessary current data in
both storage systems. What strategy should the team implement to
efficiently manage the lifecycle of data in S3 and DynamoDB, ensuring
compliance with retention policies and cost effectiveness?" The options are
A, Enable S3 Object Lock in compliance mode on the S3 bucket to prevent
unintentional object deletions, and configure DynamoDB TTL to automatically
remove outdated data regularly. B, Set up a S3 Lifecycle policy to move older
objects to S3 Glacier and configure DynamoDB Streams to record changes in
the DynamoDB table, storing these changes in a dedicated archival S3
bucket. C, Enable versioning on the S3 bucket and configure S3 Lifecycle
policies to remove old object versions after a certain time for DynamoDB,
enable TTL to automatically delete items that have exceeded their useful
lifespan. D, Utilize S3 Intelligent-Tiering to automatically transfer infrequently
accessed objects to more cost-effective storage classes, and set up
DynamoDB Global Secondary Indexes (GSIs) to efficiently query data without
affecting the performance of the main table." Looking at these choices, one
way to go about this is to go straight to the choice that meets the
requirement of keeping only the necessary current data in both storage
systems, which is achieved in C by enabling versioning and lifecycle policies
in S3 and enabling TTL in DynamoDB. The other way to go about this
question is via elimination of choice. In A, the S3 Object Lock feature is
primarily used to prevent deletions for compliance and data protection
reasons, not for managing the lifecycle of non-essential data. In B, having a
S3 lifecycle policy to move objects to Glacier is a cost-effective way to
archive all their data, but it doesn't directly address deleting unnecessary
data. Lastly, in D, S3 Intelligent-Tiering helps with cost optimization for
infrequently accessed data, but it doesn't handle data deletion. And
DynamoDB global secondary indexes improve query performance, but don't
contribute to managing or deleting outdated data, hence why C is for sure
the correct answer for this question.

Practice Exam Questions: Data Operations and Support

Now let's do a recap of the third domain of the data engineering certification,
data operations and support. The first question is, which service is primarily
used for interactive data analysis and can run SQL queries on data? Is it
Amazon EMR, AWS Glue, Amazon Athena or Amazon DataBrew? So for
interactive data analysis, Amazon EMR is not a suitable choice. While EMR
can run SQL queries through tools like Hive, Presto, and Spark SQL, it's
designed for processing vast amounts of data using big data frameworks
such as Apache Hadoop, Spark, and Flink. Basically, it's more suited for
complex data processing tasks and large-scale data analytics rather than
interactive querying. AWS Glue is an ETL service that's primarily used for
data preparation and transformation rather than direct interactive data
analysis using SQL. Amazon Athena is an ad hoc interactive query service
that allows you to analyze data on the fly by enabling you to run SQL queries
on data in S3 and other data sources. Lastly, Amazon DataBrew allows you to
visually clean and normalize your data without writing code. So it's an
excellent tool for cleaning, structuring, and enriching your data, but it's not
suitable for running interactive queries. Therefore, the correct answer is
Amazon Athena. Next question is, how can you create a persistent, central
data catalog for an EMR Apache Hive metadata store with the least
operational overhead? Generally speaking, there are two ways to create a
data catalog, internally with an EMR, this is the default option, or externally
outside of EMR. The key word here is least operational overhead. So we
should focus on external data catalogs, given that the default option involves
a ton of operational overhead. For external catalogs, you have two main
options. You can either go with an AWS Glue Data Catalog or an external Hive
metastore using either Amazon RDS or Aurora. AWS Glue is the simpler and
more effective solution, as it's easier to set up and configure. The RDS or
Aurora option involves setting up a relational database to serve as the Hive
metastore. So it requires more setup and maintenance compared to AWS
Glue. Next, let's take a look at this one. You need to scan S3 buckets across
your data pipeline to make sure they don't contain PII. Next, let's take a look
at this one. You need to scan S3 buckets across your data pipeline to make
sure they don't contain PII. How can this be achieved? Commonly provided
options usually include Macie, GuardDuty, WAF, and DataBrew. If the
question revolves around data discovery, data classification, automated
monitoring or data protection, especially in the context of PII and sensitive
data, then an answer that involves enabling Amazon Macie is your best bet.
Macie uses machine learning and pattern matching to automatically
discover, classify, and protect sensitive data like personally identifiable
information in your AWS environment. Be careful though because Macie can
be used as a distractor, even in questions where data discovery and
classification are mentioned. In these types of questions, the primary focus
will be other aspects of data security or management such as encrypting or
decrypting data, access management and managing permissions, intrusion
detection and prevention. For detecting and responding to attacks, the
correct answer might involve AWS GuardDuty or AWS WAF. Data backup and
disaster recovery questions might involve AWS Backup or other database
recovery features. Lastly, if the question revolves around logging and
monitoring, the correct answer might involve services such as AWS
CloudTrail, CloudWatch or AWS Config. Next, you're tasked to enable a group
of non-technical data analysts in validating and enriching incoming data. The
process should be easy to automate once it's defined. What service can help
accomplish this? So for this one, we're looking for a low or no-code solution
because our data analysts are non technical, and our requirements entail
validating and enriching the incoming data. Choices for this type of question
usually include Amazon Athena, AWS Glue, Amazon EMR, and AWS Glue
DataBrew. Now if the question is about enabling non-technical users, next
you're tasked with enabling a group of non-technical data analysts in
validating and enriching incoming data. The process should be easy to
automate once it's defined. What service can help accomplish this? Choices
for this type of question usually include Amazon Athena, AWS Glue, Amazon
EMR, and AWS Glue DataBrew. Given that the question here is about
enabling non-technical users to easily clean, normalize, and enrich the data,
then AWS Glue DataBrew is your best option. DataBrew provides a visual
interface that allows users to apply transformations, validation, and even
enrichments to their data without writing any code. From there, the data can
be invested into S3 and ultimately analyzed via Athena and QuickSight. And
if the question involves performing complex transformations using SQL and
Python where coding is allowed, then Athena Notebook would be a more
suitable choice. Here's the last one. Your company has several applications,
sending custom CloudWatch logs to several log groups. You need to
aggregate data, monitor application logs, and set up anomaly detection for
these logs as they arrive in real time. How can this be achieved? Here the
solution is going to involve Amazon OpenSearch. Other choices might include
Amazon CloudWatch, Amazon Kinesis, and AWS Lambda. If the question
involves near real-time aggregation, monitoring, and anomaly detection for
CloudWatch logs, then OpenSearch is likely to be the best fit. However, if the
question focuses on simply storing logs, performing basic monitoring or
simple logging without the need for real-time analytics or anomaly detection,
other services such as CloudWatch logs may be more appropriate in that
scenario.

Practice Exam Questions: Data Security and Governance

If you've made it this far, you're so close to the finish line where we review
the last domain for the exam, and that is data security and governance. Let's
start with this scenario. You're trying to connect an EC2 instance to a private
RDS instance. The routing and IAM permissions are correct. What is the likely
issue? So commonly provided options usually include security groups,
network access control lists, IAM roles, and VPC peering. If the question
involves troubleshooting connection, whereby IAM permissions and routing
are configured properly, then the likely issue could be related to security
groups. Let's go over the troubleshooting steps for such a scenario. First, you
would need to verify that both the security groups and the network access
control lists, or NACL, for brevity are configured to allow database traffic
across your network. You do so by ensuring that the database security group
allows inbound traffic on the appropriate port from the security group
associated with the EC2 instance and that the security group of the EC2
instance allows outbound traffic to flow to the database security group. If all
is good, then the next troubleshooting step is to confirm that the NACLs for
these subnets containing the EC2 instance and the RDS instance are
configured to allow the necessary database traffic. Therefore, key words like
connectivity, issues with security groups or inbound and outbound traffic
rules could hint that security groups are to blame for connection failures. On
the other hand, if the question focuses on IAM roles, VPC peering or NACL
issues, then security groups might be a distractor in that case. Next, your
Redshift database contains PII that shouldn't be exposed to analysts, but
cannot be removed from the data. How can you give analysts access to the
database? So with such questions, most likely you'll be dealing with granting
SELECT and UPDATE privileges. You can grant SELECT and UPDATE privileges
explicitly on tables and columns. This allows fine-grained control over
specific data elements within the database. The second option is row-level
security policies, which, as the name conveys, allow you to restrict access to
specific rows based on predefined conditions such as user roles or
permissions. Another option is resorting to dynamic data masking. This
allows you to hide specific PII fields from analysts. So if this is our raw data,
you can use the DDM feature to perform a partial mask where you hide a
piece of the data or perform a full mask to mask the entire column. Dynamic
data masking applies transformation at the time of the querying, so it
doesn't affect the underlying data. Next, we've got this scenario where
you're required to encrypt data at rest in your organization's S3 bucket.
However, you're also required to have control over encryption keys and their
associated key policies. How can this be achieved? The encryption options
for S3 are S3 managed keys. These are managed entirely by AWS. So you
don't have control over the key policies, and therefore, they don't meet the
requirements. The second option is KMS keys. These can be managed by
AWS or by you as a customer. The third option is customer-provided keys.
The main difference between KMS keys and customer-provided keys is that
you would need to fully manage the customer-provided keys. So more
control, but also more operational overhead. Now, given that the question
doesn't specify a need for managing keys outside of AWS nor does it require
the highest level of operational control, you can stick to using KMS keys. This
option provides the necessary control over the encryption keys while
minimizing operational overhead compared to managing customer-provided
keys. Finally, data engineers that manage a Redshift cluster in another AWS
account need access to a materialized view from a Redshift cluster that you
manage. How can you share the materialized view without giving those data
engineers access to your Redshift cluster? To achieve this, your best bet is
Redshift's data-sharing feature. This feature will allow you to securely share
data without having to provide direct read-only access to your Redshift
cluster. This feature reflects changes in a live manner. So the shared cluster,
which is Redshift Cluster B in this diagram, will always be up to date. You can
also achieve this across different regions, accounts, and even between
different cluster types. And with this lesson coming to an end, we've come to
the end of our data engineer exam review course. This was Noreen Hasan. I
wish you the best of luck on your exam. And until next time, remember all
Gurus get there one lesson at a time.

Course 2