Module 3 - Adding A Storage Layer
Module 3 - Adding A Storage Layer
Finally, you will be asked to complete a knowledge check that will test your understanding
of key concepts covered in this module.
As each module introduces new features, those parts of this larger diagram will be revealed.
In this module, you start with one of the simplest of architectures that can be
implemented on AWS, which is creating a static website by hosting it entirely on Amazon
S3. You will also learn about the various Amazon S3 storage options and some key
considerations for when you choose a Region on AWS.
The café has a single location in a large city, where they sell desserts and coffee. The
business is owned by Frank and Martha, a husband-and-wife team who work at the café.
Their daughter, Sofía, and their other employee, Nikhil—who is a secondary school
student—also work at the café.
The café currently doesn’t have a marketing strategy. They mostly gain new customers
when someone walks by, notices the café, and decides to give it a try. The café has a
reputation for high-quality desserts and coffees, but their reputation is limited to people who
have visited, or who have heard about them from their customers.
Sofía suggested that they should expand community awareness of what the cafe has to
offer. Frank and Martha agreed. The café doesn’t have a web presence yet, and they don’t
currently use any cloud computing services. However, that situation is about to change.
The first challenge will be to create a basic website for the café.
You will learn more details about the business requirements—and how to use Amazon Web
Services to meet those business requirements—throughout this module.
Introducing Section 2: Using Amazon S3.
Amazon S3 is an object storage service. It enables you to store virtually unlimited amounts
of data. Data files are stored as objects. You place objects in a bucket, which you define.
Every bucket must have name that is globally unique across Regions. This means that
the bucket name must be unique across all AWS customer accounts.
The objects you store can vary in size from 0 bytes to 5 TB. Though individual objects
cannot be larger than 5 TB, you can store as much total data as you need.
First, it has a key, which is the name that you assign to an object. You use the object key to
retrieve the object. In the AWS Management Console, you can create a directory inside a
bucket, and upload an object to that directory. However, in reality, Amazon S3 does not
know about directories, so the key value includes the full path relative to the bucket root.
Objects also include a version ID. In a bucket, a key and version ID uniquely identify an
object. You will learn more about versioning later in this module.
The value of the object is the actual content that you store. It can be any sequence of bytes.
Object values are immutable, which means that after you upload an object, you cannot
modify the value. If you want to modify the object, you must make a change outside of
Amazon S3 and then reupload the object.
Objects also include metadata, which is a set of name-value pairs you can use to store
information about the object. You can assign metadata, which is referred to as user-defined
metadata, to your objects in Amazon S3. Amazon S3 also assigns system-metadata to
these objects, which it uses for managing objects.
Amazon S3 provides many features that make it an important component of many solutions
built on AWS.
First, it provides durability, which describes the average annual expected loss of objects. 11
9s of durability means that every year, there is a 0.000000001 percent chance of losing an
object. For example, if you store 10,000 objects on Amazon S3, you can expect to incur a
loss of a single object once every 10,000,000 years on average. Amazon S3 redundantly
stores your objects on multiple devices across multiple facilities in the Amazon S3 Region
you designate. Amazon S3 is designed to sustain concurrent device failures by quickly
detecting and repairing any lost redundancy. Amazon S3 also regularly verifies the integrity
of your data by using checksums.
Amazon S3 also provide four 9s (or 99.99 percent) of availability. Availability refers to
your ability to access your data quickly, when you want it. It also provides a virtually
unlimited capacity to store your data, so it is scalable. Amazon S3 has robust security
settings. It provides many ways to control access to the data that you store, and also
enables you to encrypt your data.
Now that you know about many Amazon S3 features, how can you use these features to
address your needs?
In this section of the module, you will learn about four common use cases that use Amazon
S3 as an essential part of a robust architectural solution.
On common use scenario for Amazon S3 is to use it for media hosting. In this use case,
Amazon S3 is used to store and distribute videos, photos, music files, and other media. This
content can be delivered directly from Amazon S3 because each object in Amazon S3
has a unique HTTP URL.
Alternatively, Amazon S3 can serve as an origin store for a content delivery network (CDN),
such as Amazon CloudFront. The elasticity of Amazon S3 makes it well-suited for hosting
web content that needs bandwidth to address extreme demand spikes. Also, because you
do not need to provision storage for Amazon S3, it works well for fast growing websites that
host data-intensive, user generated content, such as video and photo-sharing sites.
By default, all S3 buckets are private and can be accessed only by users who are explicitly
granted access. It is essential that you manage and control access to Amazon S3 data.
AWS provides many tools and options for controlling access to your S3 buckets or
objects, such as:
Using Amazon S3 Block Public Access. These settings override any other policies or
object permissions. Enable Block Public Access for all buckets that you don't want to
be publicly accessible. This feature provides a straightforward method for avoiding
unintended exposure of Amazon S3 data.
Writing AWS Identity and Access Management (IAM) policies that specify the users
or roles that can access specific buckets and objects.
Writing bucket policies that define access to specific buckets or objects. This
option is typically used when the user or system cannot authenticate by using IAM.
Bucket policies can be configured to grant access across AWS accounts or to grant
public or anonymous access to Amazon S3 data. If bucket policies are used, they
should be written carefully and tested fully. You can specify a deny statement in a
bucket policy to restrict access. Access will be restricted even if the users have
permissions that are granted in an identity-based policy that is attached to the users.
Creating S3 Access Points. Access points are unique hostnames that enforce
distinct permissions and network controls for requests that are made through it.
Customers with shared datasets can scale access for many applications by creating
individualized access points with names and permissions that are customized for
each application.
Setting access control lists (ACLs) on your buckets and objects. ACLs are less
commonly used (ACLs predate IAM). If you use ACLs, do not set access that is too
open or permissive.
AWS Trusted Advisor provides a bucket permission check feature. It is a useful tool
for discovering if any of the buckets in your account have permissions that grant
global access.
Here are three different general approaches to configuring access to objects in an S3
bucket.
The scenario on the left shows the default security settings for Amazon S3. By default,
all Amazon S3 buckets and the objects stored in them are private (protected). The only
entities with access to a newly created, unmodified bucket are the account administrator
and the AWS account root [Link] resource owner can grant specific access permissions
to others, but anyone not granted those permissions will not have access.
The scenario in the middle shows an occasion where S3 security settings have been
disabled and anyone can publicly access the objects stored in the bucket.
The scenario on the right shows a case where Amazon S3 was configured to provide
controlled access. User A was granted access to the objects in the bucket, but User B was
denied access. Controlled access scenarios are common. They can be configured by the
bucket owner by using one or more of the tools or options for controlling access to Amazon
S3 data this module discussed earlier.
When your objective is to protect digital data, data encryption is an essential tool. Data
encryption takes data that is legible and encodes. Encrypted data is unreadable to anyone
who does not have access to the secret key that can be used to decode it. Thus, even if
an attacker gains access to your data, they cannot make sense of it.
You have two primary options for encrypt data stored in Amazon S3.
When you set the Default encryption option on a bucket, it enables server-side encryption.
With this feature, Amazon S3 encrypts your object before it saves the object to disk. And
then Amazon S3 will decrypt it when you download the object.
Client-side encryption is the other option. When you use this approach, you encrypt the data
on the client side before you upload it to Amazon S3. In this case, you manage the
encryption process, the encryption keys, and related tools. Like server-side encryption,
client-side encryption can reduce risk by encrypting the data with a key that is stored in a
different mechanism than the mechanism that stores the data itself.
A second Amazon S3 use case is to use the service to host a static website. On a static
website, individual webpages include static content. They might also contain client-side
scripts.
To host a static website, configure an S3 bucket for website hosting. Then, upload your
website content to the bucket.
The example shows that the static site might consist of HTML files, images, videos, and
client-side scripts in formats such as JavaScript.
With this approach, you do not need to run a virtual machine that hosts a web server. In
fact, you do not need to run a server. However, you can still host a website. Amazon S3
provides a low-cost solution for web hosting that includes high performance, scalability, and
availability
Amazon S3 provides customers with a highly secure and durable storage infrastructure.
Versioning offers an additional level of protection. It provides a way to recover data if an
application fails, or when customers accidentally overwrite or delete objects.
Versioning is a method of keeping multiple variants of an object in the same bucket. You
can use versioning to preserve, retrieve, and restore every version of every object stored in
an S3 bucket.
If you delete an object, instead of removing it permanently, Amazon S3 inserts a
delete marker, which becomes the current object version. You can always restore
the previous version.
Overwriting an object results in a new object version in the bucket. You can always
restore the previous version.
Now, the educator might choose to demonstrate Amazon S3 versioning using the AWS
Management Console.
Cross-origin resource sharing (CORS) defines a way for client web applications that are
loaded in one domain to interact with resources in a different domain. With CORS support,
you can build rich client-side web applications with Amazon S3 and selectively allow
cross-origin access to your Amazon S3 resources.
To configure your bucket to allow cross-origin requests, you create a CORS configuration. A
CORS configuration is an XML document with rules that identify:
The origins that you will allow to access your bucket.
The operations (HTTP methods) that will support for each origin. In this example,
PUT, POST, and DELETE requests are allowed from the
[Link] origin, which could be configured using Amazon Route
53 to be another S3 bucket.
Other operation-specific information.
For more information about CORS, see the Cross-Origin Resource Sharing (CORS) AWS
documentation.
In this guided lab, you will complete the following tasks:
1. Create a bucket in Amazon S3
2. Upload content to your bucket
3. Enable access to the objects
Your educator might choose to lead a conversation about the key takeaways from this
guided lab after you have completed it.
You can also use Amazon S3 as a data store for computation or large-scale analytics, such
as financial transaction analysis, clickstream analytics, and media transcoding. Amazon S3
can support these workloads because of its horizontal scaling ability, which enables
multiple concurrent transactions.
In the example here, an Amazon Elastic Compute Cloud (Amazon EC2) Spot Fleet is spun
up when the bid price for Spot Instances is low, or when an Amazon EMR cluster is spun
up. Regardless, after the compute capacity is available, raw unprocessed data is extracted
from Amazon S3 and also from another data source. The data is run through compute
algorithms that integrate and transform it. The resulting processed data is loaded into a
different Amazon S3 bucket. Now that the data has been processed, the compute capacity
is terminated to save on costs. Finally, an analytics tool, such as Amazon Quick Sight,
might be used to harvest meaningful insights from the processed data. This is just one
example scenario of how Amazon S3 can play an essential role for data storage in a large
-scale analytics solutions architecture.
In the fourth and final use case discussed in this module, Amazon S3 is used as a data
backup solution. Because of its highly durable and scalable nature, Amazon S3 works well
as a data backup and archival tool.
In the scenario, data is backed up from an on-premises corporate data center, and also
from a large number of Amazon EC2 servers. These servers run applications that generate
data.
Additionally, you can move long-term data from Amazon S3 standard storage to Amazon
Simple Storage Service Glacier. This process will discussed in further detail later in this
module. Another Amazon S3 option you can configure on your buckets—to achieve even
higher levels of durability—is cross-Region replication. In cross-Region replication, objects
that are uploaded to a bucket in one Region will be automatically copied to other S3 buckets
in other Regions.
Many customers develop big data analytics applications that use Amazon S3 for object
storage. These applications often require access to an object immediately after a write.
Prior to December, 2020, Amazon S3 provided eventual consistency for overwrite PUTS
and DELETES in all Regions. However, Amazon S3 is now strongly consistent for all new
and existing S3 objects in all AWS Regions.
Amazon S3 achieves high availability by replicating data across multiple servers within
AWS data centers. If a PUT request is successful, the data is safely stored. Any read (GET
or LIST) that is initiated following a successful PUT response will return the data written by
the PUT. This strong read-after-write consistency exists automatically for all applications,
without changes to performance or availability.
While objects are strongly consistent, Amazon S3 bucket configurations have an eventual
consistency model. For example, if you delete a bucket and immediately list all buckets, the
deleted bucket might still appear in the list. However, within a short period of time, if you run
the list bucket command again, the deleted bucket will no longer appear in the list buckets
results.
Now that you have built a website using Amazon S3, here is a comparison of the different
Amazon S3 storage classes and their characteristics.
S3 Standard offers high durability, availability, and performant object storage for frequently
accessed data. Because it delivers low latency and high throughput, S3 Standard is
appropriate for a wide variety of use cases, including cloud applications, dynamic websites,
content distribution, mobile and gaming applications, and big data analytics. It provides
durability across at least three Availability Zones.
S3 One Zone-IA stores data in a single Availability Zone. It is ideal for customers who want
a lower-cost option and who do not need the availability and resilience of S3 Standard or S3
Standard-IA. It’s a good choice for storing secondary backup copies of on-premises data or
easily re-creatable data. You can also use it as cost-effective storage for data that is
replicated from another AWS Region.
Amazon S3 Glacier is a secure, durable, and low-cost storage class for data archiving. You
can reliably store any amount of data at costs that are competitive with or cheaper than on-
premises solutions. To keep costs low, but suitable for different needs, you have three
options for retrieving data, with varying access times and cost:
Expedited retrievals are typically made available within 1–5 minutes
Standard retrievals typically complete within 3–5 hours
Bulk retrievals typically complete within 5–12 hours
Amazon S3 Glacier Deep Archive is the lowest-cost storage class for Amazon S3. It
supports the long-term retention and digital preservation for data that might be accessed
once or twice in a year. Data is stored across at least three geographically dispersed
Availability Zones, protected by 11 9s (99.999999999 percent) of durability, and can be
restored within 12 hours
For more details about Amazon S3 Storage Classes, see the AWS Documentation
You can configure the lifecycle of your objects to manage how they are stored throughout
their lifecycle. A lifecycle configuration is a set of rules that define actions that Amazon S3
applies to a group of objects.
After anS3 lifecycle policy is set, your data will automatically transfer to a different storage
class without any changes to your application.
By using lifecycle policies, you can cycle data at regular intervals among different Amazon
S3 storage types. This cycling reduces your overall cost because you pay less for data as it
becomes less important over time. In addition to being able to set lifecycle rules per object,
you can also set lifecycle rules per bucket.
For more information about object lifecycle management, see the Object Lifecycle
Management AWS documentation details.
With Amazon S3, you pay only for what you use. There is no minimum fee. There are four
cost components to consider when you decide which Amazon S3 storage class best fits
your data profile—storage pricing, request and data retrieval pricing, data transfer and
transfer acceleration pricing, and data management features pricing.
In the guided lab earlier in this module, you uploaded files to Amazon S3 by using the web
browser interface provided by the AWS Management Console. It ist he simplest way to
move data in to or out of Amazon S3. It offers a wizard-based approach, including the
option to drag and drop files that you want to copy into a bucket.
In this section of the module, you will learn about some additional options available for
moving data into and out of Amazon S3.
Two of those options include using the AWS Command Line Interface (AWS CLI) or the
AWS SDKs.
An example AWS CLI upload command is shown. In the command, you specify aws to
invoke the AWS CLI, then you specify the service, which is S3. Next, you issue a cp(or
copy) subcommand, followed by [Link] which is the local file (that exists on your computer)
that should be copied. Finally, the s3://AWSDOC-EXAMPLE-BUCKET/[Link] parameter
indicates the bucket where the file should be uploaded, and the key (AWSDOC-EXAMPLE-
BUCKET/[Link]) where the object value (contents) should be stored.
Note that files must be at least 5 MB in size to use the multipart upload feature.
Amazon S3 Transfer Acceleration enables fast and easy data transfer into an S3 bucket by
taking advantage of Amazon CloudFront and AWS edge locations, which are globally
distributed. This data is then routed to Amazon S3 over an optimized network path.
If you tried to transfer 100 petabytes of data over the internet, with an upload speed of 10
Gbps (assuming a TCP/IP overhead of 10%), it would take approximately 1018 days
(almost three years) to finish uploading the data. That would not be practical. In such cases,
using AWS Snowmobile to transfer the data would be a better option.
Snowmobile uses multiple layers of security designed to protect your data, including
dedicated security personnel, GPS tracking, alarm monitoring, 24/7 video surveillance, and
an optional escort security vehicle while in transit. All data is encrypted with 256-bit
encryption keys managed through AWS Key Management Service (AWS KMS) and
designed to ensure both security and full chain-of-custody of your data.
Introducing Section 5: Choosing Regions for your architecture.
There are many considerations when you decide what Region to host your data in.
First, you should consider data privacy laws and your regulatory compliance requirements.
Data you store on AWS is subject to the laws of the country and locality where it is stored.
In addition, some laws dictate that if you are operating your business in their jurisdiction,
you cannot store that data anywhere else. Similarly, compliance standards (such as the
U.S. Health Insurance Portability and Accountability Act, or HIPAA) have strict guidelines on
how and where data can be stored.
Second, proximity is an important factor in choosing your Region, especially when latency is
a critical factor. In most cases, the latency difference between using the closest Region and
the farthest Region is relatively small, but even small differences in latency can impact
customer experience. Customers expect responsive environments, and as time passes and
technology becomes more and more powerful, those expectations also rise.
When you choose a Region, a third important consideration is the availability of AWS
services and features. Though AWS strives to make services and features available
everywhere, the complications that arise from having a global reach make it challenging to
accomplish that goal. Instead of waiting until a service is available everywhere before
launching it, services are released when they are ready. Service availability is then
expanded as soon as possible.
A fourth consideration when you choose a Region is cost. Service costs can differ
depending on which Region they are used in. For example, an Amazon EC2 instance in the
us-east-1 Region might not cost the same as if it ran in the eu-west-1 Region. Typically, the
difference in cost might not be enough to supersede the other three considerations.
However, in cases where the latency, compliance, and service availability differences
between Regions are minimal, you might be able to save by using the lower-cost Region for
your environment.
Finally, in circumstances where your customers are in different areas of the world, consider
optimizing their experience by replicating your environment in multiple Regions that are
closer to them. Because you would then be distributing your load across multiple
environments, your costs for components in each environment might go down even as you
add more infrastructure. For example, adding a second application environment might allow
you to cut your processing and storage capacity requirements in half in each environment.
Because AWS is designed to enable that kind of flexibility, and because you only pay for
what you use, you could scale your existing environment down as a way to mitigate the cost
of adding another environment.
The downside to that approach is that you now have two environments to manage. Also, not
all of your components will scale down enough to mitigate all the costs of the new
components. Additionally, you might need to maintain one single storage source of truth in
one Region, such as a primary Amazon Relational Database Service (Amazon RDS)
instance. Your secondary Region would need to communicate with the storage instance,
which might increase latency and cost for those operations.
Sofía mentioned to Nikhil that she would like the café to have a website that will showcase
the café visually through images. The website should also provide customers with business
details, such as the location of the store, the business hours, and telephone number.
Nikhil is pleased to create the first website for the café. During this activity, you will take on
the role of Nikhil and work on producing the results that everyone back at the café hopes
you can deliver. Perhaps you can even exceed their expectations!
In this challenge lab, you will create a static website for the café. The website will be
hosted on Amazon S3. After the S3 bucket is created and properly configured for
website hosting, a web browser should be able access the website directly by using
the assigned Amazon S3 endpoint URL.
Look at the answer choices and rule them out based on the keywords that were
previously highlighted.
[Link]
[Link]
%[Link]