10

Application
Integration Services
AWS oers a suite of services that enable you to build architectures that enable
communication between the dierent components of your application in a bid to move
away from monolith designs. ese integration services facilitate design patterns for
distributed systems, serverless applications, and decoupled applications.
Ultimately, decoupling your application from traditional all-in-one monolith architectures
ensures a reduced impact when making changes. It also facilitates easier upgrades and
new features being released faster.
In this chapter, we will look at several services that oer integration capabilities.
ese include messaging solutions between application components using a queuing
service, notication services, which can be used for application-to-application (A2A)
notications or application-to-person (A2P) type notications, event-driven workow
designs, and coordinating multiple services into serverless workloads.
In this chapter, we will cover the following topics:

• Understanding notication services such as Amazon Simple Notication

Service (SNS)
• Decoupling your application architecture with Amazon Simple Queue
Service (SQS) and Amazon MQ
362 Application Integration Services

• Designing event-driven workows to connect your application data with various

AWS services using EventBridge
• Coordinating multiple AWS services into serverless workloads with Amazon Step
Functions and Amazon Simple Workow Service (SWF)

Technical requirements
To complete the exercises in this chapter, you will need access to your AWS Free Tier
account, as well as permissions to access the various AWS services. You will also need
access to the VPC you built in Chapter 6, AWS Networking Services – VPCs, Route53,
and CloudFront.

Understanding notication services such as

Amazon SNS
Amazon SNS is a push-based messaging and notication system that can be used to allow
one application component to send messages to other application components or directly
to end users.
Amazon SNS uses a publisher/subscriber model where one application component
will act as a publisher of messages and the other application components will consume
those messages as subscribers. Amazon SNS allows you to design high throughput,
many-to-many messaging between distributed systems, microservices, and event-driven
applications.
Let's look at an example. Suppose you want to be notied if any of your IAM users
upload an object to a particular Amazon S3 bucket that they have access to. To achieve
this, you can congure S3 event notications to send out an alert whenever the
s3:ObjectCreated:* action occurs. is notication can be sent to an SNS topic
(discussed later), which you subscribe to using your email address. is way, every
time your users upload a new object to your S3 bucket, Amazon SNS will send out a
notication to you via email. is is an example of A2P messaging using SNS.
Let's look at another example. Suppose you host an S3 bucket that allows end users to
upload images in a default format, and you have a requirement to convert those images
into multiple formats. To achieve this requirement, you can use Amazon Lambda, which,
as we discussed in Chapter 7, AWS Compute Services, is a serverless compute solution that
allows you to run code in response to an event or trigger.
 Understanding notication services such as Amazon SNS 363

For this specic example, you can congure your S3 notication service to send a message
to an Amazon SNS topic and have the Lambda function subscribe to that topic. e
message can include information about the new image that has been uploaded and can
trigger your Lambda function to access the image in the bucket, create dierent formats
of the image, and save them in another S3 bucket. is automated process is an example
of A2A messaging using SNS.
Next, we will take a look at a key component of the Amazon SNS services, specically
SNS endpoints.

Amazon SNS endpoints

As we mentioned previously, Amazon SNS is a push-based messaging solution, enabling
one or more publishers to push messages to one or more subscribers. With Amazon SNS,
your subscribers need to use a supported endpoint type. ese endpoints are depicted in
the following diagram:

Figure 10.1  Amazon SNS subscriber endpoints

364 Application Integration Services

Amazon SNS A2A endpoints include Amazon SQS, HTTP/S endpoints, AWS Lambda,
and Amazon Kinesis Firehose. Data from Amazon Kinesis Firehose can then be ooaded
and stored in Amazon S3 buckets, AWS Elasticsearch, and Amazon Redshi, as well as
other third-party service providers.
Amazon SNS A2P endpoints include email, mobile text messages, and mobile
push endpoints.
Amazon SNS also ensures high levels of message durability. Messages are stored and
replicated on multiple devices across geographically separated servers and data centers.

Amazon SNS topics

At the heart of the Amazon SNS service is the SNS topic feature, which is a logical access
point that acts as a communication channel between your publishers and subscribers.
Before you can send out messages to your subscribers, you need to create a topic. Your
publisher needs to be made aware of which SNS topic to send messages to and your end
clients must subscribe to the topic to be able to receive those messages.
In the following diagram, we can see that an application component allows us to upload
objects to an Amazon S3 bucket (1). Amazon S3 can be set up with an event notication
service that pushes out a notication, stating that an upload took place to an Amazon
SNS topic. In this diagram, an admin has subscribed to the SNS topic. Any notications
resulting from the objects being uploaded to the S3 bucket are then sent to the admin:

Figure 10.2  Example  conguring an SNS notication for an S3 event notication

 Understanding notication services such as Amazon SNS 365

Your publishers also need permission to be able to publish messages to the topic. In the
previous example, where we wanted to send a notication to an administrator every time
a new object was uploaded to an S3 bucket, you would also need to create permissions
that grant the S3 bucket the ability to send messages to the SNS topic. You can do this by
conguring an IAM policy that species which bucket can send messages to the topic and
attach it directly to the SNS topic as an inline policy.
Subscribers to your topic will then have messages pushed out to them whenever a new
message is published by the publisher. In the case of the preceding example, the publisher
is the S3 notication service, and the subscriber is your administrator's email address.
Note that when you create a topic, you need to assign it a name. is can be up to
256 characters in length and can contain hyphens (-) and underscores (_). Amazon
SNS will assign an Amazon Resource Name (ARN) to the topic you create, which will
include the service name  in this case, sns  the Region, the AWS account ID, and
the topic name. So, for example, an SNS topic called new-recipe-upload-alert,
created in the London Region, with an AWS account ID of 123456789789 will have
an ARN of arn:aws:sns:eu-west-2:1234567890123789: new-recipe-
upload-alert.
Next, we will look at the topics you can create on Amazon SNS, which will depend on the
application's use case.

Standard and FIFO topics

When conguring Amazon SNS, you create a standard topic by default. Standard topics
are used when the message's delivery order is not going to aect your application in any
way and where duplicating messages will not create any issues in your workow. All
supported delivery protocols support standard topics.
In addition, you can create FIFO topics. ese are designed to ensure strict message
ordering and prevent message duplication. Note that only the Amazon SQS endpoint
(specically, the Amazon SQS FIFO queue) can subscribe to a FIFO topic. We will discuss
the Amazon SQS service later in this chapter.
366 Application Integration Services

Amazon SNS Fanout scenario

A key feature oered by Amazon SNS is the ability to replicate messages pushed out to an
SNS topic across multiple endpoints. is is known as the Fanout scenario and it allows
parallel asynchronous processing.
Let's look at an example. Let's say that you are a theatre company and that you sell
tickets for your various performances. You are required to process online payments from
customers and issue them their tickets. At the same time, you are also required to store
information on all sales in AWS's data warehousing solution, which is oered by Amazon
Redshi. One way to design this architecture is depicted in the following diagram:

Figure 10.3  Example of an Amazon SNS Fanout scenario

In the preceding diagram, incoming ticket sales are sent to an SNS topic, which then gets
replicated to an SQS queue and an Amazon Kinesis Data Firehose stream. Any messages
that are sent to the SQS queue are processed by the payment function to complete the sale
transaction. Additional queues may be added to the architecture for order fulllment and
customer notication.
Next, the same message is processed by Amazon Kinesis Data Firehose, which feeds the
data into an Amazon Redshi cluster. Note that to stream data from Amazon Kinesis
Firehose to Redshi, you need to deliver the data into an S3 bucket and then issue an
Amazon Redshi COPY command to load the data into your Amazon Redshi cluster.
We will discuss Amazon Kinesis in the next chapter.
 Understanding notication services such as Amazon SNS 367

At the same time, messages will continue to reside in the S3 bucket, which can be archived
using the life cycle management process, as we discussed in Chapter 5, Amazon Simple
Storage Service (Amazon S3). is can help address any compliance requirements to store
historic information on all ticket sales.

Amazon SNS pricing

Amazon SNS is a managed service with no upfront cost. You pay based on usage and this
is based on the type of topic that's used; that is, standard topics or FIFO topics. Standard
topics are charged based on the number of API requests made per month and the number
of deliveries to the various endpoints. For example, mobile push notications are charged
at $0.50 per million notications aer you have exhausted your free tier threshold of 1
million notications.
e maximum payload size for your messages is 256 KB. Except for SMS messages, you
are billed for every 64 KB chunk as one request. So, a payload size of 256 KB is equal to
four requests. Furthermore, if you need to send messages that are larger than 256 KB, you
can use the Amazon Extended SNS Client Library, which allows you to send payloads
via the Amazon S3 service. When you do this, additional Amazon S3 storage costs are
incurred.
With regards to FIFO topics, you are charged based on the number of published messages,
subscribed messages, and their respective amount of payload data.
In this section, we looked at the Amazon push-based messaging solution oered by
Amazon SNS. Amazon SNS works based on a publisher/subscriber model and enables
you to design and architect A2A messaging and A2P messaging. Amazon SNS can help
you build integration between dierent application components, allowing you to design
distributed systems, microservices, and serverless architectures.
You also learned about some of the core features of Amazon SNS, including standard and
FIFO topics, as well as the Fanout scenario concept.
In the next section, we will look at another messaging integration service known as
Amazon SQS. Amazon SQS is a pull-based messaging solution and lends itself well
to designing decoupled architectures, enabling you to migrate away from monolith
application architectures.
368 Application Integration Services

Decoupling your application architecture with

Amazon SQS and Amazon MQ
Amazon SQS is another fully managed messaging integration solution that enables
you to decouple your application components into distributed systems and facilitate
the design and architecture of microservices. One of the primary advantages of using
a queuing system such as Amazon SQS is the ability to move away from monolithic
application designs. In a monolithic design, where all the components of your applications
are dependent on each other and always need to be available to each other, you oen
suer from frequent failures and outages. A queueing system such as Amazon SQS
can help the dierent components of your application work independently and queues
can hold messages in the form of requests/tasks until capacity becomes available. With
asynchronous processing and the ability for dierent components to scale independently,
you benet from higher levels of availability, where each component can scale as needed
without impacting the overall workow.
In the following diagram, we can see how Amazon SQS can be used to queue messages
between various components of your application and achieve a decoupled architecture
(also known as loose coupling):

Figure 10.4  Amazon SQS use case example

 Decoupling your application architecture with Amazon SQS and Amazon MQ 369

In the preceding diagram, a media transcoding example makes use of both the Amazon
SNS and Amazon SQS services to convert raw videos uploaded by users into various
formats and resolution sizes. In this architecture, we have dierent auto-scaling groups
that are provisioning a eet of servers, with each eet responsible for converting the
videos into a specic format. Separate queues are created to handle messages destined for
the dierent eets of servers to process. Here is a quick breakdown of the workow:

1. Users upload videos via a frontend web server farm that is part of an auto-scaling
group designed to scale out and scale in based on demand.
2. e videos are uploaded to a master bucket.
3. At the same time, an SNS notication is sent out to multiple Amazon SQS queues
in a fanout conguration (refer to the SNS Fanout scenario discussed earlier in
this chapter).
4. Each SQS queue holds messages for the appropriate app server to pull when
capacity is available.
5. e relevant app servers retrieve the messages from the appropriate SQS queue,
which identies the videos that need to be processed in the master bucket. e app
servers then retrieve the raw videos from the master bucket.
6. e app servers convert the format and resolution of the videos and upload the
completed videos in the correct format into the transcode bucket.

Whereas Amazon SNS oers a push-based message notication solution, Amazon SQS
is a fully managed pull-based message queue system that will also retain the messages
for a short duration (the default is set to 4 days but this can be congured to a maximum
duration of 14 days).
is means that if you have backend services that need to process lots of messages in the
queue from frontend web requests, you can retain those messages until your backend
services can process new messages in the queue. Amazon SQS increases the overall fault
tolerance of your application solutions, allowing your decoupled application components
to run independently.

Amazon SQS queue types

Amazon SQS oers two types of queues designed to help address dierent use cases. ese
are discussed next.
370 Application Integration Services

Amazon SQS standard queues

Standard queues support a nearly unlimited number of API calls per second,
(SendMessage, ReceiveMessage, or DeleteMessage) and are designed for
messages to be delivered at least once. However, this does mean that on an odd occasion,
duplicate copies of the message could be delivered. In addition, messages may not be
delivered in the order in which they were introduced into the queue. So, your application
must be able to cope with messages that are not delivered in the order in which they
entered the queue, as well as the occasional duplicate message:

Figure 10.5  Amazon SQS standard queue

Standard queues are particularly useful when you need to process vast amounts of
transactions per second. However, note that there is a 120,000 quota for the number
of inight messages for a standard queue. A typical example of where you might use
standard queues is, for example, when you need to process a high number of credit card
validation requests for an e-commerce application.

Amazon SQS FIFO queues

FIFO stands for rst-in rst-out. FIFO queues are designed to preserve the order of your
messages, as well as ensuring only one-time delivery with no duplicates.
FIFO queues only oer throughput at a rate of 300 transactions per second. is means
that they cannot oer unlimited throughput; however, high throughput of messages
can be oered by using a process known as batching, which oers support for 3,000
transactions per second, per API method (SendMessageBatch, ReceiveMessage,
or DeleteMessageBatch). ese 3,000 transactions represent 300 API calls, each with
a batch of 10 messages.
In addition, as depicted in the following diagram, messages are delivered in the order in
which they were introduced into the queue and Amazon SQS will preserve this order:

Figure 10.6  Amazon SQS FIFO queue

 Amazon MQ 371

So, FIFO queues are ideal for those applications where the order of events is important,
such as when you're making sure that user-entered commands are run in the right order.
FIFO queues also ensure no duplicates are created more than once, such as processing
payment transactions. However, they are not ideal where exceptional levels of scaling
might be required or where you need to process greater than 3,000 transactions per
second (with batching).

Amazon SQS pricing and security

To ensure sensitive data is protected, you can encrypt messages using Amazon Key
Management Service (KMS). In terms of pricing, there are no upfront costs. You pay
based on the number and content of requests and the interactions with Amazon S3 and
AWS KMS. Note that as part of the free tier, you also get the rst 1 million requests for
free every month.
In this section, we looked at another managed messaging solution oered by Amazon
SQS. Amazon SQS is a pull-based message queuing solution that allows you to decouple
your application components, enabling them to work independently of each other.
Messages are stored in either standard or FIFO queues and application components
retrieve these messages as required when there is available capacity.
In the next section, we will look at Amazon MQ, which is a message broker service
designed for Apache ActiveMQ and other message brokers.

Amazon MQ
A message broker is a piece of soware designed to help you facilitate communications
between application components to exchange information. Message brokers allow
dierent services to communicate with each other directly, even if those services are
written in dierent languages or run on dierent platforms.
Many organizations have existing message brokering services within their on-premises
environments that support their on-premises applications. One such service is Apache
ActiveMQ, which is probably one of the most popular Java-based message brokers.
When clients are looking to migrate their on-premises applications, you must consider
where such third-party message brokering services are being consumed. AWS oers a
service known as Amazon MQ, which is a fully managed message broker service that
provides compatibility with popular message brokers.
Amazon recommends using Amazon MQ to migrate applications from existing message
brokers where compatibility with APIs such as JMS or protocols such as AMQP 0-9-1,
AMQP 1.0, MQTT, OpenWire, and STOMP is required.
372 Application Integration Services

If, as part of the migration, you are looking to completely rearchitect your application
layer, then you may wish to consider Amazon SNS and Amazon SQS instead, as you do
not require third-party message brokers. Amazon recommends these services for new
applications that can benet from nearly unlimited scalability and simple APIs.
In this section, we looked at Amazon MQ, which enables customers to easily migrate to
a message broker in the cloud and oers compatibility with existing messaging brokers
such as Apache ActiveMQ and RabbitMQ. With Amazon MQ, you reduce your overall
operational overhead when provisioning, conguring, and maintaining message brokers
that depend on connectivity with APIs such as JMS or protocols such as AMQP 0-9-1,
AMQP 1.0, MQTT, OpenWire, and STOMP.
In the next section, we will look at event-driven workow services and the services oered
by Amazon to help architect event-driven solutions for your applications.

Designing event-driven application workows

using AWS EventBridge
Amazon EventBridge is a serverless event bus service that allows you to stream real-time
events from your applications, SaaS-based services, and AWS services to a variety of
targets. ese targets can include AWS Lambda, Kinesis, an HTTP/S endpoint, or another
event bus service in another account. Amazon EventBridge helps you create application
architectures where you need to react and perform some action against those events that
are generated.
Events can be generated when there is a change in the state of a given resource, such as
when an EC2 instance changes its state from a running state to a stopped state. Another
example of an event is when your auto-scaling group launches or terminates an EC2
instance. Additional functionality, as required by your application architecture, can be
created by reacting to such state changes.
With EventBridge, you set up rules that dene matching incoming patterns or events.
When an event occurs, as dened by the rule, it can be sent to a target for further
processing. For example, an event that resulted in a critical server being stopped (by
accident) can be sent to a Lambda function to have it restarted automatically.
 Designing event-driven application workows using AWS EventBridge 373

EventBridge can also be congured to trigger events at a dened schedule. For example,
if you have a large eet of EC2 instances that are used to test new applications, and you
generally run your tests every weekday from Monday to Friday during normal business
hours, then there is no need to have the servers running outside of those business hours.
You can set up a scheduled event to trigger a Lambda function that stops the servers
at 6 P.M. Monday to Friday and restarts them every weekday at 8 A.M. is ensures
that outside of normal business hours, your servers are in the stopped state. If these are
on-demand EC2 instances, you do not get charged while those servers remain stopped.
Amazon EventBridge is an updated solution from a previous version known as Amazon
CloudWatch Events. With CloudWatch Events, you were limited to a default event bus
that enabled you to route all AWS events, as well as custom events. However, with the
new Amazon EventBridge, you can introduce custom event buses in addition to the
default event bus. Custom event buses can be created exclusively for your workloads and
enable you to control access to events that are limited to a set of AWS accounts or custom
applications. In addition, you can use content-based ltering and advanced rules for
routing events. EventBridge can handle more processing, reduce the load on downstream
events, and use partner event sources such as Zendesk, PagerDuty, and Datadog.
In the following diagram, we can see how EventBridge works at a high level. We can see
the sources of events, the types of buses that can be used, and the potential targets for
those events:

Figure 10.7  How EventBridge works

374 Application Integration Services

e following are the key concepts and components of Amazon EventBridge:

• Events: As mentioned earlier, events represent a change in the state of a given

environment or resource. is can be a state change in an application, an AWS
resource, or even a SaaS partner service or application.
• Rules: A rule enables you to match an event to a target for processing. You can have
a single rule route to a single target or a parallel processing route to multiple targets.
ere is no ordering of how the rules are processed but you can customize the JSON
that's sent to the target to ensure only data of interest is passed on to the target.
• Targets: A target can process these events and perform some action. Targets include
Lambda functions, SNS topics, ECS tasks, and SQS queues. Events are passed on to
targets in JSON format.
• Event buses: Event buses can receive events. You have a default bus in your AWS
account that is used to receive events from AWS services, but you can also create
custom events for your custom applications. Partner event buses can be used to
receive events from partner SaaS applications and services, which are then directed
to your AWS account.

In this section, we looked at Amazon EventBridge, which allows you to stream

real-time events from your applications, SaaS-based services, and AWS services to a
variety of targets for processing. Targets can include AWS Lambda functions, Kinesis
streams, ECS tasks, and SQS queues, among others. In addition, you can congure
Amazon EventBridge to handle scheduled events that are triggered on a dened schedule.
Amazon EventBridge oers more exibility and advanced features compared to the
previous CloudWatch Events services.
In the next section, we will look at task-oriented integration services such as Amazon Step
Functions and Amazon SWF.

Coordinating multiple AWS services into

serverless workloads with Amazon Step
Functions and Amazon SWF
In this section, we'll look at two dierent AWS services that enable you to design
task-based workows between your application components. e rst service we
will look at is AWS Step Functions, while the second will be AWS SWF.
CoordinatingmultipleAWSservicesintoserverlessworkloadswithAmazonStepFunctionsandAmazonSWF375

AWS Step Functions

Applications tend to have several components that make up individual workows and
processes. Each workow represents an element of the application that then leads on to
the next to provide a complete end-to-end solution. Amazon Step Functions enables you
to dene these workows as a series of state machines that contain "states" that make
up the workow. ese states make decisions based on input, perform some action, and
produce an output to other states.
States can be any of the following types:

• Success or fail state: Where the execution stops with a success or failure
• Wait state: Where the state waits for a timeout period or a scheduled time
• Parallel state: Where the state performs parallel branches of execution
• Map state: Where the state accesses a list of items such as a list of orders
• Choice state: Where the state chooses between branches of execution
• Task state: A state that focuses on carrying out a specic task and may call other
AWS services, such as Lambda functions, to perform the task

e state machine coordinates work through the dierent states and uses the task state
to perform the actual work. Steps Functions helps you visualize your workow as a series
of event-driven steps, as well as the state of each step in your workow to make sure that
your application runs in a dened order.
Amazon Step Functions makes use of the Amazon States Language (ASL). is is
a JSON-formatted structured language that helps you dene your state machines,
including states such as Task states, which perform certain actions. ASL is used to
dene how states transition from one state to the next, as in the case of the Choice state,
or when you need to stop execution with an error, as in the case of a Fail state, and so on.
376 Application Integration Services

With Step Functions, you can also introduce human interaction, particularly where
manual intervention is required. Let's look at an example of a credit card application
process. You are likely to have several steps that form part of the application process. In
the following diagram, we can see those steps in detail:

Figure 10.8  Example of a credit card application workow

In the preceding diagram, we have a Step Functions workow process that illustrates how
a potential customer could apply for a new credit card. e initial workow would involve
signing up for a credit card, which would require the customer's details to be veried. e
next step could involve enabling a customer to choose the level of credit required. is
workow may include the following steps:

1. New customers sign up for a credit card. e initial process involves checking the
user's details. Several parallel Lambda functions can be invoked to perform the
required verication. For example, in the UK, the customer's name and address
could be veried against the electoral roll.
2. If the automatic verication process is successful, then a review is not required,
and the next step is invoked. If a review is required, then human intervention may
be required to perform additional verication tasks. If the human verication is
successful, the next step can be invoked; otherwise, the application can be rejected.
CoordinatingmultipleAWSservicesintoserverlessworkloadswithAmazonStepFunctionsandAmazonSWF377

3. Next, the customer is oered an auto-approval credit amount. e customer

is also given the choice to select a higher credit value, which will be subject
to additional reviews. If the customer chooses a credit value that is within the
auto-approval credit amount, then the application is automatically approved.
If the customer chooses a credit value higher than the auto-approval credit amount,
then another human intervention step is required. Here, the credit card company
may request additional information such as salary slips to check the customer's
credit-worthiness.
4. If the human intervention is successful, the application is approved; if not, the
application will be rejected. ere may be additional steps in the workow for
appealing against the rejection decision.

Step Functions enables us to build distributed application solutions and design

microservices interactions to provide a complete end-to-end solution. Next, we will look
at the dierent types of workows you can set up for Step Functions.

Workow types
With Amazon Step Functions, you can congure two types of workows, as follows:

• Standard workow: ese have an exactly once workow execution and can run
for up to 1 year. Standard workows are ideal if you require human interaction
and approval processes as part of your workow. You are charged on a per-state
transition basis, which is each time a step in your execution is completed. Standard
workows also provide access to execution history and visual debugging.
• Express workow: ese have at least once workow execution but can only run
for up to 5 minutes. Express workows are ideal for automated tasks and high event
rate workloads, such as streaming data processing and IoT data ingestion. You are
charged based on the number and duration of executions. Express workows also
oer unlimited state transition rates. Finally, all execution history is sent to Amazon
CloudWatch.
378 Application Integration Services

Here is an example of a simple workow that creates a task timer. In this example, a Step
Functions state machine is being congured that implements a Wait state and uses a
Lambda function to send out an Amazon SNS notication aer the waiting period is over.
e message that's sent out by the task is a simple Hello World message. e following
screenshot shows the workow and associated JSON:

Figure 10.9  AWS Step Functions task timer example

In this section, we looked at the two types of Step Functions execution types and identied
the core dierence between the two. Step Functions workows can run for up to 1 year
(when using the standard workow type), so they are particularly useful for long-running
application models that may also require human interaction. An example is a health
insurance claim application process, which may require human intervention to verify the
hospital bills and treatment that's dispensed to the claimant. In the next section, we will
examine another task-oriented application integration service known as Amazon SWF.

Amazon Simple Workow Service (SWF)

Amazon SWF is another task-oriented application integration service that allows you to
coordinate work across distributed components of your application. Such coordination
of tasks may involve processes such as managing dependencies, scheduling tasks, and
handling the retries and timeouts of tasks to complete the logical workow for your
application.
 Exercise 10.1  Amazon S3 event notication using Amazon SNS 379

Amazon SWF has a concept where you implement "workers" to complete your tasks.
Workers can run either in the cloud across AWS EC2 instances or on the compute services
available in your on-premises locations. As part of the logical workow, Amazon SWF
also allows you to incorporate human interaction within the logical distribution of tasks
like Amazon Step Functions, as discussed in the previous section. With Amazon SWF, you
can store tasks, assign them to workers, track progress, and maintain states.
While all of this sounds fairly like Amazon Step Functions, a key dierence is that with
Amazon SWF, you must write decider programs in any language that gets the latest state
of each task from Amazon SWF and uses it to initiate subsequent tasks.
Amazon Step Functions, on the other hand, oers a fully managed service that has a
more productive and agile approach to coordinating application components using visual
workows. If you are building new applications on AWS, you should consider using
Amazon Step Functions. However, if you require external services to interact with your
processes or you need to launch nested processes where a child process needs to return
results to a parent process, you should consider using Amazon SWF.
In this section, we provided a brief introduction to Amazon SWF. Amazon SWF enables
you to coordinate tasks across your distribution application components while oering
capabilities such as maintaining their execution state durably and reliably.

Exercise 10.1 – Amazon S3 event notication

using Amazon SNS
In the previous chapter, you designed, architected, and deployed a complete web
application using several AWS services. One such service was the Amazon S3 service,
where you created a bucket to host your application source code repository. e source
code was comprised of multiple les that helped you build your web application.
Maintaining this source code is of paramount importance and any changes that are made
to the code need to be monitored. ere are several best practice strategies you can use to
manage your source code, including using DevOps principles. In this exercise, your senior
administrator, Alice, would like to know whenever a new le (object) gets uploaded to
this source code repository, which is stored in the Amazon S3 bucket.
380 Application Integration Services

Amazon S3 comes with a feature known as event notications. is feature enables you
to receive notications when certain events occur in your S3 bucket, such as an object
being created or deleted. e service can be congured to send out such notications
to an Amazon SNS topic, which an administrator can subscribe to using an email as
the endpoint. Let's congure an Amazon S3 notication to send email alerts to Alice
whenever a new le is uploaded (that is, created) to the S3 bucket that hosts the source
code repository.
is exercise is divided into four main steps, as described in the following sub-sections.

Step 1 – creating an SNS topic and subscribing

to the topic
e rst step is to create an SNS topic that will be used as the logical access point that
Alice will subscribe to. Messages sent to this topic will then be emailed to Alice:

1. From the AWS Management Console, search for SNS in the top search box and
select the service to be taken to the Amazon SNS dashboard.
2. If you have never created an SNS topic before, you should see the Amazon SNS
splash screen.
3. Click on the far le-hand menu icon, denoted by the three lines, to expand
the sidebar.
4. Next, click on the Topics link from the menu.
5. Click the Create topic button in the right-hand pane of the screen.
6. On the Create topic page, in the Details section, select the Standard type
under Type.
7. Enter a name for the topic; for example, source-code-changes. Next, enter
a display name for the topic; for example, Source Code Changes Alert.
8. Leave all the remaining settings as their default values and click the Create
topic button.
 Exercise 10.1  Amazon S3 event notication using Amazon SNS 381

9. Once the topic has been created, you will be redirected to the topic page. Make
a note of the topic's ARN, as per the following screenshot:

Figure 10.10  Amazon SNS topic

10. Now that you have created a topic, you can create a subscription for it. We will be
using email as the endpoint for notications, and you can use your email address to
receive the notications.
11. In the bottom pane of the topics page, as per the previous screenshot, you will nd
a section to create subscriptions. Click on the Create subscription button.
12. On the Create subscription page, you will note that the topic ARN is already
selected. If not, ensure that you paste the topic ARN that you made a note of earlier.
13. Next, under Protocol, select Email from the drop-down list.
14. In the text box under Endpoint, provide your email address.
15. Click the Create subscription button at the bottom of the page.
382 Application Integration Services

16. You will get a conrmation statement to say that your subscription has been created.
However, its status will be set to Pending conrmation. AWS will have sent you a
conrmation request to your email account. You will need to log into your email
account and conrm the subscription to activate it. I have just logged into my email
account to do the same, as per the following screenshot of my Gmail account:

Figure 10.11  Email subscription request for topic subscription

17. Once you conrm your subscription, return to the Amazon SNS dashboard and
click on the Topics link from the le-hand menu.

Now that you have conrmed your subscription to the topic, you can congure an
access policy that will grant the Amazon S3 service the permissions to send notications
to the topic.

Step 2 – conguring your SNS topic policy

For Amazon S3 to send notications to the SNS topic you just created, you will need to
congure an access policy. An access policy denes who or what can access your topic
and publish messages to it. We have provided a sample policy document in the GitHub
repository for this study guide that you will need to amend [Link]
PacktPublishing/AWS-Certified-Cloud-Practitioner-Exam-Guide.
You will need to have the following information before editing the policy:

• e ARN of the SNS topic, which you made a note of earlier.

• e Amazon ARN of the S3 bucket, which you created in the previous chapter. You
can nd the bucket ARN by clicking on the Properties tab on the bucket details
page within your Amazon S3 dashboard.
 Exercise 10.1  Amazon S3 event notication using Amazon SNS 383

• e AWS account ID (which you can obtain by clicking on your account name in
the top right-hand corner of the screen and making a note of the 12-digit number
next to My Account).

Open the sample access policy document in Notepad or a text editor of your choice, as per
the following screenshot:

Figure 10.12  Sample access policy

Replace the values in the policy, as highlighted by the arrows in the preceding screenshot,
with the following:

• For Sid, change example-statement-ID to any relevant information you

would like to use; for example, source-code-change-policy.
• For Resource, change SNS-topic-ARN to the ARN of your topic, making sure
to place the ARN in double quotes.
• For ArnLike, change arn:aws:s3:*:*:bucket-name to the ARN of your
bucket name.
• For StringEquals, change bucket-owner-account-id to your AWS
account ID.
384 Application Integration Services

Save the le and keep it handy for the next step of steps:

1. Navigate back to the Amazon SNS dashboard and from the le-hand menu, click
ron Topics.
2. Click on your SNS topic in the middle pane, which will redirect you to the topic's
details page, as per the following screenshot:

Figure 10.13  SNS topic details page

3. Click on the Access policy tab in the bottom section of the pane.
4. You will nd a default access policy that allows only the topic owner to publish to
the topic.
5. In the top half of the pane, click on the Edit button.
6. Next, expand the Access policy – optional section.
7. Next, highlight and delete the existing policy that is in the JSON editor, and paste in
a copy of your amended access policy instead.
8. Finally, click the Save changes button at the bottom of the page.

Now that you have set up the SNS topic and an appropriate access policy, it is time to set
the Amazon S3 event notication service.
 Exercise 10.1  Amazon S3 event notication using Amazon SNS 385

Step 3 – setting up the Amazon S3 event notication

service
In this step, you will congure the event notication service on your Amazon S3 source
code bucket, which hosts your application repository to send out alerts every time a new
le is uploaded to the bucket:

1. Navigate to the Amazon S3 dashboard and click on the Buckets link from the
le-hand menu.
2. From the right-hand pane, click on the Amazon S3 bucket that you created in the
previous chapter to host your source code les.
3. Next, click on the Properties tab and scroll down until you reach the Event
Notications section.
4. Click the Create event notication button.
5. Enter a name for your event, such as New files added alert.
6. In the Event types section, tick the box that states All object create events.
7. Scroll further down until you reach the Destination section.
8. Select SNS topic from the Destination options.
9. Under Specify SNS topic, select the SNS topic that you created earlier in Step 1
from the SNS topic drop-down list.
10. Finally, click the Save changes button.

Now that you have congured S3 to send event notications to your SNS topic, it is time
to test the conguration.

Step 4 – testing the conguration

In this step, we will test out the conguration of our Amazon S3 event notication service:

1. In the Amazon S3 dashboard, from the le-hand menu, select Buckets.

2. From the right-hand pane, select your Amazon S3 bucket, which will contain the
source code les.
3. Next, click on the Upload button.
4. Go ahead and upload any random le you have access to. Alternatively, you can
create a text le, save it, and then upload that text le instead. You can either use
the Add les button to browse for a le on your computer or simply drag and drop
a le from another le explorer window into the upload area.
386 Application Integration Services

5. Upload your le to the Amazon S3 bucket by clicking on the Upload button at the
bottom of the page.
6. Once the upload has succeeded, click the Close button. Your object should be
visible in the list of objects in the bucket.
7. Access your email account once again and check whether you have received a
notication from AWS, alerting you to the fact that a new object has been uploaded.
Refer to the following screenshot as an example:

Figure 10.14  Amazon S3 event notication alert email

As you can see, AWS has sent me an email, alerting me to the fact that an object
was uploaded (created) in my Amazon S3 bucket. e email contains lots of
information, including the time of the event, the alert's name, the bucket in
question, the name of the object that was uploaded, as well as its size. As you can
appreciate, this can be very useful for auditing purposes.
Amazon S3 event notications can use other destinations too, such as an SQS queue or
a Lambda function. In this exercise, you learned how Amazon SNS can be used to push
out notication messages to an email address.
In the next exercise, you will perform a cleanup to remove any resources that are no
longer required from our AWS account.
 Exercise 10.2  cleaning up 387

Exercise 10.2 – cleaning up

In this exercise, you will delete the resources that you created in the previous exercise as
part of the cleanup process:

1. Navigate back to the Amazon SNS console.

2. From the le-hand menu, select Topics.
3. Next, from the right-hand pane, select the source-code-changes topic. Click the
Delete button.
4. You will be prompted to conrm the delete request with a dialog box. Type delete
me in the conrmation text box and then click the Delete button. e topic will be
deleted.

Now that you have deleted the Amazon SNS topic, you can also delete the Amazon S3
bucket as we no longer require it:

1. Navigate to the Amazon S3 console.

2. From the le-hand menu, click on Buckets.
3. From the right-hand pane, select the bucket that you uploaded your source code
to earlier.
4. You can only delete buckets if they are empty. is means that you have to delete the
objects in your bucket rst. With the bucket selected, click the Empty button.
5. Next, you will be prompted to conrm that you wish to delete the objects by typing
permanently delete in the conrmation text box. en, you can click on the
Delete button to empty the bucket.
6. Now that the bucket has been emptied, you can delete it.
7. Click the Exit button to go back to the list of buckets. With the bucket still selected,
click the Delete button. Next, in the conrmation text box, type in the name of the
bucket to conrm that you wish to delete it and click the Delete bucket button.

Your Amazon S3 bucket will be successfully deleted.

Next, we will provide a summary of this chapter and the key concepts that you learned.
388 Application Integration Services

Summary
In this chapter, we examined the key application integration services that allow you to
build highly robust and distributed application solutions. e array of services oered
by AWS for application integration enables communication between the decoupled
components of your applications, allowing you to move away from a monolithic
architecture to one that can be built using microservices. e application integration tools
available from AWS also help you design serverless solutions more easily, allowing you to
further benet from cost savings associated with server-based solutions.
e various services you learned about in this chapter included Amazon SNS, Amazon
SQS, and Amazon MQ, which are message-oriented application integration services.
ese enable communication between application components, which allows you to build
loosely coupled application architectures.
Amazon Step Functions and Amazon SWF are task-oriented application integration
services that oer workows that run for up to 1 year and can incorporate human
intervention as part of the workow process. Amazon Step Functions also helps you
coordinate application components using visual workows.
Finally, we looked at Amazon EventBridge, which is a serverless event bus service that
makes it easier to build event-driven applications. EventBridge can ingest and process
events that are generated by your applications, partner Soware-as-a-Service (SaaS)
applications, and other AWS services.
In the next chapter, we will look at a wide range of analytical services that are on oer
from AWS that allow you to stream data from a wide range of sources, perform complex
queries on ingested data, build data lakes, and build visualization dashboards and
reporting.

Questions
Answer the following questions to test your knowledge of this chapter:

1. Which AWS services does Amazon CloudWatch use to send out email alerts to
administrators when alarms are triggered and enter the Alarm state?

A. Amazon SNS
B. Amazon SES
C. Amazon CloudTrail
D. Amazon Email