AWS Application Integration Overview
AWS Application Integration Overview
Dr. [Link]
Assistant Professor Senior Grade II
School of Computer Science and Engineering
Vellore Institute of Technology, Vellore
BCSE355L -AWS Solutions Architect
Module 7
2
AWS Application Integration
• AWS Application Integration is a group of managed cloud services offered by Amazon Web Services that
allow developers to connect, coordinate, and integrate multiple applications and microservices.
• These services remove the need for complex custom integration code and make communication between
distributed systems much easier.
• Application integration is the process of getting independently built software systems to work together
without manual intervention.
• Modern application design encourages the flexible exchange of data between applications for increased
efficiency, modularity, and reusability.
• Application integration allows your developers to build applications that reuse existing services and
systems. This way, they can do more with less coding.
• It also facilitates automation, as applications can communicate with each other for complex enterprise
workflows.
AWS Application Integration
AWS Application Integration
Amazon SQS (Simple Queue Service)
• Simple Queue Service is a distributed messaging queue that helps in solving
producer-consumer connectivity for distributed systems.
• SQS enables your services to send and store messages to a queue at any volume
without losing messages even if the consumer is down.
• Messaging queue is a technique of sending context about a process between
services in a web application.
• Amazon Simple Queue Service (SQS) is a fully managed message queuing
service that enables you to decouple and scale microservices, distributed systems,
and serverless applications.
Amazon SQS (Simple Queue Service)
One service, say Service A , is producing messages and another service
,say Service B , is consuming messages.
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
How SQS Works:
1.A producer (sender) creates a message and sends it to an SQS queue.
[Link] message is stored in the queue until it’s delivered to a consumer (receiver).
[Link] a consumer retrieves the message from the queue, it becomes invisible to other
consumers for a user-defined period of time (the visibility timeout). This prevents other
consumers from processing the same message and ensures that the message is processed
at least once.
[Link] the consumer processes the message, it sends a delete request to SQS to remove
the message from the queue.
Amazon SQS (Simple Queue Service)
Types of SQS queues
1. Standard Queue:
• Messages in the queue are not ordered. This means messages sent in one order might not
be received in the same order.
• Support an unlimited number of messages/transactions per second (TPS).
• Messages might get duplicated to ensure At-least once delivery.
Amazon SQS (Simple Queue Service)
Types of SQS queues
[Link] (First In First Out)
• FIFO is used for scenarios where either or both the order of event and duplicate message can not
be tolerated.
• Messages in the queue maintain the Order in which the message is being pushed to the queue.
• This can support a max of 30,000 messages with batch and 3,000 without batching.
Messages are never duplicated in this queue.
Amazon SQS (Simple Queue Service)
Features of SQS:
[Link] least Once Delivery: This means a message once sent into the queue will be consumed
at least once. In a standard queue, there can be duplicates and in FIFO it will be consumed
exactly once no duplicates are allowed.
[Link]: When a consumer fails it does not impact the queue. Since it is decoupled all
the messages that are in the queue will remain and can be processed once the service is up
again.
[Link] Locking: When a message is being consumed it becomes unavailable to other
consumers. If the message fails the message will again become visible. This is part of
the visibility timeout.
Amazon SQS (Simple Queue Service)
Features of SQS:
4. Unlimited messages: There are no limits on the number of messages you can send in
the queue. So in case, there is a spike in requests the queue will handle the spike just
fine.
5. Batches: We Can send 10 messages at a time with no extra cost.
6. Secure: Secured by AWS IAM (identity and access management) and uses HTTPS and
TLS.
7. reliability: Queue is stored in a single AWS Region with multiple redundant
Availability zones (AZ). So, if a single region is affected it does not make messages
unavailable.
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
Long Polling
• The consumer waits up to 20 seconds for a message before the request returns, reducing empty
polling.
• It lowers cost and improves efficiency by delivering messages as soon as they arrive.
Visibility Timeout
• Once a consumer receives a message, it becomes hidden from others during processing.
• The timeout must balance avoiding duplicate processing and minimizing delays if a consumer
fails.
Dead Letter Queue (DLQ)
• A DLQ stores messages that repeatedly fail processing in the main queue.
• It enables troubleshooting and prevents problematic messages from blocking the system.
Amazon SQS (Simple Queue Service)
s
Amazon SQS (Simple Queue Service)
Basic Amazon SQS architecture
s
Amazon SQS (Simple Queue Service)
Basic Amazon SQS architecture
Distributed queues
• There are three main parts in a distributed messaging system: the components of your
distributed system, your queue (distributed on Amazon SQS servers), and
the messages in the queue.
• In the following scenario, your system has several producers (components that send
messages to the queue) and consumers (components that receive messages from the
queue). The queue (which holds messages A through E) redundantly stores the
messages across multiple Amazon SQS servers.
Amazon SQS (Simple Queue Service)
Message lifecycle
Amazon SQS (Simple Queue Service)
Amazon SQS (Simple Queue Service)
• Component 1 sends Message A to the SQS queue, making it available for consumption.
• Component 2 retrieves Message A and the visibility timeout period begins, hiding the
• Component 2 processes the message and deletes it before the visibility timeout expires,
• The SNS topic replicates the message and sends it to all subscribers
• HTTP/S EC2 Endpoints – messages delivered via HTTP requests to your servers
Amazon SNS (Simple Notification Service)
Amazon SNS (Simple Notification Service)
Amazon SNS (Simple Notification Service)
Amazon SNS (Simple Notification Service)
Amazon SNS (Simple Notification Service)
Domains:
A domain is a container for a set of related workflows and their associated actors, tasks, and deciders. It
provides a way to isolate and manage different sets of workflows within a given AWS account.
Deciders and Workers:
•Deciders: Deciders are responsible for making decisions about what activities to execute next in the
workflow. They evaluate the state of the workflow and determine the appropriate next steps.
•Workers: Workers are responsible for executing the activity tasks in the workflow. They perform the actual
work defined by the activities.
Amazon SWS (Simple Workflow Service)
How Amazon SWF works
Amazon SWS (Simple Workflow Service)
How Amazon SWF works
[Link] Web Server Front End accepts a video URL from a user and starts the workflow execution in Amazon SWF.
[Link] SWF creates a workflow that consists of multiple tasks such as:
1. Encode video
2. Add DRM protection (Digital Rights Management)
3. Decision tasks to control flow
3.A Decider service monitors the workflow state and makes decisions on:
1. What task to run next
2. When to run it
3. In what order tasks should be executed
[Link] Decider uses long polling to get tasks from SWF and returns decisions after evaluating the workflow history.
[Link] dispatches tasks to appropriate workers. Each type of work is handled by a separate worker:
1. Encoding worker handles Encode Tasks
2. DRM worker handles Adding DRM Tasks
[Link] also use long polling to get pending tasks from SWF.
[Link] processing the task, each worker returns the results back to SWF.
[Link] stores Execution History, which keeps a log of:
1. Input data (video URL)
2. Each task completion status
3. Overall execution completion
[Link] all tasks are completed successfully, the workflow finishes.
Amazon SWS (Simple Workflow Service)
What Does Simple Workflow Do?
Amazon SWS (Simple Workflow Service)
What Does Simple Workflow Do?
Simple Workflow provides you with the infrastructure that you need to implement
workflows such as the one above. It does all of the following (and a lot more):
•Stores metadata about a Workflow and its component parts.
•Stores tasks for Workers and queues them until a Worker needs them.
•Assigns tasks to Workers.
•Routes information between executions of a Workflow and the associated Workers.
•Tracks the progress of Workers on Tasks, with configurable timeouts.
•Maintains workflow state in a durable fashion.
Amazon SWS (Simple Workflow Service)
Features of AWS SWF
Amazon SWS (Simple Workflow Service)
Features of AWS SWF
•Workflow Management: AWS SWF allows for defining, executing, and managing complex workflows.
You can design workflows that include multiple steps, with the ability to manage the execution order and
dependencies between tasks.
•Task Coordination and Execution Control: It provides robust task coordination, ensuring that tasks are
executed in the right order, without overlap, and according to the workflow logic. You can control various
aspects such as task assignment, prioritization, timeouts, and retries.
•Scalability and High Throughput: AWS SWF is designed to be highly scalable, supporting a large
number of workflow executions and tasks. This makes it suitable for applications that need to handle high
volumes of workflow tasks efficiently.
•Reliability and Fault Tolerance: SWF guarantees that tasks are tracked and executed at least once. If a
task fails or times out, the service can automatically retry it based on the defined policies, ensuring the
reliability of the workflow process.
•Integration with AWS Ecosystem: It integrates seamlessly with other AWS services, allowing you to
leverage the capabilities of services like AWS Lambda, Amazon EC2, Amazon S3, and Amazon RDS
within your workflows.
•Monitoring and Logging: SWF provides integration with Amazon CloudWatch for monitoring workflow
and task metrics. This allows for setting alarms and notifications based on workflow performance.
Amazon SWS (Simple Workflow Service)
Best Practices for Using AWS SWF
•Clearly Define Workflow Logic: Ensure that your workflow logic is well-defined and modular.
Break down complex processes into smaller, manageable tasks. This makes it easier to understand,
update, and debug your workflows.
•Handle Failures Gracefully: Design your workflow to handle failures and exceptions effectively.
Implement retry logic and timeouts for activities to manage transient issues. Use Dead Letter Queues
(DLQs) for activities that fail repeatedly to analyze and address the root cause.
•Optimize Task Assignment: Utilize SWF’s task lists effectively to distribute tasks across your
worker processes. This ensures load balancing and efficient processing of tasks.
•Secure Your Workflows: Implement appropriate IAM (Identity and Access Management) policies to
control access to your SWF resources. Ensure that only authorized entities can start workflows,
complete tasks, or access workflow data.
•Test Thoroughly: Test your workflows extensively under various scenarios, including failures and
edge cases. Automated tests and staging environments are recommended for ensuring the reliability of
your workflows
AWS Step Functions
• AWS Step Functions allow you to coordinate multiple AWS services into serverless
workflows.
• A workflow is represented as a state machine, defined using the Amazon States Language
(a JSON-based declarative language).
• Step Functions handle the execution of these workflows, ensuring that tasks are
performed in the correct sequence with built-in fault tolerance and retry mechanisms.
• Its integration capabilities are extensive, supporting connections with Amazon EC2,
Amazon ECS, on-premise servers, Amazon API Gateway, and Amazon SQS queues, to
name a few.
• This ensures that workflows can be comprehensive and interact with a broad ecosystem
of AWS services.
AWS Step Functions
• With AWS Step Functions, you can create workflows, also called State machines, to
build distributed applications, automate processes, orchestrate microservices, and create
data and machine learning pipelines.
• Step Functions is based on state machines and tasks. In Step Functions, state machines
are called workflows, which are a series of event-driven steps. Each step in a workflow is
called a state. For example, a Task state represents a unit of work that another AWS
service performs, such as calling another AWS service or API. Instances of running
workflows performing tasks are called executions in Step Functions.
• The work in your state machine tasks can also be done using Activities which are
workers that exist outside of Step Functions
AWS Step Functions
Key characteristics include:
Serverless Nature: No need to provision or manage servers.
State Management: Tracks the state of each workflow step, making it easy to
handle branching logic and retries.
Integration: Works seamlessly with AWS services like Lambda, S3,
DynamoDB, API Gateway, SageMaker, and more.
Visual Interface: Provides a graphical view of your workflows, making it easy
to debug and monitor.
AWS Step Functions
Types of Step Functions:
Standard Workflows:
• Best for long-running and durable workflows.
• Stores detailed execution history.
• Supports retries and error handling over extended periods (up to one year).
• Use Case: Loan processing, data pipeline orchestration, or multi-step transaction
workflows.
Express Workflows:
• Designed for high-volume and short-duration tasks.
• Optimized for performance and cost but provides less detailed execution history.
• Use Case: Event-driven workflows like processing IoT sensor data or streaming logs.
AWS Step Functions
Core Components of AWS Step Functions
State Machine: A state machine is a definition of the workflow. It consists of states,
transitions, and rules for execution. Common state types include:
Flow States (Flow states control the flow of execution of the steps)
• Choice State: Adds conditional branching logic.
• Wait State: Delays the workflow for a specified time.
• Parallel State: Executes multiple branches concurrently.
• Map state: runs child workflows for each item in a dataset
Task State: Performs a unit of work (e.g., invoking a Lambda function).
Fail/Success State: Defines workflow completion
AWS Step Functions
Core Components of AWS Step Functions
AWS Step Functions
Steps to Create a State Machine
[Link] in to your AWS account
2. In the search box type Step Functions and click on it.
AWS Step Functions
Steps to Create a State Machine
3. Now click on State Machines
AWS Step Functions
Steps to Create a State Machine
4. click on Create State Machine
AWS Step Functions
Steps to Create a State Machine
5. Under Define state machine selects Author with code snippets, and under Type
select Standard
AWS Step Functions
Steps to Create a State Machine
6. Now insert the JSON code, on the basis of which your state machine (workflow) will be created
AWS Step Functions
Steps to Create a State Machine
7. Now you will see based on the JSON code a State diagram has been created on the right side
AWS Step Functions
Steps to Create a State Machine
8. Now click on Next
AWS Step Functions
Steps to Create a State Machine
9. Under Specify details provide any name to your state machine, and under Permissions select Create a role,
by this AWS will automatically create a role for the permissions required by State Machines for you
AWS Step Functions
Steps to Create a State Machine
10. Let everything default, scroll down and click on Create State machine
AWS Step Functions
Steps to Create a State Machine
11. Now you shall see a message prompting that the State machine successfully created
AWS Step Functions
Example: Determine if the returned number is greater than , less than, or equal to a
specified number.
{ "GreateThan": {
"Comment": "A simple Step Functions state machine", "Type" : "Task",
"StartAt": "AddNumbers", "Resource": "ARN:OF:greater-than-lambda",
"States": { "Next": "FinalState"
"AddNumbers": {
},
"Type": "Task",
"LessThan": {
"Resource": "ARN:OF:add-numbers",
"Next": "AnswerState" "Type" : "Task",
}, "Resource": "ARN:OF:less-than-lambda",
"AnswerState": { "Next": "FinalState"
"Type" : "Choice", },
"Choices": [ "EqualTo": {
{ "Type": "Fail",
"Variable": "$.number", "Cause": "No Matches!"
"NumericGreaterThan": 500, },
"Next": "GreateThan" "FinalState": {
}, "Type": "Task",
{ "Resource": "ARN:OF:final-state-lambda",
"Variable": "$.number", "End": true
"NumericLessThan": 500, }
"Next": "LessThan"
}
}
}
],
"Default": "EqualTo"
AWS Step Functions
Common Use Cases for AWS Step Functions
Microservices Orchestration: Coordinate independent services to complete a larger task.
For example, in e-commerce, validate orders, charge payments, and notify customers using
different microservices.
Data Processing Pipelines: Automate workflows for processing large datasets. For
instance, retrieve data from S3, clean and transform it using Lambda, and store results in a
database.
Machine Learning Workflows: Automate ML model training, testing, and deployment
using SageMaker integrated with Step Functions.
ETL (Extract, Transform, Load): Automate ETL tasks like extracting data from multiple
sources, transforming it, and loading it into a data warehouse.
DevOps Automation: Automate CI/CD pipelines, including running tests, building
applications, and deploying them to environments.
Batch Job Processing: Orchestrate jobs that require multiple steps, such as video
transcoding or report generation.
AWS Well-Architected Framework
• The AWS Well-Architected Framework is a set of best practices and guidelines to
help you design, build, and operate reliable, secure, efficient, and cost-effective systems
in the cloud.
AWS Well-Architected Framework
• The AWS Well-Architected Framework is a set of best practices and guidelines to
help you design, build, and operate reliable, secure, efficient, and cost-effective systems
in the cloud.
AWS Well-Architected Framework
AWS Well-Architected Framework
AWS Well-Architected Framework
[Link] Excellence Pillar
The operational excellence pillar is a capacity to manage and monitor systems.
It improves supporting systems processes and procedures.
It includes:
➢ Making small and reversible changes
➢ Prediction of system disruptions
➢ Performing code tasks
➢ Making documentation notes
AWS Well-Architected Framework
[Link] Pillar
➢ Cost optimization pillar helps you run your cloud services at the
lowest price points.
application that requires a lot of resources at peak times and less during off-peak hours.
• Use a scalable, load-balanced cluster. This approach allows for the distribution of
• Enable detailed monitoring. Thorough monitoring allows for the collection of CloudWatch
metric data at a one-minute frequency, which can help to ensure a faster response to load
changes.
AWS cloud elasticity
• Design for horizontal scaling: One of the most significant advantages of cloud
computing is the ability to scale your application using a distributed architecture that can
be easily replicated across multiple instances.
• Use Elastic Load Balancing: ELB can automatically detect unhealthy instances and
redirect traffic to healthy ones. It distributes incoming traffic across multiple instances of
your application, helping to ensure that no single model becomes overloaded.
• AWS CloudWatch allows you to monitor the performance of your application and the
resources it uses. You can set up alarms to trigger Auto Scaling actions based on metrics
such as CPU utilization, network traffic, or custom metrics.
High Availability and Fault Tolerance
• High availability means a system will almost always maintain uptime, albeit sometimes
in a degraded state. About AWS, a system has high availability when it has 99.999%
uptime, also known as "five nines." To put that in perspective, the system would be down
for a mere five minutes and fifteen seconds a year.