Module:7
AWS Application Integration &
Architectural Best Practices
Application integration
• Application integration on AWS is a suite of services that enable communication
between decoupled components within microservices, distributed systems, and
serverless applications.
• You don’t need to refactor your entire architecture to benefit—decoupling
applications at any scale can reduce the impact of changes, making it easier to
update and faster to release new features.
Amazon SQS (Simple Queue Service)
• Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted
queue that lets you integrate and decouple distributed software systems and components
• An e-commerce website where customers can place orders. When a customer places an
order, you need to process it (e.g., verify payment, check inventory, package it, and arrange
shipping). Instead of handling all these steps instantly, you can use SQS to manage them
efficiently.
Amazon SQS (Simple Queue Service)
[Link] Placement: When a customer places an order, the order information (like
items, customer ID, payment status) is sent as a message to an SQS queue.
[Link] the Order: Several microservices (or worker applications) can pull the
message from the SQS queue to process different steps:
1. Service 1: Confirms payment.
2. Service 2: Checks inventory.
3. Service 3: Packages the order.
4. Service 4: Arranges shipping.
• Each service reads from the queue when it's ready, processes its task, and moves
to the next one, without all steps having to happen immediately or
simultaneously. This approach ensures scalability since you can add more services
to process the queue faster if there’s high demand.
Basic Amazon SQS architecture
The following diagram outlines the parts of a distributed messaging system and explains the lifecycle of an Amazon SQS
message.
Distributed queues
• There are three main parts in a distributed messaging system: the components of your distributed system, your
queue (distributed on Amazon SQS servers), and the messages in the queue.
• In the following scenario, your system has several producers (components that send messages to the queue) and
consumers (components that receive messages from the queue). The queue (which holds messages A through E)
redundantly stores the messages across multiple Amazon SQS servers.
Message lifecycle • Section one description for the previous lifecycle
diagram. A producer (component 1) sends
message A to a queue, and the message is
distributed across the Amazon SQS servers
redundantly.
• Section two description for the previous lifecycle
diagram. When a consumer (component 2) is
ready to process messages, it consumes
messages from the queue, and message A is
returned. While message A is being processed, it
remains in the queue and isn't returned to
subsequent receive requests for the duration of
the visibility timeout.
• Section three description for the previous
lifecycle diagram. The consumer (component 2)
deletes message A from the queue to prevent
the message from being received and processed
again when the visibility timeout expires.
Amazon SNS (Simple Notification Service)
• Amazon Simple Notification Service (Amazon SNS) is a managed service
that provides message delivery from publishers to subscribers (also known
as producers and consumers).
• Publishers communicate asynchronously with subscribers by sending
messages to a topic, which is a logical access point and communication
channel.
• Clients can subscribe to the Amazon SNS topic and receive published
messages using a supported endpoint type, such as Amazon Data Firehose,
Amazon SQS, AWS Lambda, HTTP, email, mobile push notifications, and
mobile text messages (SMS).
Amazon SNS (Simple Notification Service)
Differences between Amazon SQS and Amazon SNS
• Amazon SQS, Amazon SNS offer highly scalable and easy-to-use managed messaging
services, each designed for specific roles within distributed systems. Here's an enhanced
overview of the differences between these services:
• Amazon SQS decouples and scales distributed software systems and components as a
queue service. It processes messages through a single subscriber typically, ideal for
workflows where order and loss prevention are critical. For wider distribution, integrating
Amazon SQS with Amazon SNS enables a fanout messaging pattern, effectively pushing
messages to multiple subscribers at once.
• Amazon SNS allows publishers to send messages to multiple subscribers through topics,
which serve as communication channels. Subscribers receive published messages using a
supported endpoint type, such as Amazon Data Firehose, Amazon SQS, Lambda, HTTP,
email, mobile push notifications, and mobile text messages (SMS). This service is ideal for
scenarios requiring immediate notifications, such as real-time user engagement or alarm
systems. To prevent message loss when subscribers are offline, integrating Amazon SNS
with Amazon SQS queue messages ensures consistent delivery.
Amazon Simple Workflow Service
• The Amazon Simple Workflow Service (Amazon SWF) provides a way to build, run, and scale
background jobs that have parallel or sequential steps.
• With Amazon SWF, you can coordinate work across distributed components, tracking the state
of tasks.
• In Amazon SWF, a task represents a logical unit of work that is performed by a component of
your application. Coordinating tasks across the application involves managing intertask
dependencies, scheduling, and concurrency in the logical flow of your application.
• Amazon SWF gives you control over implementing tasks and coordinating them without
worrying about underlying complexities such as tracking their progress and maintaining their
state.
• When using Amazon SWF, you implement workers to perform tasks. Workers can run either on
cloud infrastructure, such as Amazon Elastic Compute Cloud (Amazon EC2), or on your own
premises.
• You can create tasks that are long-running, or that may fail, time out, or require restarts—or
that may complete with varying throughput and latency.
• Amazon SWF stores tasks and assigns them to workers when they are ready, tracks their
progress, and maintains their state, including details on their completion.
AWS Step Functions
• With AWS Step Functions, you can create workflows, also called State machines,
to build distributed applications, automate processes, orchestrate microservices,
and create data and machine learning pipelines.
• Step Functions is based on state machines and tasks. In Step Functions, state
machines are called workflows, which are a series of event-driven steps.
• Each step in a workflow is called a state. For example, a Task state represents a
unit of work that another AWS service performs, such as calling another AWS
service or API. Instances of running workflows performing tasks are called
executions in Step Functions.
• The work in your state machine tasks can also be done using Activities which are
workers that exist outside of Step Functions.
• In the Step Functions' console,
Step Functions you can visualize, edit, and
debug your application’s
workflow.
• You can examine the state of
each step in your workflow to
make sure that your application
runs in order and as expected.
• Depending on your use case,
you can have Step Functions call
AWS services, such as Lambda,
to perform tasks.
• You can have Step Functions
control AWS services, such as
AWS Glue, to create extract,
transform, and load workflows.
You also can create long-
running, automated workflows
for applications that require
human interaction.
AWS Well-Architected Framework,
• Designing a cloud architecture for a growing e-commerce platform on AWS requires implementing the AWS
Well-Architected Framework’s six pillars to ensure scalability, security, and efficiency while controlling costs.
• Here's how each of the six pillars can be applied:
• Operational Excellence
• Security
• Reliability
• Performance Efficiency
• Cost Optimization
• Sustainability
CONT
Operational Excellence
This pillar focuses on continuously improving and automating operations to
deliver business value effectively. For an e-commerce platform:
• Automation of processes
• Monitoring and Logging
• Operational Runbooks
• Infrastructure as Code
Security
Protecting customer data and meeting security compliance requirements is
critical for e-commerce platforms.
• Data Encryption
• Identity and Access Management
• Secure Network Architecture
• DDoS Protection
CONT
Reliability
Ensuring high availability, fault tolerance, and disaster recovery is
vital for handling unpredictable traffic spikes and maintaining
operations during peak seasons.
• Auto Scaling
• Multi-AZ and Multi-Region Architectures
• Backup and Recovery
• Health Checks and Fault Tolerance
Performance Efficiency
This pillar focuses on using computing resources efficiently to meet
system and business requirements, especially during traffic spikes.
• Right Sizing of Resources
• Caching
• Database Performance
• Auto Scaling and Load Balancing
CONT
Cost Optimization
Minimizing costs while maintaining the necessary resources to handle unpredictable traffic is essential.
• Reserved Instances and Savings Plans
• Serverless Architectures
• Data Tiering and Archiving
• Cost Monitoring and Alerts
Sustainability
• The sustainability pillar focuses on minimizing the environmental impacts of
running cloud workloads. Key topics include a shared responsibility model for
sustainability, understanding impact, and maximizing utilization to minimize
required resources and reduce downstream impacts.
By following the AWS Well-Architected Framework’s six pillars, the e-commerce platform will be able to handle unpredictable
traffic spikes with automated scaling, protect sensitive customer data through robust security practices, maintain high availability
during peak shopping seasons with resilient architectures, and optimize both performance and cost by leveraging the right mix of
AWS services and best practices. This ensures a sustainable, secure, and cost-efficient platform as it grows.
Design Principles for AWS Cloud Architectures
Scalability
• Horizontal Scaling: Use load balancers (like Elastic Load Balancer) to distribute workloads across
multiple instances or services, enabling your application to handle more traffic.
• Auto Scaling: Use Amazon EC2 Auto Scaling to automatically adjust the number of instances
based on demand, ensuring optimal resource usage without manual intervention.
Elasticity
• Dynamic Resources Allocation: Architect applications to scale resources up or down as demand
changes. Use serverless options like AWS Lambda, DynamoDB, and S3, which automatically scale
based on workload.
• Event-driven Architecture: Implement event-driven services to reduce the need for constantly
running servers, using services like Amazon SQS, Amazon SNS, and Amazon EventBridge.
High Availability & Fault Tolerance
• Multi-AZ and Multi-Region Deployments: Distribute applications across multiple Availability
Zones (AZs) or Regions to ensure availability and fault tolerance in case of outages.
• Backup & Disaster Recovery: Use AWS Backup and other services to implement disaster
recovery. Store backups in different regions and follow the 3-2-1 backup rule.
CONT
Security
• Shared Responsibility Model: AWS and the user share responsibility. AWS secures infrastructure, while users manage
application-level security, including IAM policies, network security, and data encryption.
• Identity and Access Management (IAM): Implement least privilege access using AWS IAM, enabling fine-grained
permissions and multi-factor authentication.
• Encryption and Data Protection: Use encryption for data at rest and in transit, employing AWS Key Management Service
(KMS) for managing encryption keys securely.
Cost Optimization
• Right-sizing Resources: Regularly evaluate resource usage and use tools like AWS Cost Explorer to find opportunities for
cost savings. Choose the appropriate instance types, and consider spot instances for non-critical workloads.
• Reserved Instances and Savings Plans: Use Reserved Instances for predictable workloads and Savings Plans for long-term
commitments to reduce costs.
• Serverless Architecture: Use serverless services (e.g., Lambda, S3) that scale automatically and incur costs based on
actual usage.
CONT
Performance Efficiency
• Managed Services: Use managed services like RDS (Relational DB Services), ElastiCache, and DynamoDB to reduce the
operational burden and optimize performance without needing to configure and manage infrastructure manually.
• Caching: Utilize caching layers such as Amazon CloudFront (for content delivery) and Amazon ElastiCache (for in-memory
caching) to improve response times.
• Monitor and Optimize: Use Amazon CloudWatch for monitoring and AWS Trusted Advisor for ongoing performance
recommendations.
Operational Excellence
• Automation: Automate infrastructure management tasks with AWS CloudFormation, AWS OpsWorks, and infrastructure
as code (IaC) practices.
• Monitoring and Logging: Implement comprehensive logging with AWS CloudWatch, AWS CloudTrail, and other logging
services to gain insight into application performance and security events.
• Disaster Recovery and Testing: Regularly test disaster recovery processes and perform simulated failures to ensure the
architecture can recover from unexpected events.
CONT
Reliability
• Automatic Recovery: Design for automatic recovery from failures, using AWS services that support resilience and
durability, such as Amazon RDS with Multi-AZ deployment.
• Health Checks and Self-Healing: Set up health checks for all services, and create self-healing mechanisms, like auto-
scaling with health checks, to replace failing components.
Data Management
• Data Replication and Backup: Implement regular backups, data replication (e.g., cross-region replication in S3), and data
lifecycle policies to ensure data durability.
• Data Lake and Analytics: Use AWS Glue, Amazon Athena, and Amazon Redshift for efficient data processing and analysis,
allowing seamless integration with storage services like S3.