AWS for Cloud Computing
Module 3: AWS Database Services
• AWS Lambda
• Amazon Dynamo DB
• Amazon ECS (Elastic Container Service) & Amazon S3 Glacier
• Amazon Kinesis, Amazon Redshift
• Amazon EMR (Elastic MapReduce), AWS Disaster Recovery and
Backup.
Amazon Kinesis
• Kinesis is a platform on AWS that sends your
streaming data.
• Amazon Kinesis Service that allows users to process
a large amount of data (which can be audio, video,
application logs, website clickstreams, and IoT
telemetry ) per second in real time.
Amazon Kinesis
What is streaming data?
Streaming data is data which is generated continuously from thousands of
data sources, and these data sources can send the data records
simultaneously and in small size.
Stock prices
You tube, Facebook, OTT
IoT Sensor data
[Link]
Core Services of Kinesis
• Kinesis Streams
• Kinesis Firehose
• Kinesis Analytics
Kinesis Streams
• Kinesis streams consist of shards.
• Shards provide 5 transactions per second for reads, up
to a maximum total data read rate of 2MB per second
and up to 1,000 records per second for writes up to a
maximum total data write rate of 1MB per second.
• Suppose we have got the EC2, mobile phones, Laptops,
IOT which are producing the data. They are known as
producers as they produce the data.
Kinesis Streams
Kinesis Streams
• The data is moved to the Kinesis streams and stored in the
shard. By default, the data is stored in shards for 24 hours.
You can increase the time to 7 days of retention.
• Once the data is stored in shards, then you have EC2
instances which are known as consumers.
• They take the data from shards and turned it into useful
data.
• Once the consumers have performed its calculation, then
the useful data is moved to either of the AWS services, i.e.,
DynamoDB, S3, EMR, Redshift.
Kinesis Firehouse
• Kinesis Firehose is a service used for delivering streaming data
to destinations such as Amazon S3, Amazon Redshift, Amazon
Elasticsearch.
• With Kinesis Firehouse, you do not have to manage the
resources.
• EC2, mobile phones, Laptop, IOT which are producing the
data. They are also known as producers.
• Producers send the data to Kinesis Firehose.
Kinesis Firehouse
Kinesis Firehouse
• Kinesis Firehose does not have to manage the resources such as shards, you do
not have to worry about streams, you do not have to worry about manual
editing the shards to keep up with the data, etc.
• It’s completely automated.
• You do not have to worry even about the consumers.
• Data can be analyzed by using a Lambda function.
• Once the data has been analyzed, the data is sent directly over to the S3.
• One important thing about Kinesis Firehouse is that there is no automatic
retention window, but the Kinesis Firehouse only sends the data over directly to
S3 or other location.
Kinesis Analytics
• Kinesis Analytics is a service of Kinesis in
which streaming data is processed and
analyzed using standard SQL.
• This is very similar to Firehouse.
• But, Kinesis Analytics allows you to run the
SQL Queries of that data which exist within
the kinesis firehose.
• You can use the SQL Queries to store the
data in S3, Redshift or Elasticsearch cluster.
• Essentially, data is analyzed inside the
kinesis using SQL type query language.
Applications
• Real-time application monitoring: Amazon kinesis will provide the real
time data of the applications like if you consider the health application it
will provides the live feed of the data by which you can take care of the
health by which the issues that is pointed by amazon kinesis.
• Fraud detection and prevention: Amazon Kinesis will helps you to
protect the data from fraudulent activity by analyzing transaction data by
which you can detect the suspicious patterns and blocks fraudulent
transactions before they happen.
• Personalized recommendations and marketing: Amazon kinesis will
helps you in analyzing the data of the customers by which you can
understand your customers very better. You can recommends the
personalised products in real time to the costumers.
• IoT analytics and predictive maintenance: Your connected gadgets’ full
potential is unlocked with Kinesis. Through the examination of sensor
data from electronics, automobiles, or machinery.
Amazon Redshift
• AWS Redshift is big data analytics service.
• Redshift is a fast and powerful, fully managed, petabyte-scale data
warehouse service in the cloud.
Amazon Redshift
• Redshift warehouse is a collection of computing resources
known as nodes, and these nodes are organized in a group
known as a cluster.
• Each cluster runs in a Redshift Engine which contains one or
more databases.
Redshift Configuration
• Redshift consists of two types of nodes:
• Single node
• Multi-node
Single node: A single node stores up to 160 GB.
• Multi-node: Multi-node is a node that consists of more than one
node. It is of two types:
• Leader Node -It manages the client connections and receives
queries.
A leader node receives the queries from the client applications,
parses the queries, and develops the execution plans.
It coordinates with the parallel execution of these plans with the
compute node and combines the intermediate results of all the
nodes, and then return the final result to the client application.
• Compute Node -A compute node executes the execution plans.
Then intermediate results are sent to the leader node for
aggregation before sending back to the client application. It can
have up to 128 compute nodes.
• Redshift warehouse is a collection of computing resources known as
nodes, and these nodes are organized in a group known as a cluster.
• Each cluster runs in a Redshift Engine which contains one or more
databases.
• Easy to setup, deploy and manage
Redshift features Automated Provisioning, and Backup
• Cost-effective
No upfront costs, pay as you go
Scale quickly to meet your needs
Amazon Redshift automatically scales up
or down the nodes according to the need
changes.
Faster performance
Secure
You can also enable encryption, all the
data written to disk will be encrypted.