Day-7 | AWS Project using Shell Scripting

Overview:

This session covers a real-time shell script project used by DevOps engineers to manage cloud
infrastructure efficiently. The project focuses on tracking cloud resource usage to ensure cost-
effectiveness.

Why Move to Cloud Infrastructure?

Organizations move to cloud platforms like AWS or Azure for two primary reasons:

 Manageability:

 Cloud infrastructure reduces the need for creating and maintaining physical data
centers. It minimizes the overhead of managing servers, upgrading them, and
handling security patches.

 Cost Efficiency:

 Cloud providers follow a pay-as-you-go model, where organizations only pay for the
resources they use. This contrasts with physical infrastructure, where costs are
incurred regardless of usage. The cloud allows companies to scale resources
efficiently, optimizing costs.

The Need for Resource Usage Tracking

To maintain cost-effectiveness, organizations must track their cloud resource usage. For example, if a
developer creates EC2 instances or EBS volumes and leaves them unused, they still incur charges.
DevOps engineers must monitor such unused resources to prevent unnecessary costs.

Methods for Tracking Cloud Resource Usage:

 Lambda Functions with Python:

 Organizations can automate resource tracking using AWS Lambda functions and
Python scripts.

 AWS SDK or CDK:

 Developers can also use the AWS Software Development Kit (SDK) or Cloud
Development Kit (CDK) to manage cloud resources programmatically.

 Shell Scripting:

 While Python or SDK-based solutions are common, they aren’t mandatory. You can
also use shell scripting to achieve the same results. Shell scripts are a simple yet
powerful tool for DevOps engineers to manage resource tracking, especially for
those comfortable with Linux commands.

Shell Scripting for Resource Management

Imagine an organization named [Link] using AWS services. The organization requires you to
deliver a daily report, say at 6 PM. This report will track how many resources (e.g., active EC2
instances, S3 buckets, Lambda functions, and IAM users) are being used. Typically, this information
would be sent to a reporting dashboard, but for this exercise, we will simulate sending it directly to
a manager.

Steps to Implement the Project:

 Write the Shell Script:

 Use AWS CLI commands to gather resource data (e.g., EC2, S3, Lambda, and IAM
information).

Why AWS CLI? AWS CLI interacts with AWS services directly from your terminal. You can also
use Boto3 (AWS SDK for Python) or other tools, but since we have already learnt shell
scripting and AWS CLI, we’ll combine our knowledge of shell scripting with AWS CLI commands to
get the necessary outputs.

 Store the data in a file that will act as your daily report.

 Schedule the Script Using Cron Job:

 Set up a Cron job that runs the shell script daily at 6 PM.

 Testing and Verification:

 Test the shell script by running it manually and verifying that the resource data is
captured correctly.

 Test the Cron job to ensure the script executes at the scheduled time automatically.

Prerequisites:

Before we dive into scripting, ensure the following prerequisites are in place:

 AWS CLI Installation:

 First, connect to your EC2 instance and check that AWS CLI is installed on your Linux
box.

 Verify installation by running the command: aws --version

 AWS CLI Configuration:

 Configure the AWS CLI with your credentials so that it can interact with your AWS
account. Run: aws configure

 You’ll need to provide:

 AWS Access Key

 AWS Secret Access Key

 Default Region

 Output format (e.g., JSON)

These credentials allow the AWS CLI to authenticate and interact with your AWS account.

Step 1: Writing the shell script

a) Create a shell script file

b) Start writing the commands inside it.

Output:
When running the command aws ec2 describe-instances, the output provides the complete JSON
detailing all the EC2 instances. However, if the goal is to simply count the number of instances, it's
more efficient to extract and display only the Instance IDs.

c) Updating the shell script to display only the Instance IDs of the EC2 instances
Output:
jq stands for "JSON Query". It is a command-line tool that processes JSON data. We can use jq to
filter, extract, and manipulate JSON output in a more human-readable way. Here's a simple example
of using jq to extract the names of EC2 instances:

This command lists only the InstanceId of all EC2 instances.

d) Storing the output of the shell script in a text file

To redirect the output of the script to a file, you can use the redirection operator (> for overwriting
or >> for appending) at the end of each command. Here's how you can modify your script to redirect
the output to a file called aws_resource_usage.txt:

 > overwrites the file each time the command is executed.

 >> appends the output to the file instead of overwriting it.

e) Share the Output File

To share the output file with your manager after the script runs, you can upload the file to an S3
bucket and share the link. Add the following command at the end of your script:

Step 2: Schedule the script using Cron job

 Open the crontab configuration: Run the following command in your terminal:

 Add the cron job: Add the following line at the end of the crontab file:

Here’s what each part means:

 0 - Minute (0)

 18 - Hour (18 means 6 PM)

 * * * - Every day, every month, every day of the week

 /path/to/your/[Link] - The full path to your script

 Save and exit the Crontab

After adding the cron job, save the changes and exit the editor. The cron service will automatically
pick up the changes, and your script will run every day at 6 PM.