Day 6b - Shell Scripting for DevOps (Part-2)
Automating Node Health Analysis
In this session, we'll focus on creating a custom node script that can detect the health of a virtual
machine (VM).
Use Case: Imagine a scenario where someone reports an issue with a virtual machine, and they need
help to diagnose the problem. Instead of manually running several commands, we’ll create a script
that does this in one go.
Main Goal: The goal is to automate the health check process using a shell script that outputs vital
system information, like disk space, memory, and CPU usage, which helps troubleshoot performance
issues on a VM.
Writing the Node Health Shell Script
1. Using Shebang to specify the script’s executable (Bash).
2. Adding Metadata like author, date, and script purpose to clarify the script’s intent.
3. Running Commands to check disk space, memory in GB, and no. of CPUs using df -h, free -g,
and nproc.
4. Improving Readability: There are two ways to improve the readability of the output that
we’ll get after executing the [Link] file.
To make the output clearer, we use echo statements that describe what each
command does:
Copy
Copy
echo "Disk Space:"
df -h
echo "Memory Status:"
free -g
echo "CPU Info:"
nproc
This approach improves readability but may not be scalable for larger scripts.
Debugging with set -x, which shows each command and its output, making the
script more understandable without excessive echo statements.
5. Here is the example:
6. Execute the file and have a look at the output. Before executing the file, grant permissions.
Shell Command to List and Filter the Processes
When working on a virtual machine (VM), one of the key tasks is monitoring and managing
processes. Processes represent all the tasks and programs currently running on your system, such as
applications or system services.
Listing All Processes: ps -ef
On a Linux-based system, which doesn’t have a graphical user interface (GUI) for task management,
you can list all running processes using the ps -ef command. Here’s a breakdown:
ps: Displays information about active processes.
-e: Shows all processes running on the system.
-f: Provides full details of each process (user, PID, command used to run it, etc.).
Below is the snapshot of output I got after running the ps -ef command.
Filtering Specific Processes: Using grep
Example 1: To display only the processes related to "Amazon," you would use:
ps -ef | grep “amazon”
This shows only one process related to "Amazon" running on the VM. Ignore the last line, we’ll talk
about that later.
Here’s how it works:
ps -ef generates the list of all processes.
| passes this output to grep.
grep Amazon filters the processes, showing only those that include "Amazon" in
their description.
Example 2: Using pipe and grep for filtering the output of a shell script file.
Consider the following script [Link]:
Copy
Copy
#!/bin/bash
echo 1
echo 11
echo 12
echo 55
echo 99
The output of the above script will include all the numbers but if you want to filter only the numbers
containing "1", you can pipe the output through grep:
Copy
Copy
./[Link] | grep 1
This will print:
Copy
Copy
11
12
Extracting Process IDs with awk
The requirement is to get the process IDs of all the processes related to Amazon.
To do this, you'd typically use the ps command to list all running processes, grep to filter processes to
show only those related to Amazon, and then awk to extract specific details such as the Process ID
(PID).
Output:
awk vs grep: Key Differences
grep: Searches for a pattern in the output and displays entire matching lines. It is
great for quickly finding lines that contain a specific string.
awk: Goes a step further by allowing you to extract specific fields (columns) or
perform operations on the data. It doesn’t just match lines—it can analyze and
manipulate them.
Understanding Pipes in Shell Scripting:
How Pipes Work in Shell Scripting: In shell scripting, a pipe (|) is used to send the output of one
command (the left-side command) as input to the next command (the right-side command). For
example:
Here, command1’s output is passed as input to command2.
However, this rule doesn’t always apply uniformly, depending on how the commands handle their
input and output.
Example:
Consider the following command:
At first glance, you might expect this command to output something like:
But what actually happens when you run this? Output:
Why Does This Happen? Pipes pass data from one command’s STDOUT to another’s STDIN, but the
receiving command must be capable of reading from STDIN.
Commands like echo are built to print arguments directly, while other commands, such
as grep or cat, read from STDIN and would behave differently when used with a pipe.
In the case of the date command, its output is passed through STDOUT, but echo does not read
from STDIN, so the output of date is essentially discarded.
Best Practices in Shell Scripting: Using set -e and set -o pipefail
1. set -e: Stop the Script on Error
The set -e option is one of the most widely used commands in shell scripting. Its purpose is simple: it
exits the script when an error occurs.
Why is set -e important?
Imagine you are writing a shell script that has multiple steps, like:
Create a user.
Create a file.
Add the username to the file.
Now, let’s say the first step (creating the user) fails. Without set -e, the script would continue to
execute the next steps, which would result in an incorrect or incomplete output. For example, the file
may be created, but it would not contain the username because the user wasn't created. The script
would complete execution even though the critical first step failed.
However, with set -e, the script will exit immediately after encountering the error in the first step.
This prevents the subsequent steps from running, ensuring that if a step fails, the entire process
stops.
2. Limitation of set -e with Pipes
In pipelines (e.g., cmd1 | cmd2), set -e only checks the last command. If cmd1 fails
but cmd2 succeeds, the script won’t stop.
3. Solution: Using set -o pipefail
Ensures that the script stops if any command in the pipeline fails, and not just the last one.
Example:
How DevOps Engineers Handle Application Errors Using Log Files
1. Searching for Errors in Log Files
When an application fails, the first step for any DevOps engineer is to check the logs for error
messages. These logs typically contain various levels of logging information (e.g., trace, info, error),
but the engineer’s focus is usually on the error logs.
Basic Search Using grep
Here:
cat prints the contents of the log file.
grep "ERROR" filters and shows only the lines containing the word "ERROR."
This is a straightforward way to quickly identify error logs, especially in large files.
2. Handling Remote Log Files (curl command)
In modern architectures, log files are often too large to store locally on the server. Therefore, many
organizations upload them to external storage services like:
Amazon S3
Google Cloud Storage
Azure Blob Storage
These services help manage large logs effectively. Once logs are stored remotely, how do engineers
retrieve and analyze them?
Retrieving Logs Using curl
The curl command is a versatile tool used for transferring data from external sources. In this context,
it’s commonly used to accessing log files from external storage directly into the terminal for analysis.
Example of using curl:
This command:
Retrieves the log file from the provided URL using curl.
Filters the output to show only error messages using grep.
Understanding curl for API Requests
In addition to fetching log files, curl can be used for making API calls, much like tools such
as Postman or the requests module in Python. For instance, you can use curl to interact with APIs
and retrieve data or send requests.
Example of using curl for API calls:
In this case, curl performs a GET request to the specified API endpoint.
3. Comparing curl with wget
Another common command for retrieving files is wget.
curl vs. wget: Key Differences
curl:
Outputs data directly to the terminal or console by default.
Can be used to interact with APIs and perform requests such as GET, POST,
etc.
Best for when you do not need to save the file locally, but want to inspect or
process it immediately.
wget:
Downloads the file and saves it to the local disk.
It’s ideal for downloading files when you need a local copy to work with.
Example using wget:
Here, wget downloads the log file and stores it locally, where you can then process it using grep.
Using the find Command in DevOps
In DevOps, managing numerous files across directories is a routine task, and the find command
becomes invaluable. It helps locate files based on name, size, or modification date, making it a crucial
tool for troubleshooting in production environments.
1. Basic Usage of the find Command
To locate a file named pam.d:
/: Searches the root directory.
-name pam.d: Specifies the file to locate.
2. Using sudo to Avoid Permission Issues
Running find without root access can lead to "Permission denied" errors, especially in protected
directories. To bypass this, use sudo:
This grants root privileges, allowing the command to search all directories without restriction.
3. Switching to Root User: sudo su -
When you need broader control over system tasks, switching to the root user is essential. Use the
following command:
This provides full administrative access, enabling you to execute privileged commands efficiently.
Alternatively, for temporary root access, prepend sudo to any command:
For instance:
4. Advanced find Options
You can refine your search further using various filters:
Search by file extension:
Search by file size:
Search by modification date:
if-else in Shell Scripting
Syntax:
if [ condition ]: The condition is placed within square brackets [ ]. The brackets are used to
evaluate the condition, and there must be spaces between the brackets and the condition.
then: If the condition is true, the commands under then are executed.
else: If the condition is false, the commands under else are executed.
fi: This is the end of the if block. It's simply the reverse of if.
Example of if-else :
Key Points:
No Spaces in Assignment: Variables should be assigned without spaces, e.g., a=4 (not a = 4).
Condition Operators:
-gt: Greater than
-lt: Less than
-eq: Equal to
Closing the if block: Always end the if condition with fi.
for Loop in Shell Scripting
A for loop is used when you want to perform a task multiple times without manually repeating the
same code. For example, if you are tasked with printing numbers from 1 to 10, or listing the names of
all students in a class, a for loop simplifies the task by automating it.
Basic Syntax of a for Loop in Bash:
for i in {start..end}: This sets the loop variable i, which iterates from the start value to
the end value.
do: This keyword tells the loop what actions to perform in each iteration.
echo $i: This prints the value of i during each iteration.
done: This marks the end of the loop.
Example: Printing Numbers from 1 to 100
Output:
Understanding the Trap Command in Linux
The Trap command in Linux is a powerful tool used to handle and control signals. Though it's not
frequently used in every script, it can play a crucial role when writing complex scripts.
What are Signals in Linux?
Before diving into the Trap command, it's essential to understand signals in Linux. Signals are
software interrupts sent to a program to inform it that a specific event has occurred. There are many
types of signals, each with different purposes, but some commonly used ones include:
SIGINT: Interrupt signal (triggered by pressing Ctrl + C).
SIGKILL: Kill signal to terminate a process.
SIGHUP: Hangup signal, often used to reload configurations.
For example, pressing Ctrl + C in the terminal sends the SIGINT signal to a running process, which
usually stops the process.
Introduction to the Trap Command
The Trap command is used to trap and respond to these signals. When a script is running, if a
specific signal (like SIGINT from Ctrl + C) is sent, the Trap command can intercept that signal
and perform a custom action.
For example:
You might not want a script to be terminated if someone presses Ctrl + C.
Or, you may want to clean up files or log a message before allowing the process to terminate.
How the Trap Command Works
The basic syntax for the Trap command is:
'commands': The action you want to perform when the signal is received.
signal: The signal you want to trap (e.g., SIGINT).
Why Use Trap?
In real-world scenarios, using the Trap command can be very beneficial. Let’s go over a couple of
cases:
1. Preventing Data Corruption: Suppose you have a shell script populating a database. If
someone interrupts the process using Ctrl + C, it could result in only partial data being
inserted. By trapping the signal, you can handle the situation gracefully, perhaps by rolling
back the operation or deleting partial data, thus preventing inconsistencies.
In this example, if someone tries to interrupt the process, all temporary files are deleted, ensuring no
incomplete data remains.
Key Signals in Linux
There are many signals in Linux, but these are the most common ones you might encounter:
SIGINT (2): Interrupt from the keyboard (Ctrl + C).
SIGTERM (15): Termination signal.
SIGKILL (9): Forcefully kills a process (cannot be trapped).
SIGHUP (1): Terminal hangup, often used to reload configurations.
Each signal serves a different purpose and can be trapped using the Trap command (except SIGKILL,
which cannot be trapped).