Data Analysis With Unix
Why Unix?
Shell Scripting Efficiency:
• Bash excels in automating system administration tasks and
managing shell scripts efficiently due to its tight integration with the
Unix shell.
• Python scripts may require additional libraries or external tools for
certain system-level tasks that Bash can handle natively.
Why Unix?
Quick System Operations:
For quick system operations, such as file manipulation,
directory traversal, or command execution, Bash can be
more straightforward and faster to write.
Why Unix?
Integration with Command Line:
Bash seamlessly integrates with the command line, allowing direct
execution of commands and scripts without needing a separate
interpreter.
Why Unix?
Native Unix Functionality:
Bash provides direct access to Unix system calls and functionalities,
making it powerful for tasks like text processing, file manipulation,
and piping between commands.
Why Unix?
Better for Simple Tasks:
• For simple, one-off tasks or scripts that primarily involve executing
shell commands or manipulating files and directories, Bash scripts
can be more concise and efficient.
View a Text File
Cat Example_cat.txt
Find all the files in a directory
find directory_name -type f
grep – to manipulate or search text pattern within file
grep "example" [Link]
wc – word count
wc [Link]
Output of wc
• Number of lines
• Words
• Bytes
cut
• The cut command in Unix-like operating systems is used to extract sections from each line of
input or from files.
cut -d: -f2,3 [Link]
• -d: specifies the delimiter as a colon (:).
• -f1 specifies to extract the first field from each line.
sort
sort [Link]
uniq - remove duplicate line in the file
uniq [Link]
Using Pipe | - used to combine multiple commands
• data flows from left to right through the pipeline
sort [Link] | uniq | wc -l
Filters
The commands till now are
called filters.
Data passes through a filter.
Moreover, a filter can modify
data a bit on the way through
them. All filters read data from
the standard input and writes
data to standard output. Filter
can use the standard output of
another filter to be its standard
input while using the pipe “|”
operator.
awk
• awk is a versatile and powerful text processing tool in Unix-like
operating systems.
• It is mainly used for searching, filtering, and processing text or data
files line by line.
• awk operates on records, which are typically lines of text, and can
perform various operations such as pattern matching, data
extraction, and text manipulation.
awk
awk 'pattern { action }' filename
1. Print the first field of each line in a file : awk '{ print $1 }' filename
2. Print lines where the second field is greater than 10: awk '$2 > 10' filename
3. Calculate the total sum of the second field in a file:
awk '{ sum += $2 } END { print sum }' filename
Rapid Recall
Take the file [Link] and calculate the average grade
Solution
awk '{ total += $2 } END { print "Average grade:", total/NR }' [Link]
join
join [Link] [Link]
gnuplot
Use [Link] and answer the following
1. What is the average salary of employees in each department?
2. How many employees were hired in each year?
3. Which employee has the highest salary?
4. What is the total salary expense for each department?
5. How many employees hold each position?
6. What is the average tenure (in years) of employees in each department?
7. Identify the employee(s) who joined the company before a specific date.
8. How many employees have a salary greater than a certain threshold?
9. What is the distribution of employees across different departments?
10. Calculate the average salary increase percentage for employees who were promoted within the company.
Use [Link] to answer
• Extract the second column (names) from a CSV file named "[Link]" and print them in
reverse order
Plot
• Plot the salaries from the previous dataset