100% found this document useful (1 vote)
83 views9 pages

Bash Scripting for Data Automation

This document discusses shell scripting in bash and provides examples of common shell commands. It covers commands for searching/parsing files, manipulating variables and strings, looping, conditional logic, parsing arguments, changing directories, and automating tasks. The document also provides examples of using these commands to automate tasks like file searching/parsing, regression testing, and more.

Uploaded by

Priya Jeejo
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
100% found this document useful (1 vote)
83 views9 pages

Bash Scripting for Data Automation

This document discusses shell scripting in bash and provides examples of common shell commands. It covers commands for searching/parsing files, manipulating variables and strings, looping, conditional logic, parsing arguments, changing directories, and automating tasks. The document also provides examples of using these commands to automate tasks like file searching/parsing, regression testing, and more.

Uploaded by

Priya Jeejo
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
  • Shell Commands
  • Introduction
  • Exercises
  • Log File Parsing
  • Argument Parsing
  • File Parsing
  • Regression Automation
  • Report

Shell Script

Table of Contents
1) Introduction.............................................................................................................................................2
2) Shell Commands......................................................................................................................................2
3) Exercises..................................................................................................................................................4
a) Search & Count....................................................................................................................................4
b) Search and replace..............................................................................................................................4
c) Comparison..........................................................................................................................................4
d) Log file parsing.....................................................................................................................................4
e) Argument parsing................................................................................................................................6
f) File parsing...........................................................................................................................................6
4) Regression Automation...........................................................................................................................7
g) Setup....................................................................................................................................................7
h) Running...............................................................................................................................................7
i) Reporting..............................................................................................................................................7
j) Directory...............................................................................................................................................7
k) Test list.................................................................................................................................................7
l) Logs.......................................................................................................................................................7
m) Report.................................................................................................................................................8

1|Page P J Nandagopal
Shell Script

a) Introduction
This document talks about shell scripting using bash shell. Scripting is necessary for everyone:
(1) To automate hectic manual work
(2) Reduce the time taken to complete the task
(3) Avoid manual errors (oversee, miscalculation)
(4) Off office hours
(5) Avoid particular person presence

Bash shell is linux/unix shell which has set of commands for simplifying manual operation.

b) Bash shell commands


No Command Purpose Description
1 grep Find a pattern in a file grep “pattern” file;
cat file | grep “pattern”
cat README | grep “Section” | grep “How To
Run”
2 find Find a file find . –name ‘*.txt’
find /home/nanda/project “[Link]”
3 sed Replace pattern with sed “s/abcd/1234/g” [Link] > [Link]
new pattern mv [Link] [Link]
sed “s/abcd/1234/1” [Link] (single instance)
sed “s/^abc/123/g” [Link] (SOL)
sed “s/abc$/123/g” [Link] (EOL)
sed “s/ab[.]/123/g” [Link]
sed “s/ab[.*]/123/g” [Link]
sed “s/abc/123/gc” [Link] (user permission for
each replacement)
4 expr Arithmetic operation var=10
for count, add, var=`expr ‘$var + 1’` (Increment)
subtract, multiply var=`expr ‘$var \* 2` (Multiply by 2)
division var=`expr “$var \/ 2”`
var=`expr “$var % 2”`
5 echo Print variables & text name=”nanda”
echo “Hai how are you?”
echo “Hai $nameee how are you?”
6 awk Used for extracting File [Link] contains “A B C D”
fields cat [Link] |awk ‘{print $2 “ and ” $4}’
Output will be “B and D”
7 for Fixed condition loop

[Link] contains names of verif engr


for entry in `cat [Link]`
do

2|Page P J Nandagopal
Shell Script

echo “$entry”
done

// Verilog For iii=0; iii<3; iii++


for iii in 0 1 2
do
echo “$iii”
done

// Raghu: Another way - to be tried


for iii in 1 to 10
echo “$iii”
done
8 while Dynamic condition count=1
loop // -lt, -le, -ge, -gt, -eq
while [ “$count” –le 10 ]
do
count=`expr $count + 1`
done

while [ “$tstname” != “perf” ]


9 >, >>, |& tee Write and append to echo “Nanda” > [Link]
file echo “Gopal” >> [Link]
\rm –f [Link]
echo “Nanda” |& tee [Link]
10 abcd=1 Variable declaration Type of variable is dynamically inferred
abcd=”hai” depending on value assigned.
11 function func_1 ()
{
echo “I am in func_1”
}

abcd=1
func_1;

// Raghu: Whether params can be passed?


12 $1, $2, … Command line
arguments parsing

if [ “$1” ]
then
if [ “$1 = “Yes” ]
func_1;

3|Page P J Nandagopal
Shell Script

elif [ “$1” = “No” ]


then
echo “Function not trigerred”
fi
else
echo “No argument passed”
fi

sh [Link]
No argument passed

sh [Link] No
Function not trigerred

sh [Link] who
Prints nothing

13 wc -c Word count ls | wc -c will print number of files in the


wc -l directory
14 pushd . Going inside and cd /home/nanda/ccore
popd coming out of rm –f $HOME/[Link]
directory for file in `cat [Link]`
do
echo “- $dir -“ >> $HOME/[Link]
pushd .
cd $dir
grep “L2CacheLoad” * >> $HOME/[Link]
popd
Done

Pushd .
Cd dir1; (cd tests/i2c)
Pushd .
Ls |wc -l
Cd dir2; (cd wr_rd_tsts)
Grep |awk |sed
Popd
Popd

4|Page P J Nandagopal
Shell Script

c) Exercises
a) Search & Count
Count how many oranges in the context.
Open an input file “[Link]” and type the following as shown below:
A man went to shop and bought 6 oranges. He went to another shop and bought 8 oranges. He went
to one more shop and bought 6 apples.

b) Search and replace


File “[Link]” Contents (ls-dR will give the list of all directory and sub directory)
/home/nanda/ccore/tb
/home/nanda/ccore/tb/vip/i2c
/home/nanda/ccore/tb/vip/i2c/sv_lib/tests

You are in home directory: /home/nanda


There are similar directories like i2c, i3c, spi, i2s etc are available

For all directories, run your script to replace


tests --> testcases
sv_lib --> uvm_tests
count & print --> no of tests
c) Comparison
Find out which column is bigger between Column A & C (B column should be skipped) and create
“[Link] file” with column A, C, CompareResult as shown below:
[Link] file entry:
10 9 20
20 25 15
125 256 1000
1019 900 956
121 121 121
Output file (script to generate output file)
A C Result
==================
10 20 C
20 15 A
125 1000 C
1019 956 A
121 121 Equal

d) Log file parsing


Count number of errors and warnings and give summary table
Segregate different kinds of errors
Log file
Simulation started

5|Page P J Nandagopal
Shell Script

Error: File not found


ERROR: Reset not set
Error: clock path
warning: reset is active high
Error: clock path
Error: Reset not set
Error: clock path

6|Page P J Nandagopal
Shell Script

e) Argument parsing
Pass four arguments and add even arguments and subtract odd arguments and print the results
(i.e. arg2 + arg4 and arg1-arg3). Report error arguments are more or less than four.

f) File parsing
Read line by line and insert pattern “#start#” at start of each line and pattern “#end#” at end of line

7|Page P J Nandagopal
Shell Script

d) Regression Automation
Here let us see how to automate regression and generate reports.
g) Setup
Creating new directory & fresh checkout.
h) Running
Creating list of tests & combinations.
i) Reporting
Looking at all logs and generating summary.
Parsing logs and consolidating errors.
Comparing with previous regression report.

j) Directory
Base Directory ($base_dir): /home/nandag/project/kss/cpuss/verif
Run Directory ($sim_dir): $base_dir/sim
Tests Directory ($tests_dir): $base_dir/tests
Log Directory ($log_dir): $base_dir/sim/logs
Script Directory ($script_dir): $base_dir/scr

k) Test list
Make sim TEST=”test_basic_1”
Make sim TEST=”test_basic_2” WIDTH=32
Make sim TEST=”test_basic_2” WIDTH=64
Make sim TEST=”test_basic_3”

Make sim TEST=”test_feature_1” MODE=0 LAUNCH=POSITIVE


Make sim TEST=”test_feature_1” MODE=0 LAUNCH=NEGATIVE
Make sim TEST=”test_feature_1” MODE=1 LAUNCH=POSITIVE
Make sim TEST=”test_feature_1” MODE=1 LAUNCH=NEGATIVE
Make sim TEST=”test_feature_2” MODE=0
Make sim TEST=”test_feature_2” MODE=1
Make sim TEST=”test_feature_3” MODE=0 LAUNCH=POSITIVE
Make sim TEST=”test_feature_3” MODE=0 LAUNCH=NEGATIVE
Make sim TEST=”test_feature_3” MODE=1 LAUNCH=POSITIVE
Make sim TEST=”test_feature_3” MODE=1 LAUNCH=NEGATIVE

l) Logs
Inside $log_dir/testname_seed.log
End of test: “TEST PASSED” or “TEST FAILED” message will be there
Error: compile failed
Error: Elaboration failed
Error: data mismatch
Error: timing violation
Error: ID mismatch

8|Page P J Nandagopal
Shell Script

m) Report
Summary
Total Tests: 14 (Prev: 14)
Pass: 10 (Prev: 8)
Fail: 4 (Prev: 6)

Error wise
Error-1: 2 tests
Error-4: 2 tests

9|Page P J Nandagopal

Common questions

Powered by AI

The document describes using commands like 'pushd' and 'popd' to navigate directories, allowing scripts to change directory context and return to the original path, facilitating organized file management and preventing data loss or misplacement .

The strategies include creating multiple test combinations and variations (e.g., different widths, modes, and launch settings), ensuring comprehensive coverage of test scenarios by maintaining a structured and up-to-date test list within defined directories .

Log file parsing involves counting errors and warnings to generate a summary table. Different kinds of errors are segregated to aid in the focused analysis of issues, ensuring comprehensive error tracking and reporting .

The 'grep' command is used to search for a specific pattern within files. For example, 'grep "pattern" file' or 'cat file | grep "pattern"' will search and return lines in the file that match the specified pattern .

The document describes parsing command line arguments by checking if arguments are provided using conditional statements like 'if [ "$1" ]'. The arguments can trigger functions or print messages based on their value. It also includes error handling for the number of provided arguments .

The report summarization process involves examining all logs, generating summaries, and consolidating errors. It includes comparing current results with previous regression reports to track progress, highlight recurrent issues, and improve regression strategies .

Shell scripting provides automation for manual tasks, reduces the time taken to complete these tasks, minimizes manual errors such as overlook and miscalculation, allows execution outside regular office hours, and reduces dependency on a specific individual for task execution .

The 'sed' command is used for search and replace operations on text within files. It can replace patterns with new patterns globally or at specific occurrences, such as at the start or the end of a line .

The method involves comparing two columns from a dataset while skipping a third column and generating an output file that shows which column has a larger value for each row. This aids in data analysis by highlighting discrepancies or verifying data arrangements .

Arithmetic operations in shell scripts are performed using the 'expr' command, which allows for operations like incrementing a variable ('expr "$var + 1"'), multiplying ('expr "var \* 2"'), dividing, and finding the modulus ('expr "$var % 2"').

Shell Script
Table of Contents
1) Introduction...............................................................................
Shell Script
a) Introduction
This document talks about shell scripting using bash shell. Scripting is necessary for everyone:
Shell Script
  echo “$entry”
done
// Verilog For iii=0; iii<3; iii++
for iii in 0 1 2
do
  echo “$iii”
done
// Raghu: Another
Shell Script
  elif [ “$1” = “No” ]
  then
     echo “Function not trigerred” 
  fi
else
  echo “No argument passed”
fi
sh my
Shell Script
c) Exercises
a)
Search & Count
    Count how many oranges in the context.
    Open an input file “input.txt” and
Shell Script
     Error: File not found
     ERROR: Reset not set
     Error: clock path 
     warning: reset is active high
Shell Script
e)
Argument parsing
    Pass four arguments and add even arguments and subtract odd arguments and print the resu
Shell Script
d) Regression Automation
Here let us see how to automate regression and generate reports.
g)
Setup
Creating new
Shell Script
m) Report
Summary
Total Tests: 14 (Prev: 14)
Pass: 10 (Prev: 8)
Fail: 4 (Prev: 6)
Error wise
Error-1: 2 tests
Er

You might also like