2.

Basic Unix and Shell Scripting to Solve Physics-Based Problems

After this week you should be able to:

●​ differentiate between different operating systems

●​ understand the difference between the Terminal and Windows PowerShell
●​ Write basic shell commands
●​ Be able to recall and execute basic Linux commands associated with
creating and modifying file

2.1 Introduction to Unix/Linux

Unix/Linux vs. Windows: A Basic Comparison

Unix and Linux are operating systems that share a common ancestry but have distinct
characteristics. They differ significantly from Windows, a proprietary operating system
developed by Microsoft. Here's a brief overview:

Core Differences
Open Source vs. Proprietary:
-​ Unix/Linux: Open-source, meaning their source code is publicly available for anyone to
modify and distribute. This fosters a large community of developers and often leads to
faster innovation and better security.
-​ Windows: Proprietary, meaning Microsoft owns the source code and controls its
distribution. This can limit customization and innovation but also ensures a consistent
user experience.
Multi-user vs. Single-user (Historically):
-​ Unix/Linux: Historically designed for multi-user environments, allowing multiple users
to access the system simultaneously. This is still a core feature, making them ideal for
servers and workstations.
-​ Windows: Initially designed for single-user environments but has evolved to support
multiple users. However, it's still primarily used for personal computers.
Command-line Interface (CLI) vs. Graphical User Interface (GUI):
-​ Unix/Linux: Traditionally more focused on the CLI, a text-based interface that requires
users to enter commands. While modern distributions have GUIs, the CLI is still essential
for many tasks.
-​ Windows: Primarily relies on a GUI, making it more user-friendly for those unfamiliar
with the command line.
Kernel Structure:
-​ Unix/Linux: Typically use a monolithic kernel, where the operating system's core
components are tightly integrated. This can lead to better performance but can also be
more complex to maintain.
-​ Windows: Uses a microkernel, which separates the core system components into smaller
modules. This can improve modularity and security but can also introduce overhead.
Additional Considerations
-​ Security: Unix/Linux are generally considered more secure due to their open-source
nature and focus on security best practices.
-​ Performance: Unix/Linux often offer better performance, especially in server
environments, due to their efficient use of system resources.
-​ Customization: Unix/Linux are highly customizable, allowing users to tailor the system
to their specific needs.
-​ Software Ecosystem: Windows has a larger software ecosystem, with more commercial
applications available. However, Unix/Linux have a growing number of applications,
especially in areas like web development and server administration.
In conclusion, while Unix/Linux and Windows share some similarities, they have distinct
characteristics that make them suitable for different use cases. Unix/Linux are often preferred for
servers, workstations, and tasks that require high performance, security, and customization.
Windows is more commonly used for personal computers and applications that require a
user-friendly GUI and a large software ecosystem.

Other Relevant Information:

Kernel is the central component of an operating system that manages operations of computer
and hardware. It basically manages operations of memory and CPU time.

Monolithic Kernel: A Unified Core

A monolithic kernel is a type of operating system kernel where the core components, such as the
process manager, memory manager, and device drivers, are tightly integrated into a single, large
executable. This structure offers several advantages and disadvantages.

Advantages of Monolithic Kernels:

●​ Efficiency: The close integration of components can lead to efficient communication and
resource allocation, resulting in better performance.
●​ Simplicity: The monolithic design can be simpler to implement and maintain compared
to microkernels, which have a more modular structure.
●​ Predictability: The tight coupling of components can make it easier to predict and
control the system's behaviour.

Disadvantages of Monolithic Kernels:

●​ Complexity: As the system grows, the monolithic kernel can become increasingly
complex and difficult to manage.
●​ Security: A single point of failure can compromise the entire system, making it
vulnerable to security breaches.
●​ Flexibility: The tightly coupled design can limit flexibility and make it difficult to add or
remove features without affecting the entire system.

Examples of Operating Systems Using Monolithic Kernels:

●​ Linux: One of the most widely used monolithic kernels.

●​ macOS: Based on the BSD family of operating systems, which also use monolithic
kernels.
Microkernel: A Modular Approach
A microkernel is a type of operating system kernel that consists of a small core of essential
services, such as process management, memory management, and inter-process communication
(IPC). Other system components, such as device drivers and file systems, are implemented as
separate server processes that communicate with the microkernel using well-defined interfaces.

Advantages of Microkernels:
Modularity: The modular structure makes it easier to add, remove, or modify components
without affecting the core system.
Security: Each server process runs in its own protected address space, reducing the risk of
security breaches.
Reliability: The modular design can improve system reliability by isolating failures to individual
components.
Flexibility: Microkernels can be more flexible than monolithic kernels, allowing for
customization and experimentation.
Disadvantages of Microkernels:
Performance: The overhead of inter-process communication can sometimes lead to performance
degradation compared to monolithic kernels.
Complexity: The modular design can introduce additional complexity and overhead.
Standardisation: The lack of a standardised microkernel architecture can make it difficult to port
applications between different microkernel-based systems.

Basic Commands Practice:

Navigation: Use ls, cd, pwd, mkdir, rmdir to navigate and manage directories.

mkdir physics_data
cd physics_data
touch [Link]
ls

File Operations: Practice copying, moving, and deleting files.

cp [Link] [Link]
mv [Link] experiment_backup.txt
rm experiment_backup.txt

Hands-On Exercise:
Create a directory structure for a hypothetical physics project. Include directories like data,
scripts, and results. Add some sample files in each directory and practice navigating and
managing them.

2.2 File Management and Permissions

Practical Steps:
Permissions:
Check Permissions: Use ls -l to view file permissions.

ls -l [Link]

Change Permissions: Use chmod to modify permissions.

chmod 644 [Link] # Read/write for owner, read-only for others

Change Ownership: Use chown to change file ownership.

sudo chown yourusername:yourusername [Link]

Hands-On Exercise:

Change the permissions of a file so that only the owner can write to it, and verify the changes.
Try changing ownership and check if you can access the file.

Linux File System

Fig.1 : The top levels of a typical Linux filesystem

Key File Structure and Roles

-​ Root Directory (/)

Role: The starting point of the file system hierarchy, often called the "root". All other
directories and files branch from here.
Relevance: Everything is contained under /. Understanding this will help in knowing
where files and programs reside.

-​ /bin (Binaries)
Role: Contains essential system binaries and commands that are required for the system
to boot and operate, such as ls, cp, and mv.
Relevance: Frequently used commands are stored in /bin for tasks like moving between
directories or managing files. For example, running cp or rm to manage files in a project.

-​ /usr (User Programs)

Role: Contains user-installed programs and libraries. It has subdirectories like /usr/bin for
binaries and /usr/lib for libraries.
Relevance: When installing additional software (e.g., Python, LaTeX), the binaries often
go here.

-​ /lib (Libraries)
Role: Contains essential libraries for binaries in /bin and /sbin. These libraries are
necessary for basic system functionality.
Relevance: While they won’t interact with it directly, understanding that libraries stored
here are used by the system or scientific computing programs is crucial.

-​ /etc (Configuration Files)

Role: Stores system-wide configuration files. For example, network settings, user
permissions, and software configuration files are stored here.
Relevance: When managing custom configurations for physics software, one might edit
files here. For instance, configuring a computational environment or setting up remote
access to data resources.

-​ /home (User Home Directories)

Role: Each user has a personal directory here (e.g., /home/alice), where their personal
files, scripts, and projects are stored.
 Relevance: This is where workings are stored, such as physics simulations, datasets, and
LaTeX reports. It is the directory where one should focus their organisational efforts for
computational work.

-​ /dev (Devices)
Role: Holds device files that represent hardware components like hard drives, USBs, etc.
Relevance: When working with large-scale datasets in computational physics, one might
mount external storage devices from here.

-​ /mnt and /media (Mount Points)

Role: Temporary mount points for removable storage, such as USB drives or external
hard drives.
Relevance: To load external datasets or backup results from computational experiments.

-​ /tmp (Temporary Files)

Role: Stores temporary files created by programs. These files are often deleted when the
system reboots.
Relevance: In computational physics, programs like simulations might store temporary
data here. It’s a space one should know to avoid placing permanent data, as files can be
erased automatically.

-​ /var (Variable Data)

Role: Contains files that are expected to grow in size, such as system logs and databases.
Relevance: In large-scale computations, log files from programs might be stored here.
This can be useful for debugging physics simulations.

-​ /sbin (System Binaries)

Role: Stores essential system binaries that are used for administrative tasks like
networking, disk partitioning, etc.
Relevance: Advanced researchers working with supercomputers or managing their own
Linux servers may need to use administrative tools found here for setting up their
computational environment.

-​ /Users
Role: Some systems (particularly macOS and Unix-based systems) may use this to store
user directories like Alice and Bob.
Relevance: Similar to /home in Linux systems, this is where one would store their
personal projects and data.
2.3 Advanced Linux Commands

This section covers advanced Linux commands with a focus on text processing, which is
very useful in computational physics or data analysis tasks. Let’s explore grep, awk, and
sed with examples that could be applied to a physics experiment dataset.

1.​ grep – Search for Patterns in Files

The grep command is used to search for specific patterns of text within files. You can
look for keywords, phrases, or numbers that match your query.

Example: Let's say we have a physics experiment log in several .txt files and we need
to search for lines mentioning the word "voltage" in all .txt files.

bash

grep "voltage" *.txt

-​ This command will return every line containing "voltage" across all .txt files in
the directory.
-​ You can also use options like -i to ignore case and -n to show line numbers.

Hands-On Example:

Let’s assume we have a file [Link] that contains:

time: 12:00​ voltage: 2.5V

time: 12:30​ current: 0.5A
time: 01:00​ voltage: 2.7V

To search for all lines mentioning "voltage", you would use:

bash
grep "voltage" [Link]
 Output:
time: 12:00​ voltage: 2.5V
time: 01:00​ voltage: 2.7V

2.​ awk – Process and Analyze Text Data

The awk command is a powerful text processing tool that can extract and analyse
columns of data in files. It is especially useful when working with structured data like
CSVs or space-separated values.

Example: If you have a file where columns represent time, voltage, and current, and you
want to extract just the time from [Link]:
​ bash
awk '{print $1}' [Link]
-​ This command will print the first column, which is time in this case.
​
​ Hands-On Example:

Given the same [Link] file:

time: 12:00​ voltage: 2.5V

time: 12:30​ current: 0.5A

time: 01:00​ voltage: 2.7V

If you want to extract the voltage values (which are in the third column):

awk '{print $3}' [Link]

Output:

2.5V

0.5A

2.7V

You can also perform arithmetic operations in awk. For example, if the voltage values are in a file
and

you want to calculate their sum, you can use:

awk '{sum += $3} END {print sum}' [Link]

 3.​ sed – Stream Edit Text Files

The sed command is used to find and replace text in a file or stream. It's
often used to perform batch editing, such as changing variable names or
updating experiment data.

Example:If you want to replace the word "voltage" with "potential" in

[Link], you can use:
bash

sed 's/voltage/potential/g' [Link] > updated_experiment1.txt

-​ s/oldtext/newtext/g: This command replaces all occurrences of oldtext with

newtext.
-​ The result will be saved in a new file updated_experiment1.txt.

Hands-On Example:

For the same [Link] file:

​ time: 12:00​ voltage: 2.5V
time: 12:30​ current: 0.5A
time: 01:00​ voltage: 2.7V

If you want to replace all instances of "voltage" with "potential", you can run:
sed 's/voltage/potential/g' [Link]
Output:
​ time: 12:00​ potential: 2.5V
time: 12:30​ current: 0.5A
time: 01:00​ potential: 2.7V

Hands-On Exercise: Analyzing Physics Experiment Data

Let’s walk through a complete exercise where you use grep, awk, and sed to
manipulate and analyze data from a simple physics experiment.

Step 1: Create a Sample Data File

Create a file named experiment_data.txt with the following data:

time: 10:00​ voltage: 3.1V​ current: 1.2A

 time: 10:30​ voltage: 3.0V​ current: 1.1A

time: 11:00​ voltage: 2.9V​ current: 1.0A

time: 11:30​ voltage: 3.2V​ current: 1.3A

time: 12:00​ voltage: 3.0V​ current: 1.2A

Step 2: Use grep to Find Specific Terms

Suppose you want to find all the readings taken at 10:30. Use:
​

grep "10:30" experiment_data.txt

Step 3: Use awk to Extract Columns of Data

Now, extract only the voltage values from this file:

​ awk '{print $3}' experiment_data.txt

This will return the list of voltages.

​ Step 4: Use sed to Modify Data

Let’s assume you’ve realised the voltage readings are in millivolts (mV) instead of volts
(V), and you want to modify this in the data. You can replace V with mV like this:

sed 's/V/mV/g' experiment_data.txt > updated_experiment_data.txt

If you want to convert the entire data points rather than just changing the unit label, you’ll
need to perform a calculation in addition to modifying the unit label. For instance, if you
want to convert voltage readings from millivolts (mV) to volts (V), you'll need to divide
each value by 1000.

Here’s how you can do it using awk instead of sed, since sed is not designed for
mathematical operations. Suppose your data is formatted as:

time voltage

0​ 500

1​ 1500

2​ 2500
And you want to convert the voltage from mV to V.

1.​ Using awk for Calculation and Unit Conversion:

awk '{if (NR == 1) {print $1, "voltage(V)"} else {print $1, $2/1000}}'
experiment_data.txt > updated_experiment_data.txt

-​ NR == 1 handles the header row, printing "voltage(V)".

-​ For other rows, it divides the voltage value by 1000.
2.​ Combining the Unit Conversion with sed for Label Replacement:

If you also need to replace the unit label in the header or data file, do it in two
steps:

awk '{if (NR == 1) {print $1, "voltage(V)"} else {print $1, $2/1000}}'
experiment_data.txt | sed 's/V/mV/g' > updated_experiment_data.txt

This command combines awk to perform the calculation and sed to replace the
unit labels if needed.
Definition— Shell Scripting?

Shell Scripting is a programming paradigm that involves writing scripts using a shell language,
which is a command-line interpreter for operating systems. These scripts automate tasks that
would otherwise be performed manually at the command prompt.

Key characteristics of shell scripting:

-​ Interpreted: Shell scripts are executed line by line by the shell interpreter, rather than
being compiled into machine code.
-​ Text-based: Scripts are written in plain text format, making them easy to create, edit, and
understand.
-​ Command-based: Shell scripts consist of commands and control structures (like loops
and conditionals) that are familiar to users of the command line.
-​ Platform-specific: While many shell languages share common syntax, the specific
commands and features may vary between different operating systems (e.g., Bash on
Linux/macOS and PowerShell on Windows).

Common uses of shell scripting:

-​ System administration: Automating tasks like file management, user management, and
network configuration.
-​ Data processing: Performing data manipulation and analysis.
-​ Automation: Creating scripts to automate repetitive tasks, saving time and effort.
-​ Scripting other programs: Using shell scripts to control and automate other programs.
Popular shell languages:
-​ Bash: The default shell on most Linux and macOS systems.
-​ Zsh: A more feature-rich alternative to Bash.
-​ KornShell (ksh): A traditional shell with a strong following.
-​ PowerShell: The default shell on Windows systems.
 -​ By learning shell scripting, you can significantly enhance your productivity and
efficiency in working with command-line interfaces.

Why—Shell Scripting?
Shell scripting offers several compelling advantages that make it a valuable tool for system
administrators, developers, and anyone who works with command-line interfaces. Here are some
key reasons:
1.​ Automation:
Repetitive tasks: Shell scripts can automate repetitive tasks, saving time and reducing the
risk of human error.
Scheduling: Scripts can be scheduled to run automatically at specific times or intervals,
ensuring that tasks are performed consistently
2.​ Efficiency:
Batch processing: Shell scripts can process large amounts of data in batches, improving
efficiency and productivity.
Integration: Scripts can be used to integrate different tools and applications, streamlining
workflows.
3.​ Customization:
Tailored solutions: Shell scripts can be customised to meet specific needs and
requirements, providing flexible solutions to a wide range of problems.
Integration with other tools: Scripts can be integrated with other tools and programming
languages, expanding their capabilities.
4.​ Control:
Precise control: Shell scripts allow for precise control over system processes and
operations.
Debugging: Scripts can be easily debugged and modified, making it easier to identify and
fix issues.
5.​ Portability:
Cross-platform compatibility: Shell scripts can often be ported to different operating
systems with minimal modifications, making them versatile and adaptable.
6.​ Learning and development:
Foundation for programming: Shell scripting can serve as a foundation for learning other
programming languages, as it introduces fundamental concepts like variables, loops, and
conditionals.
Problem-solving skills: Developing shell scripts can help improve problem-solving skills
and logical thinking.

Overall, shell scripting is a powerful and versatile tool that can be used to automate tasks,
improve efficiency, and customise system behaviour. Whether you're a system administrator,
developer, or simply someone who wants to get more out of your command-line interface, shell
scripting is a valuable skill to acquire.

Setup:
Install a Unix/Linux Environment: Use a virtual machine (VM) or WSL (Windows Subsystem
for Linux) if you’re on Windows. Ubuntu is a good choice for beginners.
Access Terminal: Open the terminal or command-line interface in your environment.

2.4 Shell Scripting Basics

Objective:
Write and execute basic shell scripts.

Practical Steps:
Create a Simple Script:

Script Example:

#!/bin/bash
echo "Starting physics data processing..."
mkdir -p results
cp data/*.txt results/
echo "Data copied to results directory."
Save the Script: Save this script as process_data.sh.
Make Script Executable and Run:

Set Executable Permission:

chmod +x process_data.sh
Execute the Script:
./process_data.sh

Hands-On Exercise:
Write a script that automates a task specific to your physics project, such as organising data files
or generating summary reports.

2.5 Automation with Shell Scripts

Objective:
Automate complex tasks and workflows using shell scripts.
Practical Steps:
Advanced Scripting:

Use Functions:
#!/bin/bash
process_data() {
echo "Processing $1"
cp "$1" results/
}

for file in data/*.txt; do

process_data "$file"
done
Automate with cron: Schedule your script to run at regular intervals.

crontab -e
# Add a line to run the script daily at midnight
0 0 * * * /path/to/process_data.sh

Hands-On Project:

Develop a comprehensive script that handles the entire workflow of your physics project, such as
data collection, processing, and reporting. Use cron to schedule regular updates or processing.