Intermediate Linux

COMPLECS Team
San Diego Supercomputer Center

[Link]
[Link]

COMPLECS NSF award #2320934 1

About COMPLECS

COMPLECS (COMPrehensive Learning for end-users to Effectively utilize

CyberinfraStructure) is a new SDSC program where training will cover
non-programming skills needed to effectively use supercomputers.
Topics include parallel computing concepts, Linux tools and bash
scripting, security, batch computing, how to get help, data
management and interactive computing.

COMPLECS is supported by
NSF award 2320934.

COMPLECS NSF award #2320934 2

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 3

Introduction

This session is for anyone who works in a Linux environment and wants to acquire
additional skills to more effectively use advanced cyberinfrastructure (CI)
In the Linux world, intermediate is in the eye of the beholder. Everything that we’re
covering in this session would be considered basic or elementary by Linux systems
administrators.
But if you’re attending this session, you’re probably not a sys admin. Rather, it’s
more likely you’re a student, educator or researcher who uses advanced CI as part
of your work or studies. In which case, these topics really are intermediate in the
context of Linux users instead of Linux professionals.
Note that shell scripting has now been split out into a separate session so that we
can focus more deeply on intermediate topics.

COMPLECS NSF award #2320934 4

Prerequisites
We assume that you already know the Linux basics, such as

• Listing files (ls)

• Removing files (rm)
• Navigating directories (cd)
• Listing file contents (cat and optionally head, tail, less, more)
• Ability to use one or more standard editors (vi, vim, emacs, nano)
• Relative and absolute paths

COMPLECS NSF award #2320934 5

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 6

Symbolic links
Symbolic links, also known as soft links, give you a way to refer to a file by a
different name. They’re useful when you want to avoid writing out a full or relative
path name every time or if you want to have a generic executable name refer to a
specific version.
To create a symbolic link, use the ln command with the "-s" option. Without "-s",
you'll end up creating a hard link (discussed in supplementary material) by default.

$ ln -s file1 soft-file1

In the next few slides, we'll explore some of the useful properties of symbolic links
and how they're used in real life.

COMPLECS NSF award #2320934 7

Symbolic links
Symbolic links can be created even if the file being linked to does not yet exist
$ ls
file1
$ ln -s file1 file1-soft # soft link to an existing file
$ ln -s file2 file2-soft # soft link to a nonexistent file

$ cat file1
Contents of file1
$ cat file1-soft
Contents of file1

$ cat file2-soft
cat: file2-soft: No such file or directory
$ echo "Contents of file2" > file2
$ cat file2-soft
Contents of file2

COMPLECS NSF award #2320934 8

Symbolic links
Removing the symbolic link does not affect the original file
$ ls
file1 file2 file1-soft file2-soft

$ rm file1-soft file2-soft
$ ls
file1 file2

COMPLECS NSF award #2320934 9

Symbolic links
A file can have multiple symbolic links
$ ls
file1

$ ln -s file1 file1-soft1
$ ln -s file1 file1-soft2
$ ln -s file1 file1-soft3

$ ls
file1 file1-soft1 file1-soft2 file1-soft3

COMPLECS NSF award #2320934 10

Symbolic links
Symbolic links can point to directories or files in other directories using absolute or
relative paths; they can even span file systems

$ pwd
/home/sinkovit

$ ln –s /expanse/lustre/scratch/sinkovit/file4 file4-soft

$ cat file4-soft
Contents of file4

or try this file:

/expanse/lustre/scratch/mthomas/temp_project/file4

COMPLECS NSF award #2320934 11

How do I know if a file is a symbolic link?
• ls -F appends '@' to the file name
• ls -l shows 'l' in first column of output and lists the file it points to
• ls --color=auto displays name in teal (but adds overhead to filesystem operations)

$ ls -F
file1 file1-soft@ file2 file2-soft@

$ ls -l
total 2
-rw-r--r-- 1 sinkovit use300 30 Oct 10 13:20 file1
lrwxrwxrwx 1 sinkovit use300 5 Oct 10 13:21 file1-soft -> file1
-rw-r--r-- 1 sinkovit use300 18 Oct 10 13:23 file2
lrwxrwxrwx 1 sinkovit use300 5 Oct 10 13:21 file2-soft -> file2

COMPLECS NSF award #2320934 12

Symbolic links use scenario
• An application we use has multiple versions. Although we occasionally need
access to earlier versions, we normally want the latest release. By creating a
symbolic link to this most recent version, we can avoid having to update our job
scripts or workflows by just updating the link.
• Our application takes multiple input files. While one of these files is different for
each calculation, others refer to reference data that rarely changes. We can
create symbolic links that point to the latest reference file.

COMPLECS NSF award #2320934 13

Symbolic links use scenario
$ cd $HOME/application/bin; ls
cfdapp_v1.[Link] cfdapp_v1.[Link] cfdapp_v1.[Link]
$ ln –s cfdapp_v1.[Link] [Link] Create symbolic links to
the latest version of the
software and reference
$ cd $HOME/application/refdata
data
ref_v1.[Link] ref_v1.[Link] ref_v1.[Link]
$ ln –s ref_v1.[Link] [Link]

PATH=${PATH}:${HOME}/application/bin Use symbolic links in job

REFPATH=${HOME}/application/refdata script to automatically
[Link] -r ${REFPATH}/[Link] [additional args] use latest versions.

COMPLECS NSF award #2320934 14

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 15

Redirection and pipes
Most Linux commands write to standard output (stdout), which is the terminal if
you’re working on the command line. Output can be redirected to a file using the >
symbol, with >> used to append output to a file.

$ date > outfile # Write date to file

$ cat outfile
Tue Jun 28 15:31:20 PDT 2022

$ date > outfile # Overwrites content of file

$ cat outfile
Tue Jun 28 15:31:25 PDT 2022

$ date >> outfile # Appends output to file

$ cat outfile
Tue Jun 28 15:31:25 PDT 2022
Tue Jun 28 15:31:30 PDT 2022

COMPLECS NSF award #2320934 16

Redirection and pipes
A pipe (|) takes stdout from one command and directs it to stdin of a second
command. An arbitrary number of commands can be connected using pipes.

# Count number of files in current directory

$ ls -1 | wc -l
13

# List the three longest running jobs on Expanse node exp-2-09

$ squeue | grep exp-2-09 | head -3
13777427 shared NGBW-JOB cipres R 5-06:14:20 1 exp-2-09
13816762 shared NGBW-JOB cipres R 5-03:14:11 1 exp-2-09
13816766 shared NGBW-JOB cipres R 5-03:13:39 1 exp-2-09

COMPLECS NSF award #2320934 17

Redirection and pipes
Stdin can also be redirected. This is sometimes useful if you want to prevent a
command from listing the name of the input file. In the example below we avoid
having to pipe output from wc (word count) into the cut utility.

$ wc -l [Link] # Output includes file name

9 [Link]

$ wc -l [Link] | cut -d' ' -f1 # Output contains just the line count
9

$ wc -l < [Link] # Output contains just the line count

9

COMPLECS NSF award #2320934 18

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 19

Wildcards and globbing
Wildcards let you express a pattern for matching multiple file names using any
combination of the following
• ? matches any single character
• * matches any string (including empty string)
• […] matches multiple characters or ranges of characters

Globbing is the operation that expands the pattern into a list of files. Globbing is
done automatically by many Linux utilities that operate on files such as ls, rm, mv,
cat, head, tail and file

COMPLECS NSF award #2320934 20

Wildcards and globbing using ‘*’
$ ls
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]

$ ls file* # file names beginning with 'file'

[Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link]

$ ls f* # file names beginning with 'f'

[Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link]

$ ls *ghi* # file names containing 'ghi'

[Link] [Link] [Link]

COMPLECS NSF award #2320934 21

Wildcards and globbing using ‘?’
$ ls
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]

$ ls file?.dat # 'file' + any character + '.dat'

[Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link]

$ ls a?cde.* # 'a' + any character + 'cde.' + any string

[Link] [Link]

$ ls ?????.?x? # Any 5 characters + '.' + any character + 'x' + any character

[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]

COMPLECS NSF award #2320934 22

Wildcards and globbing using ‘[…]’
$ ls
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]
[Link] [Link] [Link] [Link] [Link] [Link]

$ ls file[13].dat # multiple matches using list of integers

[Link] [Link]

$ ls file[1-3].dat # multiple matches using range of integers

[Link] [Link] [Link]

$ ls [ac]x*.* # multiple matches using list of letters

[Link] [Link]

$ ls [a-c]x* # multiple matches using range of letters

[Link] [Link] [Link]

COMPLECS NSF award #2320934 23

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 24

Aliases
Aliases are just shortcuts for Linux commands. They’re useful for keeping you out of
trouble (e.g., accidentally deleting all files) and abbreviating complex commands
# Staying out of trouble (e.g., ensure rm * won't delete everything)
$ alias rm='rm -i'
$ which rm
alias rm='rm -i'
/usr/bin/rm

# Abbreviating a long command (customized listing of Slurm partitions)

$ alias partitions='sinfo -o "%15P %6a %6D %11s %15F %7c %8m %10l %10L %10G"'
$ which partitions
alias partitions='sinfo -o "%15P %6a %6D %11s %15F %7c %8m %10l %10L %10G"'
/cm/shared/apps/slurm/current/bin/sinfo

COMPLECS NSF award #2320934 25

Aliases
Aliases can be listed by running alias without arguments
$ alias
alias partitions='sinfo -o "%15P %6a %6D %11s %15F %7c %8m %10l %10L %10G"'
alias reservations='sinfo -T'
alias rm='rm -i'
alias spacktivate='spack env activate'
alias vi='vim'
alias int1core='srun --partition=shared --pty --account=use300 --nodes=1
--ntasks-per-node=1 -t 01:00:00 --wait=0 --export=ALL /bin/bash'
alias int1node='srun --partition=compute --pty --account=use300 --nodes=1
--ntasks-per-node=128 -t 01:00:00 --wait=0 --export=ALL /bin/bash'

COMPLECS NSF award #2320934 26

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 27

Command history
The history command lists previously executed commands. You can also navigate your history
with the up/down arrow keys. There are special commands to execute previous commands:

# inspect command history # execute last command that

$ history # started with "ls"
-- snip -- $ !ls
428 ls file*.txt ls file*.txt
429 rm [Link] [Link] [Link] [Link]
430 echo "Hello"

# execute last command # execute a specific command

$ !! $ !428
echo "Hello" ls file*.txt
Hello [Link] [Link] [Link]

COMPLECS NSF award #2320934 28

Table of contents
• Introduction
• Symbolic links and hard links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 29

pushd and popd – navigating between directories
The pushd and popd commands are very useful for navigating between directories. pushd
adds directories to the stack, popd removes directories from stack and dirs lists the current
content of the stack
$ pushd dir1
~/ppdem/dir1 ~/ppdem • pushd with directory name
$ pushd ../dir2 adds to top of stack
~/ppdem/dir2 ~/ppdem/dir1 ~/ppdem • pushd with no args swaps
$ pushd
~/ppdem/dir1 ~/ppdem/dir2 ~/ppdem
top two directories
$ pushd +2 • pushd +n brings nth
~/ppdem ~/ppdem/dir1 ~/ppdem/dir2 directory to top
$ dirs • popd with no args pops top
~/ppdem ~/ppdem/dir1 ~/ppdem/dir2
$ popd
directory
~/ppdem/dir1 ~/ppdem/dir2 • popd +n pops nth directory
$ popd +1 • dirs lists directory stack
~/ppdem/dir1

COMPLECS NSF award #2320934 30

pushd and popd – a concrete example
While developing an application, we want to alternate between modifying and testing the
code. To keep our work organized, code development and execution are done in separate
directories. Using pushd and popd allows us to easily move between the two.

$ cd ~/code-dir/
$ [edit code and build executable] Enter code-dir, perform tasks and
$ pushd ~/run-dir then add run-dir to the stack
$ [run code and examine output]

$ dirs dirs command confirms that two

~/run-dir ~/code-dir directories are on the stack
$ pushd
$ [edit code and build executable] With code-dir and run-dir on the
$ pushd stack, can use pushd without
$ [run code and examine output] arguments to swap top two entries

COMPLECS NSF award #2320934 31

Table of contents
• Introduction
• Symbolic links and hard links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 32

Shell and environment variables
Shell and environment variables allow you to customize your environment and
control how applications behave on a Linux system.

• Shell variables are only known within the current shell

• Environment variables are known globally and are inherited by processes and
shells that are launched by the current shell. You'll generally want to use
environment rather than shell variables.

• Shell variables are set using the KEY=value[:value2:[:value3]…] syntax; space is

not allowed around the equal sign and by convention the KEY is capitalized
• Shell variables are made into environment variables using the export command
• Value of shell/environment variables accessed using $KEY

COMPLECS NSF award #2320934 33

Shell and environment variables
$ KEY1="abcd" # set shell variable KEY1
$ KEY2="efgh" # set shell variable KEY2
$ export KEY2 # make KEY2 an environment variable
$ export KEY3="ijkl" # set environment variable KEY3
$ echo $KEY1 # display KEY1
abcd #
$ echo $KEY2 # display KEY2
efgh #
$ echo $KEY3 # display KEY3
ijkl #

$ /bin/bash # launch a new shell

$ echo $KEY1 # display KEY1
# KEY1 was only known in the parent shell
$ echo $KEY2 # display KEY2
efgh # KEY2 was inherited
$ echo $KEY3 # display KEY3
ijkl # KEY3 was inherited

COMPLECS NSF award #2320934 34

Shell and environment variables
Shell and environment variables can also be set with the declare command, using
the -x flag to export and -r to make the variable read only (within the current shell)
$ declare VAR1="abc" # shell variable
$ declare -x VAR2="def" # environment variable
$ echo $VAR1
abc
$ echo $VAR2
def

$ bash
$ echo $VAR1
# value not known by child shell
$ echo $VAR2
def # value exported to child shell
$ echo $SHLVL # shell level
2

COMPLECS NSF award #2320934 35

Shell and environment variables
All environment variables can be displayed using the env command. Partial output
is shown below
$ env
PATH=/cm/shared/apps/sdsc/1.0/bin:/cm/shared/apps/sdsc/1.0/sbin:/cm/share
d/apps/slurm/current/sbin:/cm/shared/apps/slurm/current/bin:/home/sinkovi
t/spack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/sinko
vit/.local/bin:/home/sinkovit/bin:/home/sinkovit/anaconda3/bin:/opt/AMDuP
rof_3.3-462/bin
LIBRARY_PATH=/cm/shared/apps/slurm/current/lib64/slurm:/cm/shared/apps/sl
urm/current/lib64
MANPATH=/cm/shared/apps/slurm/current/man:/usr/share/lmod/lmod/share/man:
:/usr/local/share/man:/usr/share/man:/cm/local/apps/environment-modules/c
urrent/share/man
LOGNAME=sinkovit
SHELL=/bin/bash
HOME=/home/sinkovit

COMPLECS NSF award #2320934 36

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 37

Linux configuration files
Linux has a number of configuration files for customizing your environment:
(/etc/profile, /etc/bashrc, ~/.bash_profile, ~/.bashrc and others)
To make things even more confusing, there is a hierarchy of scripts, scripts can
execute (source) other scripts, and some scripts only apply to login shells.
Fortunately, you can usually get by with just two general rules

• Customize environment variables in ~/.bash_profile

• Put aliases and functions in ~/.bashrc

COMPLECS NSF award #2320934 38

Interactive login, interactive non-login and non-interactive shells
• Interactive shell is one that reads from and writes to the user’s terminal. If you
can enter commands at the command line and see output, you’re in an
interactive shell.
• Interactive login shell is launched using ssh, locally or when new shell is launched with the
--login option. This is where you’ll usually work.
• Interactive non-login shell launched from a login shell by executing bash at the command
line or opening new terminal from Linux desktop.

• Non-interactive shell is not associated with a terminal. Usually launched by a

user executing a script or associated with automated process.

COMPLECS NSF award #2320934 39

Order of execution
Interactive login shell Interactive non-login shell
/etc/profile.d/* ~/.bashrc /etc/bashrc
/etc/profile
/etc/bashrc
~/.bash_profile
~/.bashrc
* Legend
Executes
~/.bash_login system-wide config
user files
files
Depending on Linux flavor
and choice of shell, you
~/.profile You don’t need to worry about anything in the orange boxes.
might not use these
These are maintained by the sys admins.

* Although ~/.bashrc normally sources /etc/bashrc, the latter contains logic to ensure that its content is not executed twice
See [Link] for more details on order

COMPLECS NSF award #2320934 40

The bad news is …
If you do internet searches for Linux configuration files, you’ll probably
come across figures that look like this

COMPLECS NSF award #2320934 41

But the good news is …
If you’re using bash and you’re not a
sys admin, you probably just need to
worry about two files

~/.bash_profile

~/.bashrc

COMPLECS NSF award #2320934 42

Configuration files (~/.bash_profile)
Use ~/.bash_profile for commands that should be run only once, such as
customizing environment variables (e.g., $PATH)

# .bash_profile
# Get the aliases and functions This will usually be in your .bash_profile by
if [ -f ~/.bashrc ]; then default and sources your .bashrc file
. ~/.bashrc
Do not edit this part!
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin The first line will usually be present by
PATH=$PATH:$HOME/anaconda3/bin default, subsequent lines added to
PATH=$PATH:/opt/AMDuProf_3.3-462/bin customize environment
export PATH

COMPLECS NSF award #2320934 43

Configuration files (~/.bashrc)
Use ~/.bashrc for commands that should run for every new shell

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
This will usually be in your .bashrc by
. /etc/bashrc default and sources /etc/bashrc
fi Do not edit this part!

alias rm='rm -i'

alias cp='cp –I'
alias reservations='sinfo –T' Most often used for aliases and loading
modules that you will want in every shell
module load slurm
module load sdsc

COMPLECS NSF award #2320934 44

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 45

Pseudo filesystems
Following the Linux philosophy of "everything is a file", Linux provides pseudo filesystems
that allow you to access dynamically generated content using the same tools that you use
to access regular files. These are normally found in the /proc and /sys directories. As a
regular user, you'll normally use these to get hardware information.

$ cd /proc

$ ls -l cpuinfo uptime version

The ls command shows that files in the
-r--r--r-- 1 root root 0 Aug 2 10:28 cpuinfo
-r--r--r-- 1 root root 0 Oct 10 14:42 version
/proc directory are zero size …

$ cat cpuinfo | wc -c … but concatenating the contents and

99160
running through a character counter shows
$ cat version | wc -c
that they are not empty
186

COMPLECS NSF award #2320934 46

Pseudo filesystems – CPU information
lscpu command interrogates the contents of /proc/cpuinfo and organizes the results
Architecture: x86_64 NUMA node0 CPU(s): 0-15
CPU op-mode(s): 32-bit, 64-bit NUMA node1 CPU(s): 16-31
Byte Order: Little Endian NUMA node2 CPU(s): 32-47
CPU(s): 128 NUMA node3 CPU(s): 48-63
On-line CPU(s) list: 0-127 NUMA node4 CPU(s): 64-79
Thread(s) per core: 1 NUMA node5 CPU(s): 80-95
Core(s) per socket: 64 NUMA node6 CPU(s): 96-111
Socket(s): 2 NUMA node7 CPU(s): 112-127
NUMA node(s): 8 Flags: fpu vme de pse tsc msr pae mce
Vendor ID: AuthenticAMD cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx
CPU family: 23 fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb
rdtscp lm constant_tsc rep_good nopl xtopology
Model: 49
nonstop_tsc cpuid extd_apicid aperfmperf pni
Model name: AMD EPYC 7742 64-Core Processor pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2
Stepping: 0 x2apic movbe popcnt aes xsave avx f16c rdrand
CPU MHz: 3257.493 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
BogoMIPS: 4491.71 misalignsse 3dnowprefetch osvw ibs skinit wdt tce
Virtualization: AMD-V topoext perfctr_core perfctr_nb bpext perfctr_llc
mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev
L1d cache: 32K ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2
L1i cache: 32K cqm rdt_a rdseed adx smap clflushopt clwb sha_ni
L2 cache: 512K xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
L3 cache: 16384K cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr
wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale
vmcb_clean flushbyasid decodeas

COMPLECS NSF award #2320934 47

Pseudo filesystems – CPU information
lscpu command interrogates the contents of /proc/cpuinfo and organizes the results
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit Output shows we're running on an x86
Byte Order:
CPU(s):
Little Endian
128
architecture, with two 64-core AMD EPYC
On-line CPU(s) list: 0-127 7742 processors* and hardware
Thread(s) per core: 1 multithreading turned off. CPUs are
Core(s) per socket: 64
Socket(s): 2 organized into 8 NUMA nodes
NUMA node(s):
Vendor ID:
8
AuthenticAMD
We also see capacities of the L1 (data and
CPU family: 23 instruction), L2 and L3 caches.
Model: 49
Model name: AMD EPYC 7742 64-Core Processor
Stepping: 0
CPU MHz: 3257.493
BogoMIPS: 4491.71
Virtualization: AMD-V
L1d cache: 32K
L1i cache: 32K
* Google search on "AMD EPYC 7742 code
L2 cache: 512K name" shows that this is AMD Rome processor
L3 cache: 16384K

COMPLECS NSF award #2320934 48

Pseudo filesystems – CPU information
lscpu command interrogates the contents of /proc/cpuinfo and organizes the results
NUMA node0 CPU(s): 0-15
NUMA node1 CPU(s): 16-31 The remainder of the output shows how
NUMA node2 CPU(s):
NUMA node3 CPU(s):
32-47
48-63
CPUs are organized into NUMA nodes,
NUMA node4 CPU(s): 64-79 which has implication for performance of
NUMA node5 CPU(s): 80-95 shared-memory parallelism.
NUMA node6 CPU(s): 96-111
NUMA node7 CPU(s): 112-127 The output also contains many low-level
Flags: fpu vme de pse tsc msr pae mce
cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx
processor flags. You'll be most interested in
fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb the avx flags (avx, avx2, avx512), which
rdtscp lm constant_tsc rep_good nopl xtopology
nonstop_tsc cpuid extd_apicid aperfmperf pni indicate the availability of instructions that
pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2
x2apic movbe popcnt aes xsave avx f16c rdrand
can greatly improve the performance of
lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a some numerically intensive applications.
misalignsse 3dnowprefetch osvw ibs skinit wdt tce
topoext perfctr_core perfctr_nb bpext perfctr_llc
mwaitx cpb cat_l3 cdp_l3 hw_pstate sme ssbd mba sev
ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2
cqm rdt_a rdseed adx smap clflushopt clwb sha_ni
xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc
cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr
wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale
vmcb_clean flushbyasid decodeas

COMPLECS NSF award #2320934 49

Getting memory information: /proc/meminfo
/proc/meminfo pseudo-file lists key memory specs. More information than you probably need, but at
least one bit of useful data

Expanse regular memory node Expanse large Memory Node

MemTotal: 263698228 kB (total physical memory) MemTotal: 2113365024 kB
MemFree: 251035032 kB MemFree: 2003813192 kB
MemAvailable: 250623760 kB MemAvailable: 2003837576 kB
Buffers: 12824 kB Buffers: 1816060 kB
Cached: 3765288 kB
Cached: 3126364 kB
SwapCached: 19588 kB
SwapCached: 0 kB
Active: 18699308 kB
Active: 1301564 kB (good approx used memory) ...
Inactive: 2990668 kB
Active(anon): 1240284 kB
Inactive(anon): 2890076 kB
Active(file): 61280 kB Expanse GPU node
Inactive(file): 100592 kB MemTotal: 394817856 kB
Unevictable: 0 kB MemFree: 390316552 kB
Mlocked: 0 kB MemAvailable: 389003936 kB
SwapTotal: 0 kB Buffers: 8244 kB
SwapFree: 0 kB Cached: 1028748 kB
SwapCached: 0 kB
Dirty: 32 kB
Active: 176572 kB
Writeback: 0 kB
...
AnonPages: 1151660 kB

COMPLECS NSF award #2320934 50

Table of contents
• Introduction
• Symbolic links
• Redirection and pipes
• Wildcards and globbing
• Aliases
• Command history
• Navigating between directories with pushd/popd
• Environment variables
• Configuration files (~/.bashrc, ~/.bash_profile, /etc/bashrc, …)
• Pseudo-filesystems
• Linux quotes

COMPLECS NSF award #2320934 51

Linux quotes – single and double
Linux has two types of quotes
• 'Single quotes' preserve the literal value inside the quotes
• "Double quotes" recognize special meaning of $, ', \ and ! (if history expansion enabled)

$ export VAR1="abc"
$ export VAR2="def"
$ echo '$VAR1 + $VAR2'
$VAR1 + $VAR2
$ echo "$VAR1 + $VAR2"
abc + def

COMPLECS NSF award #2320934 52

Linux quotes – single and double
If your strings don't require any special characters, you can use the single and double
quotes interchangeably. This is handy if your strings have quotes. You can also use the
backslash to escape characters so that they don't have their special meaning.

$ echo "Good morning 'Bob'"

Good morning 'Bob'
$ echo 'Good morning "Bob"'
Good morning "Bob"
$ echo "Good morning \"Bob\""
Good morning "Bob"
$ echo "$VAR1 + \\\$VAR2\\"
abc + \$VAR2\

COMPLECS NSF award #2320934 53

Linux quotes – backticks
Backticks are distinct from single/double quotes and capture the output of a command

$ ME=`who am i`
$ DATE=`date`
$ echo $ME
sinkovit pts/417 2024-10-10 09:36 ([Link])
$ echo $DATE
Thu Oct 10 10:33:04 PDT 2024

COMPLECS NSF award #2320934 54

Summary
Learning intermediate Linux skills will help you to become a more effective user
of advanced cyberinfrastructure. This tutorial provides you with the tools to
customize your environment, automate tasks and construct simple workflows.
While we covered most of what you should need to know, there are many
resources on a wide range of topics in case you need to go deeper

• Advanced bash scripting: [Link]

• Forums and tutorials: [Link]

COMPLECS NSF award #2320934 55

Supplementary material
• Filesystem Hierarchy Standard
• Hard links and symbolic links

COMPLECS NSF award #2320934 56

Filesystem Hierarchy Standard (FHS)
The FHS describes the conventional layout of a Linux system. Although there are
some variations among different Linux distributions (distros), they tend to be minor
and mostly related to the /lib, /bin, /usr/lib and /usr/bin directories.
As a regular user (i.e., not a sys admin) you don’t need to know too much about the
FHS. We’re covering it here to round out your knowledge and make the Linux
environment a little less mysterious.
• Overviews
• [Link]
• [Link]
• Deep dives
• [Link]
• [Link]

COMPLECS NSF award #2320934 57

Filesystem Hierarchy Standard (FHS)
• /home contains user home directories
• /bin and /usr/bin traditionally contain essential and non-essential command
binaries, respectively, but these are often merged. For example, on CentOS Linux
/bin is a symbolic link to /usr/bin
• /sbin and /usr/sbin contain binaries for commands needed mostly by sys admins.
Like /bin and /usr/bin, these are often merged
• /lib and /usr/lib contain libraries needed by the Linux commands (often merged)
• /opt is where software was traditionally installed, but with the rising popularity
of package managers like Spack, this directory is not used much anymore

COMPLECS NSF award #2320934 58

Filesystem Hierarchy Standard (FHS)
• /etc contains configuration files for Linux shells, schedulers, networking,
accounts, package managers and others
• /root home directory for the root user
• /var log files
• /dev device files for terminals, disks, networking, etc.
• /proc and /sys are pseudofile systems that contain information about running
processes and hardware
• /tmp stores temporary files

COMPLECS NSF award #2320934 59

Not everything fits into the FHS
Especially in high-performance computing environments, not everything fits into the
FHS. This can include additional file systems and software installed using package
managers. For example, on SDSC’s Expanse resource

• Lustre parallel file systems live under /expanse/lustre/projects and

/expanse/lustre/scratch
• Qumulo (hybrid cloud storage) is mounted at /expanse/projects
• Software installed using the Spack package manager are found under
/cm/shared/apps/spack

COMPLECS NSF award #2320934 60

Supplementary material
• Filesystem Hierarchy Standard
• Hard links, symbolic links and inodes

COMPLECS NSF award #2320934 61

Hard links and symbolic links
Symbolic links and hard links give you a way to refer to a file or an inode by a
different name. They’re useful when you want to avoid writing out a full or relative
path name every time or if you want to have a generic executable name refer to a
specific version.
We’ll discuss the difference between symbolic links (also known as soft links) and
hard links in the next few slides along with some background on inodes. Both types
of links are created with the ln command

$ ln -s file1 soft-file1 # Creating a soft link

$ ln file2 hard-file2 # Creating a hard link

COMPLECS NSF award #2320934 62

inodes
An inode is a data structure that stores information (metadata) about a filesystem
object, such as a regular file or a directory. It contains:
• Permissions
• File size
• Ownership
• Timestamps
• Location of data, but not the actual data.

A file is a link to an inode, and a hard link is just another file that links to the same
inode. An inode is only deleted when all the links to it have been deleted. If the
original file is deleted, the data can still be accessed using the hard link.
A symbolic link is another name for the file. If the original file is deleted or moved,
the symbolic link still exists but can no longer be used to access the data.

COMPLECS NSF award #2320934 63

Symbolic link and hard link examples
$ ls
file1 file2
$ ln file1 file1-hard
$ ln -s file2 file2-soft

$ ls -i1 # -i option lists inode

157076717 file1
157076717 file1-hard
157076716 file2
157076768 file2-soft@

$ rm file1 file2
$ cat file1-hard
Contents of file1
$ cat file2-soft
cat: file2-soft: No such file or directory

COMPLECS NSF award #2320934 64

Symbolic links and hard links – which should I choose?
You’ll generally work with symbolic links since they let you span file systems and
can be created even if the file being linked to does not yet exist.

$ ln file3 file3-hard
ln: file3: No such file or directory
$ ln -s file3 file3-soft

$ ln /expanse/lustre/scratch/sinkovit/file4 file4-hard
ln: failed to create hard link 'file4-hard' =>
'/expanse/lustre/scratch/sinkovit/file4': Invalid cross-device link
$ ln -s /expanse/lustre/scratch/sinkovit/file4 file4-soft

$ ls
file1 file1-hard file2 file2-soft file3-soft file4-soft

COMPLECS NSF award #2320934 65

So, when will I want to use a hard link?
The short answer is not very often other than some edge cases

• Multiple users want to work with a file, which should be deleted only after each
member of the team has deleted their hard link. This allows the file to exist until
the last hard link is removed, without the original file having any special status.
Contrast with using soft links, where the owner of the original file would have to
be notified after all soft links had been removed.
• Provide flexibility for files to belong to multiple categories without having to
keep track of which copy is the original file.

COMPLECS NSF award #2320934 66