Study Notes: UNIX

1. Introduction to UNIX

UNIX Operating System Overview: UNIX is a powerful multi-user, multi-tasking operating system
originally developed at Bell Labs in 1969. It was designed to be portable (able to run on different
hardware), stable, and flexible. Multiple users can work simultaneously on a UNIX system (multi-
user), and the system can perform many tasks at once (multitasking). From its inception, UNIX
focused on simplicity and reusability: complex tasks can be performed by combining simple tools.
The design philosophy emphasizes treating many system components (like files, devices, etc.) in a
uniform way.

UNIX Architecture – Kernel and Shell: At the heart of UNIX is the kernel, the core of the OS that
manages hardware resources, memory, processes, file system, and system calls. The kernel runs with
high privileges and provides services to user programs. Surrounding the kernel is the shell, which is a
command-line interpreter that acts as the interface between the user and the kernel. Users input
commands into the shell, and the shell either executes those commands itself (if built-in) or requests
the kernel to launch external programs. This architecture can be visualized in layers: user applications
interact with the shell (or other system libraries), the shell communicates with the kernel (often via
system calls), and the kernel interacts directly with hardware. Figure 1 below illustrates a simplified
UNIX operating system structure, showing how user-level commands and shells sit above the system
call interface to the kernel, which in turn talks to the hardware.

Figure 1: Simplified UNIX architecture, showing the relationship between user applications, the shell,
system call interface, and kernel components (managing I/O, file system, CPU scheduling, etc.).

Files and Processes in UNIX: In UNIX, everything is treated as a file. Regular data files, directories,
hardware devices, and inter-process communication channels are all represented as files or pseudo-
files in the file system hierarchy. This uniform approach simplifies operations—programs use the
same read/write interface for many resources. A process in UNIX is an instance of a running program.
The kernel manages processes by providing multitasking (multiple processes running concurrently)
and isolation between them. Each process has a unique process ID (PID) and its own memory space.
Processes can spawn child processes (each new process is created by an existing one, forming a
parent-child hierarchy). The kernel’s process scheduler manages CPU time among processes to
achieve multitasking. UNIX was designed as a multiuser system, meaning multiple users’ processes
can run concurrently without interfering, with the kernel enforcing security and resource limits for
each user.

System Calls: Application programs do not access hardware or core OS functions directly; instead
they use system calls to request services from the kernel. A system call is a controlled entry point
into the kernel, allowing a program to perform privileged operations (such as reading a file, creating
a process, or allocating memory) by asking the kernel to do it on its behalf. In code, a system call
looks like a function call (for example, functions like open(), read(), write(), fork(), etc., are system
calls in UNIX). When a program makes a system call, the CPU transitions from user mode to kernel
mode to execute the request, then returns back to user mode when finished. This mechanism
provides a safe interface between user programs and hardware resources, as the kernel checks
permissions and ensures stability. Common system call categories include process control (e.g., fork,
exec), file and directory operations (open, read, write, close), device I/O, and others.

Features of UNIX: UNIX systems have several hallmark features and design principles:

 Multi-tasking and Multi-user: The OS allows multiple processes to run concurrently

(preemptive multitasking) and supports multiple users logged in at the same time, with
isolation and resource control for each user. Users can have their own environments and
processes running simultaneously on one machine.

 Hierarchical File System: UNIX organizes data into a single inverted-tree file system with a
root (/) directory at the base. Everything (files, directories, devices) exists under this unified
file hierarchy. This makes file paths consistent and allows mounting devices or filesystems in
the tree seamlessly.

 Everything is a File: With few exceptions, many resources (devices, pipes, etc.) are exposed
as files in the file system. Programs use common I/O calls to interact with devices or inter-
process communication channels as though they were files. (This principle is sometimes
summarized as “everything is a file.”)

 Portable Programming Interface (POSIX): UNIX provides a standard programming API for
software. The POSIX standards (explained below) define the system calls, C library functions,
shells, and utilities that UNIX systems should support, making it easier to port programs
between UNIX variants.

 Built-in Networking: Networking capabilities (especially TCP/IP) are integrated into UNIX.
Networking interfaces are often accessed via file descriptors (sockets) analogous to files.
Daemon processes provide network services, and tools for networking (ssh, ftp, etc.) are part
of the system.

 Stability and Security: UNIX was designed for long-running systems (servers) with an
emphasis on stability. It employs permission bits on files and processes to enforce security
(more on permissions later). System services often run as daemons (background processes)
managed by the init system.

 Composability: Small programs that do one thing well can be combined (via shell scripting or
pipelines) to perform complex tasks – this is the “UNIX philosophy.” Many core utilities can
be chained together with pipelines for powerful data processing.

POSIX and the Single UNIX Specification: Over time, many variants of UNIX appeared (from different
vendors and the open source community). To maintain compatibility, standards were developed.
POSIX (Portable Operating System Interface) is a family of IEEE standards that define the core
interfaces for UNIX-like operating systems. POSIX specifies the APIs (system calls and C library
functions), command-line shells, and utility command behaviors that compliant systems should have,
to ensure software portability. In essence, if an operating system is POSIX-compliant, programs
written for one POSIX system can compile and run on another with minimal changes, as long as they
use the standard interfaces. The Single UNIX Specification (SUS) is a standard maintained by The
Open Group (which owns the UNIX® trademark) that extends POSIX; it defines what an operating
system must support to be considered a "UNIX" system. In the 1990s, SUS was created as a superset
of POSIX – it included POSIX standards and additional requirements for systems to be UNIX-certified.
Essentially, SUS is the official definition of the UNIX operating system, and operating systems that
pass The Open Group’s certification (and meet SUS requirements) are allowed to use the UNIX®
name. In practice, SUS incorporates POSIX APIs and utilities, plus some X/Open specifications and
extensions for full UNIX functionality. (Many modern systems like Linux are POSIX-compliant but not
certified “UNIX” – only systems like IBM AIX, Oracle Solaris, HP-UX, etc., which go through
certification, carry the UNIX® trademark.)

Internal vs External Commands: When using the UNIX shell, commands can be classified as internal
(built-in) commands or external commands. Internal commands are built into the shell itself. When
you run an internal command, the shell executes it directly within the current shell process.
Examples include cd (change directory), pwd (print working directory), echo (in many shells), jobs,
alias, and others – these do not exist as separate programs on disk (even if a program by the same
name exists, the shell might bypass it). External commands are separate executable files stored in
the file system (often under directories like /bin or /usr/bin). When you run an external command,
the shell locates the program file (using the PATH environment variable) and spawns a new process
to run it. For example, ls, cp, grep, etc., are external commands (each is an executable in /bin or
/usr/bin). The key difference is that internal commands run within the shell (faster, no new process
needed) while external commands require forking a new process. Many shells provide a type or
which command to check if a given command is built-in or external. For instance, in bash, type cd will
show "cd is a shell builtin" while type ls might show "ls is /bin/ls", indicating ls is external. One
implication of this is that internal commands can affect the shell’s state (e.g., cd changes the current
directory of the shell itself), whereas external commands cannot directly change the shell’s internal
state (they run in a separate process). In summary, internal commands are part of the shell
interpreter, and external commands are standalone programs on disk.

2. Utilities of UNIX

UNIX comes with a rich set of utility programs. Below are some common user-level utilities along
with their usage and sample outputs:

 cal – Prints a calendar. By default, cal displays a calendar of the current month. You can also
supply a month and year (e.g., cal 5 2025 for May 2025).
Example:

$ cal # calendar of current month

May 2025

Su Mo Tu We Th Fr Sa

1 2 3 4
 5 6 7 8 9 10 11

12 13 14 15 16 17 18

19 20 21 22 23 24 25

26 27 28 29 30 31

This output shows the days of May 2025 aligned by weeks.

 date – Displays the current date and time. With no arguments it shows the system’s
date/time in a standard format. (It can also be used to set the date, with proper privileges, or
format the output with options.)
Example:

$ date

Tue May 20 [Link] IST 2025

This indicates the day of week, month, day, time (hh:mm:ss), timezone, and year.

 echo – Prints its arguments to standard output. Commonly used to display messages or the
value of shell variables. (Most shells have echo as a built-in command, but there is also an
external /bin/echo.)
Example:

$ echo "Hello, world!"

Hello, world!

echo simply repeats the provided string to the output. It also recognizes certain escape sequences
(like \n for newline) and options like -n (to omit the trailing newline) in some versions.

 bc – An arbitrary-precision calculator language. Typically used as an interactive calculator or

to process mathematical expressions. You can enter calculations and bc will output the
results. It supports decimals, base conversion, and many mathematical functions.
Example:

$ bc

bc 1.07 (basic calculator)

Enter an expression: 5 + 3*2

 11

Enter an expression: scale=2 # set decimal precision to 2

Enter an expression: 5/3

1.66

In this example, bc was started (it opens an interactive prompt). We performed a couple of
calculations, including setting scale for decimal precision. bc can also be used non-interactively by
echoing an expression into it or by using here-documents.

 passwd – Used to change a user’s password. When you run passwd, it will ask for the current
password (for verification) and then prompt for a new password (twice for confirmation).
Only the owner of an account (or the superuser) can change that account’s password. For
the superuser, passwd <username> can change another user’s password. (Output not shown
here for security reasons, since it doesn’t display the password as you type.) Typically, it will
respond with prompts like “Enter new UNIX password:” and “Retype new UNIX password:”
and then “password updated successfully” if all goes well.

 who – Shows who is currently logged into the system. It lists active user sessions. The output
typically includes the username, terminal (tty) they’re on, login date/time, and possibly the
host if remote.
Example:

$ who

alice tty1 May 20 09:05

bob pts/0 May 20 09:07 ([Link])

Here, user alice is logged in on the console (tty1) since 09:05, and user bob is logged in via a network
(shown by pts/0 with a remote IP in parentheses). The who command is useful for seeing current
users and their terminals. (If no one else is logged in, who may return no output.)

 uname – Prints information about the system (UNIX name). By itself, uname often prints the
OS name (e.g., “Linux” or “Darwin” or “SunOS”). With options it can provide more detail: -r
for kernel release, -m for machine hardware name, -n for network node (hostname), -a for all
information.
Example:

$ uname

Linux

$ uname -a
 Linux myhost 5.15.0-50-generic #56-Ubuntu SMP Fri Sep 23 [Link] UTC 2024 x86_64
GNU/Linux

In the second line, uname -a outputs: kernel name, hostname, kernel version and build date,
architecture (x86_64), and OS type. This gives a complete identification of the system.

 tty – Prints the file name of the terminal connected to standard input. Essentially, it tells you
which terminal device you are using. If the standard input is a terminal, it will output
something like /dev/tty2 or /dev/pts/0. If not (for example, if you pipe something into tty or
redirect it), it may respond with “not a tty”.
Example:

$ tty

/dev/pts/0

This indicates the current session is on the pseudo-terminal device 0 (commonly an SSH or terminal
emulator session). If you run tty inside a script with no terminal, it might output “not a tty”.

These basic utilities are handy for day-to-day tasks and scripts. Many more standard utilities exist
(like text processing tools, compilers, etc.), which we will encounter in later sections (especially in the
Filters section).

3. UNIX File System

File System Structure: The UNIX file system is organized as a single hierarchical directory tree. At the
very top is the root directory named / (a lone forward slash). All files and directories on the system
are descendants of this root. The structure is analogous to an inverted tree: / is the trunk at the base,
and everything else branches off from it. Directories (often called folders in other systems) can
contain files or other subdirectories, and those subdirectories can contain further files or
subdirectories, and so on, creating multiple levels. There is only one root /, which is the starting point
for absolute pathnames (more on that below). A few key points about the structure: all disks,
devices, and remote file systems are integrated into this single tree (for example, a USB drive or
another partition is “mounted” onto a directory within the tree). This unified structure means that
regardless of which physical device a file resides on, it can be accessed via a single coherent directory
hierarchy. Users each have their own home directory (typically under /home or a similar location),
and system files are organized under standard directories.

Figure 2 shows an overview of a standard Unix-style filesystem hierarchy with some common top-
level directories and their typical contents.
 Figure 2: Standard UNIX filesystem hierarchy (some common directories). The diagram illustrates the
root / at top, with major system directories like /bin, /etc, /sbin, /usr, /var, /dev, /home, /lib, etc.
Under each, some typical contents or subdirectories are indicated (for example, /bin holds essential
binaries like ls, cp, sh; /etc holds configuration files; /usr/share/man holds manual pages; etc.).

As shown above, the top-level directories each have a specific purpose. Some important directories
in a UNIX system include:

 / (root): The root of the file system. It contains all other files and directories. Only the root
user (administrator) has full access to all items here; regular users have limited access
outside their home directories.

 /bin: “Binaries” – contains essential user command binaries that are needed in single-user
mode or during system boot. Typical commands like cat, ls, cp, mv, echo, shells (sh, bash),
etc., reside here. These are programs generally usable by all users.

 /sbin: “System binaries” – similar to /bin but for essential system administration commands
(usually only used by the superuser). This includes programs needed for booting or repairing
the system, e.g. init, fsck, ifconfig, reboot, shutdown, etc. Many of these require root
privileges to run.

 /usr: Contains userland software and data that is not critical for boot (in historical UNIX, /usr
was where user home directories lived, but now it mainly stores secondary system
resources). Often split into subdirectories:

o /usr/bin: Most user command binaries that are not in /bin – a large collection of
applications and utilities. For example, editors, compilers, network tools, etc.

o /usr/sbin: Non-essential system binaries (administrative commands for system

operation not needed at boot). This might include network service daemons,
advanced admin tools, etc.

o /usr/lib: Libraries for programs in /usr/bin and /usr/sbin. (Also, kernel modules
might reside in /lib or /usr/lib depending on system.)

o /usr/include: Header files for development (C/C++ header files, etc.) – basically, files
under /usr/include are used when compiling programs (the <...>.h files).

o /usr/share: Architecture-independent shareable data. For example, /usr/share/man

(manual pages documentation) and /usr/share/doc for other docs, /usr/share/fonts,
etc. Man pages (sections 1-8) are typically stored under /usr/share/man/man1,
/man2, etc., categorized by section.

o /usr/local: This is often empty by default; it's meant for user-compiled or locally
installed software. It mirrors the structure of /usr (you might have /usr/local/bin,
/usr/local/lib, etc.). Installing software here keeps it separate from the OS-managed
software.

 /etc: Host-specific system configuration files. Almost all system-wide configuration (for
services, networking, users, etc.) lives in /etc. For example, /etc/passwd (user accounts),
 /etc/ssh/sshd_config (SSH daemon config), /etc/fstab (file systems table), and init/service
configuration scripts.

 /dev: “Devices” – contains special files that represent hardware or virtual devices. UNIX
treats devices as files so that reading/writing to these special files communicates with the
device driver. For example, /dev/sda might represent a hard disk, /dev/tty terminals,
/dev/null a null sink, /dev/pts/0 a pseudoterminal, etc. These aren’t real files on disk; the
kernel populates the /dev namespace (often via devfs or udev).

 **/lib and /lib64 (or /usr/lib): System libraries for the essential binaries in /bin and /sbin.
These include shared libraries (.so files) needed by programs at boot. On some systems, 32-
bit vs 64-bit libraries are separated (e.g., 64-bit libs in /lib64). Additionally, kernel modules
(device drivers that can be loaded/unloaded) might reside in /lib/modules.

 /home: User home directories. Each regular user typically gets a directory
/home/<username> for personal files. On some UNIX systems, the equivalent might be
/users or even under /usr/home, but /home is standard on Linux. Users have full control over
their home directory.

 /root: The home directory for the root user (administrator). Often this is /root (a directory
named "root" under the root filesystem) rather than under /home, so that it’s available even
if /home is on a separate partition that isn’t mounted.

 /tmp: Temporary files. Applications and the OS use /tmp for transient files that don’t need to
persist across reboots. Many systems automatically purge /tmp at boot or after some time.
It’s world-writable (any user can create files there, though with restrictions so users cannot
interfere with each other’s files).

 /var: “Variable” files – this directory holds files that tend to grow or change frequently
(variable in size or content). Examples: log files (/var/log), mail spool files (/var/mail), print
spool files (/var/spool), lock files (/var/lock), caches (/var/cache), and other runtime data.
Web server files (for a website) or databases might also reside under /var (e.g., /var/www
or /var/lib/mysql). Essentially, /var is for data that is dynamic, unlike /usr which is mostly
static after installation.

 Other notable directories:

o /media or /mnt: Standard mount points for removable media or temporary mounts.
For example, inserting a USB drive might be auto-mounted under
/media/<label>. /mnt is often used by sysadmins as a generic mount point for
temporarily mounting filesystems.

o /opt: For “optional” or third-party software packages. Large software suites or

commercial software might install to /opt/<ProductName>. It’s a way to keep add-on
software separate from the OS files.

o /proc: A virtual filesystem (procfs) that presents process and kernel information as
files. Under /proc you’ll find one directory per process (named by PID), containing
info like memory usage, open files, etc., as well as system info like /proc/cpuinfo,
/proc/meminfo. These files are generated on the fly by the kernel; they don’t
consume actual disk space.
 o /run: A newer directory (in many Linux systems) for runtime variable data (like
process PID files, service state, etc.) that doesn’t persist across reboots (often
mounted as a tmpfs in memory). This is similar to older /var/run.

o /etc/…, /usr/share/man, etc.: We already covered /etc. /usr/share/man holds

manual pages (organized by section directories like man1 for user commands, man2
for system calls, etc.). These can be accessed using the man command (e.g., man ls
shows the manual for the ls command). The manual pages directory is typically
under /usr/share/man as defined by the filesystem standard (with subdirs man1,
man2, etc. for each section).

In summary, the UNIX filesystem provides a coherent structure where each directory has a well-
defined purpose, making it easier to locate files. Knowing these standard directories is essential for
system navigation and administration.

Types of Files: UNIX supports several types of files within this filesystem. The ls -l command (long
listing) uses the first character of each line to indicate the file type:

 Ordinary (Regular) Files: These are the common files that contain data, text, program code,
etc. An ordinary file is not a directory or special file – it’s essentially a stream of bytes.
Examples: documents, images, executables. Regular files cannot contain other files (they are
leaves in the directory tree). In long listings, ordinary files are indicated by a leading - (dash)
character. For instance, -rw-r--r-- at the start of an ls -l line means a regular file with certain
permissions.

 Directory Files: A directory is a special file that acts as a container for other files (including
subdirectories). It is essentially a list of entries, each mapping a filename to an inode (which
is the underlying record for a file). Users often think of directories as folders. In the file
system tree, directories form the branches. A directory entry for each file includes the file
name and a reference (inode number). Directories allow the hierarchical organization of files.
In ls -l output, directories are indicated by a leading d. For example, drwxr-xr-x would denote
a directory with certain permissions.

 Special Files (Device Files): These represent hardware devices or pseudo-devices. They
appear in the filesystem (usually under /dev) but actually refer to drivers. There are two
primary kinds:

o Character special files: Provide unbuffered, character-by-character access to devices.

For example, /dev/ttyS0 (serial port) or /dev/ptmx (pseudo-terminal master). In long
listing, these are marked with a leading c.

o Block special files: Provide buffered access in fixed-size blocks – typically disk drives.
E.g., /dev/sda (first SCSI/SATA disk) is a block device. Marked with a leading b in ls -l.
These files allow programs to interact with hardware using standard read/write calls.
For instance, writing to /dev/audio could send sound data to the speakers (if
permissions allow).

 Pipes (FIFO special files): A named pipe (FIFO) is a file that provides a unidirectional inter-
process communication channel. One process can write into the pipe and another can read
 from it, allowing data to flow between them. Named pipes are created with the mkfifo
command. In ls -l, they are indicated by a leading p. (There are also unnamed pipes created
by the shell with the | operator, but those don’t have a filesystem entry; “named pipes” do
appear as files.)

 Sockets: A UNIX domain socket is a special file similar to a pipe, but it allows bidirectional
communication between processes and supports datagram (message) semantics. Sockets (in
the filesystem, created by some processes or by socket() system call and binding to a
pathname) appear with a leading s in ls -l. They are often found in /tmp or other places (for
example, X11 uses a socket file like /tmp/.X11-unix/X0). Sockets facilitate inter-process
communication on the same system (or even between systems if used as network sockets,
though those are not stored as files).

 Symbolic Links: A symlink is a special file that acts as a reference or shortcut to another file
or directory. It contains a textual pointer (the path) to the target file. In ls -l, a symlink is
indicated by a leading l (lowercase L), and the listing shows the link -> target. For example:
lrwxrwxrwx 1 user group 5 May 20 10:00 mylink -> [Link]. Opening a symlink effectively
accesses the target it points to. Symlinks can span across filesystems and can point to
directories or files. If the target is moved or deleted, the symlink becomes a broken link
(dangling pointer).

File Naming Conventions: Filenames in UNIX can be up to 255 characters long and can include
letters, numbers, dots, underscores, and many special characters. However, there are some
conventions and rules:

 Avoid using the forward slash / in filenames, as / is the directory separator (it’s not allowed
as part of a name).

 Avoid using the null character \0 (string terminator) in filenames – that’s also not allowed.

 Filenames are case-sensitive. For example, [Link] and [Link] are different names.

 Hidden files: By convention, any file or directory name beginning with a dot . is considered
hidden. ls will not show these unless you use ls -a. Typical hidden files are configuration files
in a user’s home (like ~/.bashrc).

 It’s wise to avoid whitespace or control characters in filenames (while allowed, they can
complicate command-line handling). Most UNIX tools can handle spaces if properly quoted
or escaped, but it’s conventional to use hyphens or underscores instead of spaces.

 There are no mandatory extensions to filenames (unlike some systems that rely on
extensions). Extensions (like .txt, .c, .sh) are purely for human/readability or by convention
for certain programs to identify file type. The system identifies executables via permission
bits and magic numbers, not by .exe extension, etc.

Parent-Child Directory Relationship: In the directory hierarchy, every directory (except the root) has
a parent directory and can have child directories. The parent of a directory is the directory one level
above it in the tree. For example, if you have a path /home/user/docs, the parent of docs is user, and
the parent of user is home, and the parent of home is / (root). This parent-child relationship forms
the tree structure. Directories themselves are special files that contain entries — each entry is a
filename plus an inode number reference. One of those entries is .. which points to the parent
directory (and . which points to itself, see next). The root directory / is unique in that its parent is
itself (i.e., .. in root points to / again). Understanding parent-child relationships is crucial for
navigating the file system and for relative paths.

Meaning of . and ..: These are special directory entries present in every directory:

 . (single dot) refers to the current directory itself.

 .. (double dot) refers to the parent directory.

They can be used in paths. For instance, if you are in /home/user/docs and you type cd ..,
you will move to /home/user (the parent). If you type cd ., you stay in the same directory (no
change, as it refers to current directory). These are especially useful when writing relative
pathnames or when traversing directories in scripts. They also show up if you do ls -a,
because -a (all) lists these special entries. Essentially, . and .. are part of the mechanism by
which the file system keeps track of the tree structure.

Absolute vs Relative Pathname: A pathname is the route to a file or directory in the directory tree.
There are two types:

 Absolute path: This is a full path from the root /. It always starts with /. For example:
/home/user/docs/[Link] is an absolute path to the file [Link]. No matter what your
current directory is, an absolute path uniquely identifies the location of a file in the tree.

 Relative path: This is a path relative to the current working directory. It does not start with /.
For example, if your current directory is /home/user, a relative path docs/[Link] would
refer to /home/user/docs/[Link]. Relative paths rely on your context (the PWD – present
working directory). Using . and .. is part of constructing relative paths (e.g., ../[Link]
means “go to the parent directory, then find [Link] there”).
In practice, absolute paths are safer for scripts when you need a specific file (since they don't
depend on where you run the script from), but relative paths are convenient for interactive
use and for code that operates on local directories. The shell’s PWD environment variable
always contains the absolute path of the current directory, and commands like pwd will print
that.

HOME Directory and HOME Variable: Each user has a home directory, which is the default directory
they “start in” upon login. For example, user alice might have home /home/alice. This directory is
meant for the user to store personal files, configuration, etc. The environment variable HOME (all
caps) is set by the login system to the path of the user’s home directory. So for alice,
HOME=/home/alice. This variable is used by many programs as the default location for saving files or
looking for user-specific config. In the shell, you can refer to your home directory with ~ (tilde)
shorthand. For example, cd ~ will take you to your home directory (because the shell expands ~ to
the value of HOME). Each user’s HOME is different (root’s home is usually /root, others under
/home). If you need to reference another user’s home, ~username can often be used (shell
expansion). The HOME variable is also used by commands like cd with no arguments (typing cd alone
will by default go to HOME). It’s important to note that HOME is just a convenient variable; changing
it doesn’t move your actual home directory, it just changes what programs think is your home.
Usually, it’s set correctly by the system and left unchanged.

Common File Management Commands (pwd, cd, mkdir, rmdir, ls):

 pwd – Print Working Directory. This command simply outputs the absolute path of the
current directory you are in. For example, if you are in /home/user/docs, pwd will print
/home/user/docs. This is useful if you’re not sure where you are (especially after using cd a
few times in a session) or if you need the full path for use in a script.

 cd – Change Directory. Usage: cd <directory>. This moves your current working directory to
the specified one. You can specify an absolute path (cd /var/log) or a relative path (cd ../.. to
go up two levels, for instance). As noted, cd with no argument will take you to your home
directory (because it uses HOME). cd - (hyphen) will take you back to the previous directory
you were in, toggling between two locations. cd .. goes to parent, cd . stays in place (though .
is redundant). If the directory change is successful, your shell’s PWD updates to the new
location. If unsuccessful (e.g., directory doesn’t exist or no permission), you’ll get an error.

 mkdir – Make Directory. Creates a new directory. Usage: mkdir <dirname>. For example,
mkdir newfolder creates a subdirectory named “newfolder” in the current directory. You can
also specify a path, e.g., mkdir /tmp/project. A useful option is -p which creates parent
directories as needed (e.g., mkdir -p project/docs/text will create project, project/docs, and
project/docs/text all in one go if they don’t exist).

 rmdir – Remove Directory. Deletes an empty directory. Usage: rmdir <dirname>. If the
directory is not empty, rmdir will refuse to remove it (to prevent accidental deletion of files).
You must first delete or move the contents (or use rm -r with caution). rmdir is thus only
useful for empty folders. A shortcut if a directory has become empty is to use rmdir -p to
remove it and maybe its now-empty parents.

 ls – List directory contents. Probably the most used command. By default, ls lists the names
of files and subdirectories in the current directory (excluding those that start with dot, unless
-a is used). Common options:

o -l for long format (detailed info: type, permissions, owner, size, date, name).

o -a for all files (including . and .. and dotfiles).

o -h for “human-readable” sizes (when used with -l, shows sizes in KB, MB, etc.,
instead of raw bytes).

o -R for recursive listing (list subdirectories’ contents as well).

o -F to append a character indicating type (/ for dirs, * for executables, @ for symlinks,
etc.).
For example, ls -la in a directory will show all entries with full details. ls /etc will list
contents of /etc. ls /etc/*.conf will list all .conf files in /etc (shell wildcard expansion
occurs, more on that later).
These commands are fundamental for navigation and managing directories on UNIX.

4. Ordinary File Handling

This section covers commands for creating, viewing, copying, moving, and deleting files, as well as
comparing and archiving them. Each command below is a standard UNIX utility with a brief
description:

 cat – Concatenate and print files. It reads one or more files and outputs their contents to
standard output (by default, the screen). It’s commonly used to quickly view file contents, or
to join files together (cat file1 file2 > combined). Example: cat [Link] will display the
content of [Link]. If multiple files are given, cat will print them in sequence. You can also
use cat > newfile to create a file by typing input (end with EOF/Ctrl-D). (For large files, cat will
scroll—better to use a pager like more or less.)

 cp – Copy files or directories. Usage: cp source destination. For example, cp file1 file2 copies
the contents of file1 to file2 (if file2 doesn’t exist, it is created; if it does, it’s overwritten). cp
can also copy directories when used with -r (recursive) option: cp -r dir1 dir2 copies directory
dir1 (and its contents) into dir2. Common options include -i (interactive, prompt before
overwrite) and -p (preserve attributes like timestamps, permissions).

 rm – Remove (delete) files or directories. Usage: rm filename deletes the file. There is no
recycle bin by default – rm will permanently remove the file entry (though the data might
remain on disk until overwritten). For safety, -i option prompts before each removal. To
remove a directory and its contents recursively, use rm -r directory. Be very careful with rm -
r, especially as root, because it will delete everything under the specified directory. There’s
also rm -f (force, no prompt). Note that rm by itself cannot remove a directory unless you
use -r (or rmdir for empty directories as mentioned). Example: rm *.o deletes all files ending
in .o in the current directory (shell expands the wildcard).

 mv – Move or rename files and directories. Usage: mv source target. If the target is a
directory, the source file is moved into that directory (with the same name). If the target is a
filename, this effectively renames the file. For example, mv [Link] [Link] renames
the file. You can also use mv to move directories (e.g., mv dir1 /tmp/ moves dir1 into /tmp/).
mv can move multiple files into a directory: mv file1 file2 file3 target_directory. It can also be
used to rename directories. Unlike copying, moving within the same filesystem is very fast (it
just updates directory entries), and it preserves all attributes. If moving across different
filesystems, mv will copy then delete the source. There’s an -i (interactive) flag to prompt on
overwrite, because mv can overwrite existing files without asking.

 more – Pager for viewing text one screen at a time. Instead of dumping a whole file to
screen, more lets you scroll through it (forward only). Usage: more filename. Press space to
go to the next page, Enter for next line, q to quit. It's an older program; GNU less is a more
advanced pager (which allows backward scroll, search, etc.), but more is historically included.
For example, more /etc/passwd will show the content page by page.

 lp (or lpr) – Print files to the system printer (line printer). These commands send files to the
print spooler. For instance, lp [Link] will queue [Link] for printing on the
default printer. lpr is similar, historically from BSD lineage. Options like -P allow selecting a
printer, etc. Usage might depend on system (some systems might use CUPS lp which has
 many options). The output of these commands isn’t to screen; they produce a printout. They
usually return immediately after spooling the job.

 file – Determine file type. This command examines a file’s contents (not just the name) and
tells you what type of file it is (text, binary, executable, image, etc.). Usage: file <filename>.
Example: file /bin/ls might output “/bin/ls: ELF 64-bit LSB executable,...” indicating it’s an
executable binary. file [Link] might say “PDF document, version 1.4”, etc. This is useful
because UNIX file names don't enforce type; file uses magic numbers and heuristics to guess
the type.

 wc – Word Count. It counts lines, words, and characters in files. Usage: wc filename. Output
example: 42 100 800 [Link] where these numbers are (by default) lines, words, and bytes.
You can use options: -l for lines only, -w for words, -c for bytes, -m for characters. For
example, wc -l /var/log/syslog might output the number of lines in the syslog file.

 cmp – Compare two files byte by byte. It will tell you the first location where the files differ
(or say nothing if they are identical). Typically used for binary files or to quickly find if any
difference exists. Usage: cmp file1 file2. If files differ, cmp might output something like: “file1
file2 differ: byte 5, line 1”. If no output and exit code 0, the files are identical. cmp -l can list
all differing bytes and their values.

 comm – *Compare two sorted files line by line*. It reads two sorted files and produces three
columns: lines unique to the first file, lines unique to the second, and lines common to both.
It’s useful for set operations on text files (provided they’re sorted lexicographically). Usage:
comm file1 file2. Lines that are in both will appear in the third column. You can suppress
columns with options (-1, -2, -3 to not show certain columns). For example, to see only
common lines, use comm -12 file1 file2 (suppress unique columns 1 and 2).

 diff – Difference between two files. This is commonly used for text files to see the line-by-line
changes needed to transform one file into the other. Usage: diff file1 file2. The output is a
series of commands (ed-style or unified format if -u is used) indicating what lines to change.
In normal diff output, lines starting with < are from file1 and need to be removed, lines
starting with > are from file2 to be added. Unified diff format (with -u) shows a merged
context with +/- lines, which is widely used for patches. Example:

-old line

+new line

meaning “old line” in file1 was replaced by “new line” in file2. diff can also compare directories (diff -
r dir1 dir2). If two files are identical, diff produces no output (and exit status 0).

 tar – Tape ARchiver, used for bundling files together into a single archive file (optionally with
compression). Common usage patterns:

o Create an archive: tar -cvf [Link] dir/ (c=create, v=verbose list files, f=filename
for archive). This puts dir/ and everything under it into [Link].
 o Extract an archive: tar -xvf [Link] (x=extract).

o List archive contents: tar -tf [Link].

Tar archives are just collections; they are often compressed with other tools (like gzip
or bzip2). tar has flags like -z to compress/decompress with gzip (producing .[Link]
or .tgz files) and -j for bzip2 (.tar.bz2).

 gzip / gunzip – GNU Zip compression. gzip filename compresses the file and typically
renames it [Link]. It uses the DEFLATE compression algorithm. gunzip [Link] will
decompress it (or you can use gzip -d). Gzip is common for compressing single files or
tarballs. Compression reduces file size; the original file is replaced by the compressed version
(unless you use -c to output to stdout). Similar utility is compress (older UNIX compress, not
widely used now, produced .Z files).

 zip / unzip – Zip archive utility. zip compresses and packages files into a single .zip archive
(commonly used on Windows as well). Unlike tar, zip compresses each file (optionally) and
stores multiple files; whereas tar just bundles and relies on external compression. Example:
zip [Link] file1 file2 dir/ will create [Link] containing those files (with compression).
unzip [Link] extracts files from a zip archive. zip is handy for creating archives that need
to be used on Windows, since it’s a cross-platform format. It also has features like
encryption, and partial extraction.

Effects of cp, rm, mv on Directories:

 Copying a directory with cp requires the -r (recursive) option. If you try cp folder1 folder2
without -r, it will error out saying folder1 is a directory. With cp -r folder1 folder2, it will copy
the entire directory tree of folder1 into folder2 (folder2 will contain a new subdirectory
named folder1, unless folder2 didn’t exist and you want to rename on the fly).

 Removing a directory: As mentioned, rm alone cannot remove directories (it prevents

accidental deletion of an entire directory when you perhaps meant a file). To remove a
directory and its contents, you use rm -r directoryName. This recursively deletes everything
under that directory then the directory itself. If you use rm -rf (with force), it will do so
without prompts – very dangerous if misused. Alternatively, if a directory is empty, you can
use rmdir directoryName as a safe way to remove it.

 Moving a directory: mv works on directories as well. If you mv dir1 newname, it effectively

renames the directory. If you mv dir1 /path/to/other/, it moves dir1 (and all its subtree
intact) into the target location. This is generally quick if the source and destination are on the
same filesystem (just renaming pointers). If on different filesystems, it will copy then delete
original. Permissions: you need write permission in the source’s parent (to remove it from
there) and in the destination’s parent (to create it there) to move a directory.

 One caution: cp -r will copy the contents of the directory into the target directory if the
target exists. E.g., if destDir exists and is a directory, cp -r sourceDir destDir will create
destDir/sourceDir/... copy. If destDir doesn’t exist, it will create destDir as a clone of
sourceDir. Similar with mv – if target dir exists, the source is moved inside it; if target path
doesn’t exist, it’s a rename.
5. File Attributes

When we talk about file attributes in UNIX, we often refer to metadata such as permissions,
ownership, timestamps, and other properties associated with each file. Many of these attributes are
visible when using the ls -l (long listing) command. Let’s break down what you see and how to
change these attributes:

Listing File Attributes (ls -l): A sample output of ls -l might look like this:

-rw-r--r-- 1 alice staff 1042 May 15 12:30 [Link]

Let’s decode this:

 File type and Permissions: -rw-r--r-- – this string is 10 characters. The first character indicates
type (- means regular file, d would mean directory, l for symlink, etc.). The remaining 9
characters are the permission bits, grouped in triples: rw- r-- r--. These correspond to owner,
group, and others permissions, respectively. In this example:

o Owner (alice) has rw- = read and write permission.

o Group (staff) has r-- = read-only.

o Others (everyone else) have r-- = read-only.

The - in the third position of each triplet indicates no execute permission in this case.

 Link count: 1 – the number right after the permission string is the number of hard links
pointing to this file. Here it's 1, meaning this file has no additional hard links (just one name
in the filesystem referencing its inode). For directories, this count includes subdirectory
pointers (.. from subdirs).

 Owner: alice – the user who owns the file.

 Group: staff – the group that owns the file. In UNIX, every file has an owning user and group.

 Size: 1042 – size of the file in bytes. In this case 1,042 bytes.

 Timestamp: May 15 12:30 – the last modification date and time. The format is typically Mon
Day hh:mm (or Mon Day Year if the file is over 6 months old or from a different year). This is
the mtime (modification time).

 Name: [Link] – the name of the file (with possible indicator for symlink target or trailing /
in some listings if -F was used, etc.).

So ls -l reveals key attributes: file type, permissions, number of links, owner, group, size, last modified
time, and name. Additional flags like -o (omit group) or -g (omit owner) can adjust what is shown.
Ownership and Permissions: Every file in UNIX has an owner (a user account) and a group owner (a
group of users). Permissions are defined for three categories of entities:

 User (u) – the owner of the file.

 Group (g) – users in the file’s group.

 Others (o) – everyone else.

For each of those categories, there are three permission bits:

 r (read): For files, read permission allows viewing the file’s content. For directories, read
allows listing the directory’s contents (if combined with execute).

 w (write): For files, write allows modifying the file’s content. For directories, write allows
creating or deleting entries within the directory (also requires execute on the directory to
actually make changes).

 x (execute): For files, execute means the file can be run as a program (if it’s a binary or
script). For directories, execute (often called the search bit) means you can enter the
directory (cd into it) and access files inside (given proper read/write on those files). Without
execute on a directory, you cannot cd into it or access files by name within it, even if read is
set.

In the example above -rw-r--r--:

 Owner can read/write (rw-).

 Group can read (r--).

 Others can read (r--).

No one has execute on this file (it’s likely a text file).

Another example: drwxr-x--- 2 root wheel ... somedir would mean somedir is a directory (d), owner
(root) has rwx (read/write/execute meaning root can list, create, and cd into it), group (wheel) has r-x
(can list and cd, but not create/delete files in it because no w), others have no permissions (---,
cannot access it at all).

Changing Permissions – chmod: The command chmod (change mode) is used to modify the
permission bits. You can use symbolic notation or octal notation:

 Symbolic mode: For example, chmod u+x filename adds execute permission for the owner
(user). chmod go-w filename removes write for group and others. You combine these, e.g.,
chmod u=rwx,g=rx,o= filename to set exactly (owner rwx, group r-x, others no access). A
common example: chmod a+r file (make file readable by all), chmod u+x [Link] (make a
script executable by you).
  Octal mode: This uses digits to set the bits. There are three digits (for u, g, o) in base-8,
where read=4, write=2, execute=1, and they add up. For example, chmod 750 file sets owner
7 (4+2+1 = rwx), group 5 (4+0+1 = r-x), others 0 (no permissions). chmod 644 file would set -
rw-r--r--. Octal is convenient for full control in one command and is often used by advanced
users or in scripts.

chmod can also be used recursively (-R) to change permissions in an entire directory tree.

Changing Ownership – chown and chgrp:

 chown user filename changes the owner of the file to the specified user (you typically must
be root to do this, ordinary users generally cannot transfer ownership of files they own to
someone else). You can also change the group at the same time by chown user:group
filename.

 chgrp group filename changes the group ownership (if you own the file or are root, and the
target group is one you belong to or any if root).

For example, if [Link] is owned by alice:staff, and you want to change group to project, you’d do
chgrp project [Link]. If as root you want to give ownership to bob, chown bob [Link] (and
perhaps chown bob:project [Link] to also change group).

Default Permissions and umask: When a new file or directory is created, it has a default set of
permissions minus whatever is restricted by the umask. The umask is a process-level setting that
masks out permission bits. By default on many systems, umask is often 022 (which disables group
and others write permission by default). The logic:

 For files, the system typically starts with 666 (rw-rw-rw-) as the base (read/write for all, since
execute is not automatically given to regular files).

 For directories, base is 777 (rwxrwxrwx, since you generally want traverse permissions).
Then the umask bits are removed. For example, umask 022 removes (0x022 in octal = group
and others write) from the base. So:

o New file => 666 base - 022 = 644 (rw-r--r--).

o New dir => 777 base - 022 = 755 (rwxr-xr-x).

If umask was 002 (common in user groups so that group can write), you’d get 664 for
files and 775 for dirs. If umask is 077 (very restrictive, others and group get nothing),
new files 600, new dirs 700 (only owner can access). You can see or set umask with
the umask command (e.g., umask 022 sets it). In shells, umask without args shows
the current mask (often in octal).

Timestamps: Each file has three important timestamps:

  atime (Access time) – last time the file was read.

 mtime (Modification time) – last time the file’s content was modified.

 ctime (Change time) – last time the file’s metadata changed (permissions or owner changed,
or content changed). ctime is updated on any change to the inode (which includes content
changes and attribute changes).

ls -l by default shows mtime. ls -lu shows atime. ls -lc shows ctime. Many filesystems now have
options where atime is not updated on every read (for performance) unless explicitly enabled.

The touch command is a handy tool to manipulate timestamps. touch filename will update the file’s
atime and mtime to the current time (essentially “touching” it to look recently modified) without
altering content. If the file doesn’t exist, touch will create an empty file with that name (this is a
quick way to create a blank file). You can also use touch -t with a specific timestamp or -r to reference
another file’s time. This is often used to mark files or to force rebuilds in build systems by updating
timestamps.

Finding Files – find: The find command is extremely powerful for searching for files in the directory
tree based on various criteria (name, type, size, timestamps, ownership, permissions, etc.). Basic
usage: find <path> <criteria> <actions>. For example:

 find . -name "*.c" -print searches in the current directory (.) and subdirectories for files with
names ending in .c and prints their paths.

 find /var/log -size +1M -print finds files larger than 1 MB in /var/log.

 find /home -type d -name "project" finds directories named "project" under /home.

 find /tmp -mtime +7 -delete finds files in /tmp not modified in the last 7 days and deletes
them (be careful with -delete).

 find . -maxdepth 1 -mtime -1 finds files in current dir (not recursing further than 1 level)
modified in the last 1 day.
One of find’s powerful features is the ability to execute a command on each found file via -
exec, for example: find . -name "*.py" -exec grep "TODO" {} \; would run a grep on each .py
file for "TODO". find operates depth-first through the file tree. It’s a fundamental tool for
administrative tasks (like finding all files owned by a user, or all world-writable files, etc.).

Links (Hard links vs Soft links):

UNIX provides two different ways to have multiple references to a file:

 Hard Link: A hard link is essentially an additional directory entry for a file. Each hard link is
indistinguishable from the original name; it’s just another name for the same inode (same
file content). Because it references the inode directly, hard links must reside on the same
filesystem (you cannot hard link across different mounted filesystems). Also, you cannot
 hard-link a directory (to avoid loops in the filesystem). The ln command without options
creates a hard link: ln existingfile newlinkname. After this, existingfile and newlinkname point
to the same data (same inode number). If one is deleted (rm), the other still exists and the
file content is still accessible (the link count on the inode will drop by one). Only when the
link count drops to 0 does the space get freed. Hard links share the same permissions and
owner (since that’s stored in the inode). If you do ls -li, you can see the inode number and
link count:

$ ls -li [Link] [Link]

32 -rw-r--r-- 2 user group 6 May 20 05:16 [Link]

32 -rw-r--r-- 2 user group 6 May 20 05:16 [Link]

Here, both names have inode 32, and the number after permissions is 2, meaning two hard links
total (the two names)【23†output】. They are truly the same file. Changes to one are reflected when
accessing the other (since there is no "other" – it's the same data).

 Soft Link (Symbolic Link): A soft link is a special file that points to another file path. It’s like a
shortcut. You create one with ln -s targetpath linkname. The link is a tiny file that contains
the text of the path to its target. When you open or execute a symlink, the OS transparently
redirects to the target. Symlinks can span filesystems and can point to directories or files.
They also can dangle (point to something that no longer exists). In ls -l, a symlink is indicated
by l and it shows something like:

35 lrwxrwxrwx 1 user group 20 May 20 05:16 [Link] -> testdir/[Link]

Here inode 35 is the link itself (different from the inode of the target, which was 32 as above), and it
shows an arrow -> to the target path【23†output】. The size of a symlink is the length of the path it
contains (in this case 20 characters). Permissions on a symlink (rwxrwxrwx usually) are not really
used; access is determined by the target’s permissions. If you remove a symlink, the target remains.
If you remove the target, the symlink becomes a broken link that goes nowhere (trying to use it will
result in "No such file or directory"). Symlinks are extremely common for making convenient aliases
or structuring certain directories (e.g., /usr/bin/python -> [Link] to point to a default version).

In summary, hard links are multiple directory entries for the same inode (file content), whereas soft
links are special files that refer to another file’s name. Hard links are more rigid (same filesystem, no
directories) but efficient (no extra overhead except another directory entry). Soft links are flexible
(can cross boundaries, can link directories) but are a separate object that can break if target moves
or is deleted.

6. The Shell
The shell is the command-line interpreter that provides the traditional UNIX user interface. When
you interact with a UNIX system by typing commands, you are typically communicating with a shell
(such as bash, sh, csh, ksh, etc.). Let’s break down how the shell works and key shell features:

Interpretive Cycle: The shell runs in an interpretive loop often described as read-eval-print loop:

1. Read: The shell displays a prompt (like $ or %) and waits for the user to type a command line.
The user input can contain a command, arguments, and special symbols (for redirection,
etc.).

2. Parse/Interpret: The shell then parses the input. This involves tokenizing the command and
arguments, interpreting any special characters (like wildcard * expansion, variable
substitutions VAR, command substitutions `...` or (...), quotes, etc.), and figuring out what
command to run. It also checks if the command is a shell built-in (internal command) or an
external program.

3. Execute:

a. If it’s an internal command, the shell executes it directly within the shell process.

b. If it’s an external command, the shell will fork a new process and exec the command
binary. (The shell searches for the program in the directories listed in the PATH
environment variable, unless you gave a path.)

c. If commands are connected with pipes (|), the shell sets up the pipeline (multiple
processes connected by pipe buffers) before execution.

d. It handles I/O redirections (open files for input/output as requested) before

executing the command.

4. Wait: If the command is run in the foreground (the usual case), the shell waits for the
command to finish. If it’s a background command (ending with &), the shell does not wait –
it returns immediately to the prompt while the command runs in background.

5. Print (Return to Prompt): Once the command execution is done, the shell may print any
results of job completion (if background) or just loop back to prompt. Any output from the
command itself would have already been displayed or redirected as requested. The shell
then is ready to read the next command.

This loop continues until you exit the shell (e.g., by typing exit or pressing Ctrl-D on an empty line).
This interactive cycle is what users experience. In non-interactive mode (executing a shell script), the
shell reads commands from the script file in a similar fashion and executes them sequentially.

Shell Types (Variants): There are several shells available on UNIX systems. The original was the
Thompson shell, then the Bourne shell (sh) became standard in early UNIX. Variants and popular
shells include:

 Bourne-compatible shells: These use syntax derived from the Bourne shell. Examples: sh
(the original), Bash (bash, Bourne Again SHell) which is very common on Linux, Korn shell
 (ksh), Z shell (zsh). They have similar syntax for scripting (if/then, loops, etc.) with
enhancements in each.

 C shell family: (csh and its improved version tcsh). They have a somewhat different syntax (C-
like syntax for expressions, different built-in constructs) and were popular especially for
interactive use (tcsh has nice features like autocomplete).

 There are others like Dash (a lightweight POSIX sh), fish (a user-friendly shell), etc.
Each shell has its own feature set and quirks. For example, Bash is very widely used for both
interactive and scripting, offering command history, tab completion, arithmetic, arrays, etc.
Zsh also offers advanced completion and customization. Tcsh and csh have a different
scripting style but for interactive use tcsh offers spelling correction, etc. In practice, when
writing scripts, /bin/sh is used for POSIX compatibility (which might be dash or bash in POSIX
mode under the hood), and interactive users often choose bash or zsh nowadays.

Pattern Matching (Globbing): Shells provide wildcard characters for filename pattern matching:

 * – matches any string of characters (including empty string). For example, ls *.txt will match
all files ending in “.txt” in the current directory.

 ? – matches any single character. Example: ls file?.txt would match [Link] or [Link] but
not [Link] (since ? is one char).

 [ ] – character class. Matches any one character from the set inside brackets. e.g., ls
report[123].txt matches [Link], [Link], or [Link] only. Ranges can be specified
like [A-Z] (any uppercase letter) or [0-9] (digit). You can negate with [^ ] (e.g., [^aeiou] means
any character that is not a vowel, if used).
These are often referred to as globs. Important: the shell expands these patterns to the
matching filenames before running the command. If no file matches, some shells treat it
literally, others might give an error (depending on settings like nullglob in bash). For instance,
if you have files [Link], [Link], then echo *.txt actually becomes echo [Link] [Link] which then
prints those names. Globbing does not use regular expression syntax; it’s simpler (no + or |
operators in basic globbing).

Escaping and Quoting: Sometimes you need to prevent the shell from interpreting a character (like a
space in a filename or a wildcard or a dollar sign).

 Escape with backslash (\): A backslash before a character tells the shell to treat that
character literally. Example: echo Hello\*World will output Hello*World (the * is not
expanded). Or space\ in\ name would be interpreted as space in name as a single argument
with a space in it.

 **Single quotes ('...')**: Everything inside single quotes is taken literally, no expansion at all
(except you can’t have a single quote inside, unless you do some trickery). E.g.,echo 'Today \
HOME * ?'will outputToday \HOME * ?` literally, with no variable or wildcard expansion.

 Double quotes ("..."): Inside double quotes, the shell will not do word splitting or globbing,
but it will still perform variable expansion VAR and command substitution and interpret
backslash escapes for certain characters. For example, echo "Home: $HOME" will expand the
 HOME variable, but any spaces in the expanded value won’t cause splitting into multiple
words. Generally, use double quotes around expansions to preserve spaces. If you want a
literal $ inside double quotes, you can escape it as \.
In summary: use quotes to group arguments that have spaces or to protect special
characters. A common rule is quote your variables, e.g., "FILE" to avoid issues if \FILE has
spaces or glob characters in its value.

Input/Output Redirection: Shells allow you to redirect standard input, standard output, and
standard error of commands:

 > (redirect output): e.g., ls > [Link] redirects stdout of ls to a file named [Link] (creating or
truncating it). If the file exists, it’s overwritten (truncated to 0 then filled).

 >> (append output): e.g., echo "new" >> [Link] appends to [Link] instead of truncating.

 < (redirect input): e.g., sort < [Link] takes input from [Link] instead of keyboard.

 2> (redirect standard error): By default > and >> only redirect file descriptor 1 (stdout). 2>
redirects file descriptor 2 (stderr). For example, gcc prog.c 2> [Link] will compile and
send error messages to [Link]. You can also use 2>> to append errors.

 Combine stdout and stderr: If you want to redirect both, one method in many shells is &>
(for both stdout and stderr to a file) or using 2>&1 to point stderr to the same destination as
stdout. For example: cmd > [Link] 2>&1 sends both output and errors to [Link].

 Heredoc (<<): There’s also a form like command <<EOF which allows you to provide a block
of input in the script itself, ending with a marker like EOF.

Special Devices for redirection:

 /dev/null: This is a special device file that acts as a black hole. Sending output to /dev/null
discards it. For example, somecommand > /dev/null 2>&1 will throw away both output and
errors (useful if you want to run a command and ignore all of its output). Reading from
/dev/null returns EOF immediately (so it's like an empty file).

 /dev/tty: This represents the controlling terminal of the current process. If you redirect
something to /dev/tty, it will always go to your terminal (screen), regardless of other
redirections. Conversely, you can read directly from /dev/tty to get input from the user’s
terminal even if input is redirected from somewhere else. For instance, a script could do read
-p "Enter secret: " secret < /dev/tty so that it reads from the real user, not from a file or pipe.

Pipes: The pipe symbol | is used to connect the stdout of one command directly into the stdin of
another. It allows the output of one program to be processed by another program. For example: ls -l
| grep "\.txt$" passes the listing into grep, which then outputs only lines ending in .txt. Multiple
pipes can chain many commands (forming a pipeline). Pipelines are fundamental in UNIX to filter and
transform data (the concept of using simple tools together). Under the hood, the shell creates a pipe
(buffer in kernel), and connects file descriptors between processes accordingly, running both
processes in parallel (with appropriate synchronization on the pipe). Another example: cat bigfile |
less lets you scroll through a large file by piping to the pager. Note: A pipeline can combine with
redirection, e.g., grep "ERROR" logfile | tee [Link] | wc -l (this finds lines with "ERROR" in logfile,
tee writes them to [Link] while also passing them through, and then wc -l counts them).

tee command: Mentioned above, tee is a useful utility in pipelines. It reads from stdin and writes to
stdout and to one or more files. So it “tees” the stream (like a T-splitter in plumbing). In the example,
| tee [Link] |, it took the stream of error lines and output them to [Link] and continued to
pass them to the next command (wc -l). Without tee, you’d either have to redirect to a file or to wc,
but not both. tee -a file appends instead of overwriting.

Command Substitution: Sometimes you want to use the output of one command as part of another
command’s arguments. Shells allow this via command substitution. Two syntaxes:

 Legacy: `command` (backticks).

 Modern: $(command).
They do the same thing, but the () form is generally preferred (easier to nest and read).
Example: DATE=$(date +%Y-%m-%d) will run date +%Y-%m-%d (which outputs current date
in YYYY-MM-DD format) and assign that string to the variable DATE. Another example: echo
"There are $(ls | wc -l) files here." – this runs ls | wc -l inside the substitution, capturing the
number of files, and then the echo prints the full sentence with that number. Essentially,
anything in ( ... ) is executed by the shell in a subshell and the stdout (with trailing newline
removed) is substituted in place. This is very powerful for embedding one command’s result
into another context. Be mindful of quoting the substitution if the output could contain
spaces or special characters.

Shell Variables: Shells have variables that can store values (strings, numbers, etc.), which can
influence shell behavior or be used in scripts. For example:

 To set a variable in the shell: NAME="Alice" (no spaces around =). By default this is a shell
(local) variable.

 To use it: reference with a `, e.g.,echo "Hello, NAME!"would outputHello, Alice!`.

 Environment variables: If you want a variable to be inherited by child processes, you need to
export it. e.g., export EDITOR=vim. Now EDITOR is not only set in the shell but will be in the
environment for any program you run. Common environment variables (like PATH, HOME,
etc.) are set to be exported.

 Some variables are predefined: e.g., HOME, $PATH, $USER, $PWD, $SHELL (the shell program
you’re using), etc. There are also special shell-only variables like $? (exit status of last
command), $# (number of arguments in a script context), $0 (script name), 1... (first
argument, etc. in a script or function context), and so on.

 Shell Prompt Variables: PS1 is the primary prompt string (the one you see for interactive
shell). You can customize it, e.g., export PS1="\u@\h:\w " gives a prompt like
user@hostname:/current/dir. PS2 is the secondary prompt (shown when a command spans
multiple lines, default is >).
  Modification and use: No need to declare types (all are essentially strings, though arithmetic
context can treat as numbers). For example COUNT=5 and then echo $COUNT prints 5. But
COUNT="5 things" is also fine as a string.

 Quote variables when using them, especially if they might be empty or contain spaces: For
instance, rm FILE can fail or do unexpected things if $FILE contains whitespace or is empty;
safer is rm "FILE".

Shell variables are also used for control flow in scripts and storing intermediate results, similar to
programming languages, but they are all stringly-typed unless treated numerically with specific
context (like in bash you can do ((i = i + 1)) which treats i as number).

7. Processes in UNIX

A process is a running instance of a program. The UNIX kernel handles processes by allocating each
an ID, memory space, and scheduling them to run on the CPU(s). Here’s an overview and some
commands related to process management:

Process Lifecycle Overview: When you execute a command (external program), the shell uses the
fork() system call to create a new process (which is almost a copy of the shell), and then that child
process uses exec() to replace itself with the new program. The new process gets a unique Process ID
(PID). The original process (shell) may wait for it or continue running concurrently (for background
jobs). Every process has a parent (except the initial process). In UNIX, the first process started by the
kernel is init (with PID 1). Historically, init was the service manager that started all other processes
(on modern Linux, systemd or other init systems take this role but still have PID 1). All processes are
descendants of init.

ps – Process Status: This command lists processes. Without arguments, ps might show only your
processes attached to the current terminal. Common usage:

 ps -e or ps -A to list all processes in the system.

 ps -f for a full listing (including owner, parent PID, start time, etc.).

 ps -ef (combining both) is common on System V style systems to list all processes in full
format.

 ps aux is a BSD-style usage (where a means all with terminal, u for user-oriented format, x to
include processes without a terminal).
Either way, you get a snapshot of processes at the time you run ps. Typical columns: PID, TTY
(terminal associated, or ? if none), TIME (CPU time used), CMD (the command name or
command line).
For example:
 $ ps -f

UID PID PPID C STIME TTY TIME CMD

alice 1234 1200 0 10:40 pts/0 [Link] vim [Link]

alice 1300 1234 0 10:50 pts/0 [Link] ps -f

Here, vim is running with PID 1234, parent PID 1200 (which might be the shell’s PID), and we see the
ps command itself with PID 1300 and parent 1234. The TTY is pts/0 (meaning it's tied to an
interactive terminal session).

init -> getty -> login -> shell process chain: This describes the sequence of processes involved when a
user logs in on a text console or terminal:

 init (PID 1) is started by the kernel at boot. It spawns various daemons and also getty
processes for each login terminal.

 getty waits on a terminal (could be a physical console or serial line or virtual terminal). It's
responsible for printing the "login:" prompt and waiting for a user name. Once you enter a
username, getty hands off to login.

 login prompts for a password and authenticates the user. If login is successful, it sets up the
user’s environment (home directory, groups, etc.) and then execs the user's shell (as
specified in /etc/passwd). If login fails, it may allow retries or exit, in which case init
respawns getty.

 User Shell: Now you are running a shell (e.g., bash) as an interactive process. This shell
process is a child of login (which was a child of getty, which was a child of init). The shell
prints the prompt and accepts commands – entering the interpretive cycle as described
earlier.

So the hierarchy for a text login is: init -> getty -> login -> shell -> your commands. If you run ps -ef
you might see this chain; for example, getty or atty might appear for each terminal, etc. On modern
Linux with systemd, getty is still used for virtual consoles, but the principle remains.

When you logout (exit the shell), the shell process ends. The login process would have already been
replaced by the shell (so it’s gone). init notices the shell ended (via SIGCHLD and wait) and then
usually spawns a new getty for that terminal again, so the cycle can start over for the next login.

Process States: Processes can be in various states:

 Running (R): executing on CPU or ready to run.

 Sleeping (S): waiting for an event (like I/O completion or a resource). If it’s an interruptible
sleep (most sleeps are), signals can wake it.
  Uninterruptible sleep (D): typically waiting on I/O that cannot be interrupted (seen in ps as D
state).

 Stopped (T): process is paused, usually via job control (Ctrl-Z) or because it’s being
debugged.

 Zombie (Z): process has terminated, but its parent has not yet done a wait() to collect its exit
status. A zombie is essentially a defunct process that still has an entry in the process table (to
hold the exit status for the parent to pick up). It’s not running or capable of running. Once
the parent collects the status (via wait), the zombie is removed completely. Zombies show up
in ps with <defunct>. If a parent never collects the status and itself dies, init inherits the
zombie and then immediately cleans it up.
Usually, if you see zombies, it means some parent process isn’t properly reaping children.
One or two zombies are not harmful (they take negligible resources), but many could
indicate an issue.

Background and Foreground Jobs: When using a shell, you can append & to a command to run it in
the background. This means the shell doesn’t wait for it, and you get the prompt back immediately.
The process still runs, but it’s not attached to the terminal for input. Background jobs can still output
to terminal (which can mix with your typing unless you redirect them). Each shell maintains a job
table for processes it launched. You can see these with jobs command (in the shell, not an external
command). jobs will list jobs with an index like [1] Running command... or [2] Stopped command...
etc.

 bg – if you have a stopped job (for example you pressed Ctrl-Z which suspends a job), bg
%jobnumber will resume it in the background.

 fg – bring a background job to foreground (regain control/input). Usage: fg %jobnumber. If

you omit, it defaults to the current job (marked with +).

 Ctrl-Z – keystroke to suspend a running foreground job (sends SIGTSTP). The shell will then
typically report [1]+ Stopped command.... You can then choose to bg or fg it.

 Note: these job control features work in interactive shells. In a script (non-interactive), job
control is usually not used the same way.

nice (and renice): The nice value of a process is its scheduling priority hint to the kernel (a higher
"nice" means lower priority because you're being "nice" to others). By default processes start with
nice 0. You can start a process with a modified priority using nice command. E.g., nice -n 10 long_task
& will start long_task with a niceness of +10 (meaning it will yield more often to other processes,
essentially lower priority). Only root can decrease the niceness (i.e., give higher priority than default,
negative nice value). The range is typically -20 (highest priority) to +19 (lowest). renice can change
the nice value of an already running process: e.g., renice +5 1234 makes PID 1234 a bit "nicer" (lower
priority) than it was.

This is useful when running background computations – you can lower their priority so they don’t
slow down interactive use.
nohup: This stands for "no hangup". Normally, if you run a process and then logout (or if the terminal
closes), the kernel sends a SIGHUP (hangup signal) to your processes, which by default will terminate
them. nohup allows a process to continue running even after you log out. Usage: nohup command &.
It effectively detaches the command from the terminal, ignores the HUP signal for that process, and
by default redirects its output to [Link] (if not already redirected). This is commonly used for
long-running processes on a remote session that you want to survive the session. Another modern
alternative is using a terminal multiplexer like screen or tmux, but nohup is a simple built-in way.

Killing Processes – kill: The kill command is used to send signals to processes (despite its name, you
can send any signal, not just terminate). By default kill PID sends SIGTERM (signal 15) which requests
graceful termination. Processes can handle or ignore SIGTERM (except if they explicitly don’t catch
it). If a process doesn’t respond, you can send a SIGKILL (signal 9) via kill -9 PID. SIGKILL cannot be
caught or ignored; it will immediately terminate the process (as long as you have permission to kill
it). Other signals: kill -HUP PID (SIGHUP, often to tell a daemon to reload config), kill -USR1 etc. You
can also kill by process name with pkill or killall (be careful, killall on Linux kills by name, but on some
UNIX killall might do something else).

To kill a background job in your shell, you might do kill %1 (using job number). If you have a stopped
job you don't want, kill %job will terminate it.

Job Control Commands Recap:

 jobs – list current shell's background/stopped jobs.

 bg – resume a stopped job in background.

 fg – bring a job to foreground.

 kill – send a signal to a job or process by ID.

at and batch: These are commands for scheduling processes to run at later times (one-time
execution, not recurring like cron).

 at allows you to schedule a command to run once at a specified time in the future. You enter
at <time> (for example, at 2:30am or at now + 15 minutes). Then at will give you a prompt
where you type the commands you want to run at that time, end with Ctrl-D. The commands
will be executed at the given time as your user. atq shows pending jobs, atrm removes a
scheduled job.

 batch (or sometimes at -q b or batch is part of at on Linux) will run commands when system
load is low. Essentially, you give it commands and it will execute them when it can (when
load average drops below a certain threshold). It's not commonly used nowadays, but
historically it let you defer heavy jobs until the system is not busy.
These are useful for one-time scheduled tasks. Modern systems may have them disabled by
default (requiring certain permissions) or use alternatives, but they’re standard in UNIX.
8. Customization

UNIX shells and environments are highly customizable, allowing each user to tailor their working
environment. Key aspects of customization include environment variables, shell prompts, aliases, and
history configurations:

Environment Variables: These are variables that affect how processes behave and interact. Some
important environment variables and their typical meanings:

 HOME – As discussed, the path to the user’s home directory (e.g., /home/alice). Many
programs use this to find user-specific files. The shell uses it for cd with no arguments, and ~
expansion.

 PATH – A colon-separated list of directories that the shell searches for executable commands.
For example, PATH="/usr/local/bin:/usr/bin:/bin:/home/alice/bin". When you type a
command (like gcc), the shell looks in each directory in PATH in order, for an executable file
named gcc. Customizing PATH is common when adding new tools (like adding $HOME/bin if
you have your own scripts, or adding custom software paths). Exporting PATH ensures child
processes use the same search path.

 LOGNAME and USER – Typically both contain your username. On many systems, USER is the
name of the logged-in user (same as whoami output) and LOGNAME likewise. They often are
set to the same value. Some scripts or programs might use these to identify the user.

 TERM – Indicates the type of terminal or terminal emulator in use, e.g., xterm-256color,
vt100, linux, etc. This helps programs (like text editors or UI programs) know how to interact
with the terminal (what control codes to use for colors, moving cursor, etc.). Setting TERM
incorrectly can result in garbled output in curses-based applications. Usually it’s inherited
and auto-set by ssh or terminal.

 PWD – Present Working Directory. The shell updates this environment variable each time
you change directories. It always reflects the current directory as an absolute path. Some
shells also have an OLDPWD which is the previous directory (used by cd -).

 SHELL – The path of your login shell (e.g., /bin/bash or /bin/tcsh). It’s often set by login, and
some scripts might check it to see what shell you use.

 PS1 – Primary prompt string, as mentioned. This defines how your shell prompt looks for an
interactive shell. It can include special escapes like \u for username, \h for host, \w for
current working directory, etc. For root, the default PS1 might end in # instead of $.
Customizing PS1 is a common way to make the prompt show git branch or other context
(with more advanced prompt tools).

 PS2 – Secondary prompt string. This is displayed when the shell expects more input, e.g., if
you open a quote and press enter, you’ll see > by default (meaning the shell knows the
command isn’t finished). You can change it, though that’s rare.

 Other variables: LANG or LC_ALL for locale settings, EDITOR or VISUAL to indicate preferred
text editor (used by some programs like crontab or git for commit messages), PAGER for
default pager (e.g., less or more for man pages), MANPATH for manual search path, and
many others for specific applications.
Environment variables are typically set in your shell startup files (like ~/.bash_profile, ~/.bashrc, or
~/.profile depending on shell and login vs interactive) so that they’re configured when you log in. To
set an environment variable in shell, you usually do export VAR=value (in Bourne-type shells) or
setenv VAR value (in C shell).

Aliases: An alias is a shortcut or alternate name for a command (possibly with default options). Shells
allow you to define aliases for convenience. For example:

alias ll='ls -laF'

This means when you type ll, the shell replaces it with ls -laF – which might list all files in long format,
appending indicator characters. Aliases are great for abbreviating frequently used long commands.
Other common ones: alias gs='git status', or alias ..='cd ..' to quickly go up one directory, etc. To see
current aliases, run alias with no args. Remove alias with unalias name. Note that aliases are not
exported – they only apply in the shell where defined (and its interactive sessions). So, you put them
in ~/.bashrc for instance. One thing: In scripts, it’s often recommended to avoid relying on aliases
(scripts often start with #!/bin/sh or #!/bin/bash --norc etc. which may not include aliases). Aliases
are mostly for interactive convenience.

Command History: The shell (at least modern shells like bash, zsh, etc.) keeps a history of commands
you’ve entered. This allows you to use the up/down arrow keys to navigate recent commands, or
specific history search (like Ctrl-R for reverse search by typing part of a command). The history is
stored in memory during the session and usually saved to a file (like ~/.bash_history) on logout.
Some details:

 The history command will list your history with line numbers. For example history 20 shows
last 20 commands.

 You can rerun a command from history by typing !n (where n is the history line number) or
using !prefix to rerun the last command starting with "prefix". For instance, !ls might rerun
the last ls command you executed. !! repeats the last command (same as up arrow + enter).
Be cautious with history expansion, but it’s useful (e.g., sudo !! to repeat last command with
sudo).

 Environment variables like HISTSIZE (number of commands to keep in memory) and HISTFILE
(filename to save history, default ~/.bash_history) control how history works. Also
HISTCONTROL can ignore duplicates or leading-space commands.

 For csh/tcsh, history is slightly different (history command lists, ! works similarly). Zsh has
very advanced history features (shared history, etc.).

 The history feature greatly improves productivity by letting you recall and edit previous
commands (using arrow keys or editing inline with Emacs/vi keybindings if set).
Prompt Customization: We touched on PS1. Users often customize their prompt to include
information like current directory (\w or \W for just the basename), user/host, maybe git branch via
special scripts, etc., often with colors (achieved by including terminal color codes in PS1). An example
PS1:

export PS1="\u@\h:\w\$ "

This would yield prompts like alice@myhost:/home/alice/projects$. Root’s prompt often has # at
end, which can be set by using \# or by the shell automatically if UID=0 (bash sets PS1 to # by default
for root). Customizing prompt can also show a shortened path or time or any info accessible.

Shell Startup Files: Customizations are usually placed in shell startup files:

 For login shells: ~/.bash_profile or ~/.profile (for Bourne shells), ~/.login for csh.

 For interactive non-login shells (like opening a new terminal tab): ~/.bashrc is executed.
Often ~/.bash_profile will source ~/.bashrc to unify settings. Aliases and interactive tweaks
usually go in .bashrc, while login stuff (like PATH exports) might be in .bash_profile.

Other Customizations:

 Keyboard shortcuts in shell: e.g., Ctrl-U to erase the whole line, Ctrl-W to erase a word, Ctrl-
C to cancel a running foreground process, Ctrl-D to send EOF (logout if typed at empty
prompt).

 Shell options: In bash, shopt and set can enable/disable certain behaviors. For example set -
o vi enables vi editing mode for command line (instead of default Emacs mode), set -o
noclobber can prevent > from overwriting files (unless you use >|), etc.

 Shell Functions: Users can also define their own functions in shell which can be more
powerful than aliases (they can take arguments, have scripts of commands). For instance:

extract() {

# function to extract various archive types

if [ -f "$1" ] ; then

case "$1" in

*.tar.bz2) tar xjf "$1" ;;

*.[Link]) tar xzf "$1" ;;

*.zip) unzip "$1" ;;

 *) echo "don't know how to extract $1" ;;

esac

else

echo "$1 is not a file"

fi

Then extract [Link] would call the function.

All these allow a user to create a very customized environment that suits their workflow.

9. Filters

Filters are commands that take input (usually from stdin or from a file) and transform it in some way,
writing the result to stdout. They are often used in pipelines (thus "filtering" a stream of data). Here
are some essential text processing filters and their uses:

 pr – Prepare text for printing. This command paginates or formats text into pages (with
headers, footers, page numbers) suitable for printing on line printers. By default, it adds a 5
line header (with date, page number, and optionally a centered title) and breaks text into 66-
line pages. E.g., pr [Link] will show the file with these page headers. Options include -h
"Title" to set a custom header title, -l <lines> to set page length, -2 (or other number) to print
in multi-column format, etc. pr is less used nowadays but is useful for quickly paginating text
or printing in multi-column.

 head – Show the beginning of a file. By default, it shows the first 10 lines of a file (or stdin).
Usage: head filename. You can specify the number of lines with -n, e.g., head -n 20 [Link] for
first 20 lines. Or head -20 [Link] (the -n can be optional in some implementations). Also head
-c 100 file would output first 100 bytes. It’s commonly used to preview a file or in pipelines
where you only want the first few results (e.g., ps -ef | head to see first processes).

 tail – Show the end of a file. By default, last 10 lines. Usage similar to head: tail -n 50 [Link]
to see last 50 lines of a log. A very useful option is -f (follow) which keeps the file open and
continuously outputs new lines as the file grows. tail -f /var/log/syslog will show new log
entries as they are written (use Ctrl-C to stop). This is helpful for monitoring logs in real-time.
On modern systems, tail -F will follow even if the file is rotated (it will reopen if the file is
recreated).

 cut – Cut out columns or fields of text. cut can extract specific character positions or
delimited fields from each line of input. Two modes:

o By character position: e.g., cut -c1-80 file would cut the first 80 characters of each
line.
 o By field (delimiter): e.g., cut -d':' -f1,3 /etc/passwd will use : as delimiter and extract
fields 1 and 3 from each line of /etc/passwd (which would be the username and user
ID for each account).
You can specify multiple fields (comma-separated) or ranges. cut is useful for
breaking columns (like if you have CSV or tab-separated data, etc.). Example: given a
CSV line "Alice,Engineering,60000", cut -d',' -f1,3 yields "Alice,60000".

 paste – Paste lines side by side. It takes multiple input files (or streams) and merges
corresponding lines into one line, separated by a tab (default) or another delimiter. For
example, if file1 has names and file2 has phone numbers line by line, paste file1 file2 will
produce "namephone" for each line pair. You can change delimiter with -d. It’s basically the
opposite of cut in some sense; cut slices columns out, paste can join columns together from
separate sources.

 sort – Sort lines of text. By default, alphabetically sorts input lines and outputs the sorted
result. It has many options for different types of sorting:

o -n for numeric sort (so that "2" comes before "10", etc., rather than lexicographically
where "10" would come before "2" because '1' < '2').

o -r for reverse order.

o -f for case-insensitive sorting (fold case).

o -k to specify key (column) to sort by. For instance, sort -k2 would sort by second field
onward.

o -t to specify a field delimiter if not whitespace.

o -u to output unique lines (skip duplicates).

For example, sort -t: -k3 -n /etc/passwd might sort /etc/passwd by the third field
numerically (UID). sort [Link] > [Link] sorts a file and writes sorted order to new
file.
Note: Sorting big files might use temporary disk space (check the -T option for
specifying tmp dir if needed).

 uniq – Report or filter out repeated lines. uniq by itself takes input (often that has been
sorted) and removes consecutive duplicate lines, outputting one instance of each line.
Important: uniq only collapses adjacent duplicates. So usually you combine it with sort if you
want to find unique lines in an unsorted file: e.g., sort [Link] | uniq gives unique names
(because sort brings identical items together, and uniq then removes duplicates). Options:

o -c to count occurrences. It prefixes each unique line with the number of times it
occurred in input.

o -d to show only lines that were duplicates (at least one duplicate).

o -u to show only lines that were unique (occurred exactly once).

Example: sort [Link] | uniq -c might output frequency counts of identical log
lines.

 tr – Translate or delete characters. tr operates on characters (not by lines). Common usage:

 o Transliterate: tr 'abc' 'ABC' < inputfile > outputfile – this would replace all 'a' with 'A',
'b' with 'B', 'c' with 'C' in the input. Often used to do case conversion: tr '[:lower:]'
'[:upper:]' to make text uppercase ([:lower:] and [:upper:] are character classes).

o Delete: with -d. For example, tr -d '\r' < file > newfile – useful to delete carriage
return characters (like if converting DOS text files to UNIX by removing \r).

o Squeeze: -s option to squeeze repeats of a character into one. e.g., tr -s ' ' will
replace sequences of multiple spaces with a single space.
Because tr reads from stdin and writes to stdout, usage is often in a pipeline or with
input redirection. Example: echo "hello" | tr 'aeiou' '12345' would output h2ll4
(replacing vowels with numbers). Or cat file | tr -d '[:punct:]' to remove punctuation.

 grep – Search for patterns in text (Global Regular Expression Print). grep reads input (file or
stdin) and prints lines that match a given pattern. Basic usage: grep "searchterm" filename.
This will print all lines in filename containing "searchterm". By default, grep uses basic
regular expressions for the pattern (where most characters match literally, and special regex
chars like ., *, ^, $ etc. are available, though in basic regex, ?, +, | are not special unless
escaped or unless using extended grep). Some useful options:

o -i for case-insensitive search.

o -n to prefix line numbers to matches.

o -v to invert match (print lines that do not match the pattern).

o -r or -R to recursively search directories.

o -w to match whole words only.

o -A / -B / -C to show context lines After, Before, or Centrally (both) around matches.

For example: grep -i "error" *.log could search all log files for "error" in any case.
grep -v "^#" [Link] would print all lines that are not comments (assuming
comments start with # at beginning).

Basic vs Extended Regular Expressions:

 Basic Regular Expressions (BRE): This is the default for grep. In BRE, characters like . (match
any single char), * (match 0 or more of the preceding element) are available. Parentheses ()
and | (alternation) are literal unless backslashed (so \( and \| in BRE are used for grouping
and alternation). Likewise, + and ? have no special meaning unless escaped (some versions
treat \+ and \? as one-or-more and zero-or-one).
Example BRE: grep "file[0-9]*\.txt" list would match things like [Link] (here * applies to
the character class [0-9], meaning any number of digits).

 Extended Regular Expressions (ERE): These are used by egrep or grep -E. In ERE, ?, +, {m,n}
qualifiers, | alternation, and () grouping are available without escaping. So it's generally more
powerful and usually easier since you don’t have to escape as much. egrep (extended grep)
is essentially the same as grep -E.
Example ERE: egrep -i "error|warning|critical" file would find lines with any of those words.
In basic grep you'd have to do grep -E or escape the | and parentheses if grouping.
So, grep vs egrep: historically, egrep is the extended regex version of grep. Modern grep -E is
equivalent. There's also fgrep (or grep -F) for fixed-string search (no regex, just literal). Now grep
encompasses all with options.

egrep: (Extended GREP) As mentioned, it uses extended regex. For instance, egrep "July (2019|
2020)" matches "July 2019" or "July 2020". With basic grep, you'd do grep -E "July (2019|2020)" or
grep "July \(2019\|2020\)".

Piping and combining filters: The real power of these filters is using them in combination. For
example:

 ps -ef | grep apache | wc -l – count how many apache processes are running.

 grep -R "TODO" . | cut -d: -f1 | sort | uniq – find all files with "TODO" in current directory
(recursively), cut just the filenames before the colon, sort them, and uniq to list each file
once.

 head -n 1000 [Link] | tail -n 20 – from the first 1000 lines of a log, take the last 20 (so
lines 981-1000 of original).

 cut -f2-5 [Link] | tr '\t' ',' | sort -u – take fields 2-5 from a tab-separated file, convert tabs
to commas (making it CSV of those fields), and output unique lines sorted.

Each filter does a simple transformation, but combined, they perform complex processing in a very
efficient manner (because they stream data through without heavy intermediate storage, generally).

10. Introduction to Shell Script

Shell scripting is writing a series of shell commands in a file to be executed as a program. It allows
automation of tasks, combining commands, and adding logic (conditions, loops, etc.). Here are the
basics of shell scripting concepts:

Creating and Running a Shell Script: A shell script is simply a text file containing shell commands. By
convention, it may have a .sh extension, but that's not required. The first line often is a shebang line,
e.g.:

#!/bin/bash
This tells the system to execute the script using /bin/bash. (It could be #!/bin/sh for a POSIX shell, or
another interpreter like #!/bin/csh for C shell scripts, etc.) After writing the script, you give it execute
permissions (chmod +x [Link]) and then run it either by specifying the path (e.g., ./[Link]).
Alternatively, you can run bash [Link], which doesn’t require the execute bit (because you’re
explicitly running it with bash).

Simple Script Example (non-interactive):

#!/bin/bash

echo "Hello, USER. Today is (date)."

This script greets the user and tells the date, using command substitution for date and the $USER
environment variable.

Interactive Scripts: Shell scripts can also accept user input. One common way is using the read
command. For example:

#!/bin/bash

read -p "Enter your name: " name

echo "Hi, $name!"

The read -p option prints a prompt and waits for input, storing it in the variable (here name).
Interactive scripts might also use command-line arguments or menus.

Command-line Arguments: When a script is executed, arguments given after the script name are
accessible inside the script as special variables:

 $0 is the name of the script.

 1, 2, ... are the first, second, etc., arguments.

 $# is the number of arguments passed.

 @ (or $*) represent all arguments as a list. Typically, use "\@" (in quotes) to handle
arguments with spaces correctly.
For example, consider a script [Link]:

#!/bin/bash
 # add two numbers provided as arguments

if [ $# -lt 2 ]; then

echo "Usage: $0 num1 num2"

exit 1

fi

sum=(( $1 + 2 ))

echo "Sum: $sum"

If you run ./[Link] 5 7, inside the script 1 is 5, $2 is 7, $# is 2. We used an arithmetic expansion $(($1
+ 2)) to compute the sum (more on that below). The script also checks if at least 2 args are provided,
otherwise it shows usage and exits with code 1 (non-zero exit codes typically indicate an error).

Logical Operators in Shell: Shells allow combining commands with && and || which are logical AND
and OR at the command level:

 command1 && command2 – command2 is executed only if command1 succeeds (returns

exit status 0). Example: mkdir /tmp/mydir && cd /tmp/mydir – cd will only run if mkdir
succeeded.

 command1 || command2 – command2 is executed only if command1 fails (returns non-

zero). Example: grep "foo" file || echo "foo not found" – echo will run if grep didn't find
anything (exit status of grep would be 1 in that case).
These can be combined for if-else-like constructs. Also ; can be used to separate commands
to execute sequentially regardless of success.

The if Statement: Shell scripting provides an if conditional. The general form is:

if condition; then

commands

elif other_condition; then

commands

else

commands

fi
A key concept: condition in shell is typically evaluated by a command’s exit status (0 means true,
non-zero means false). Often, the [ (test) command or the test command is used. For example:

if [ -f "/etc/passwd" ]; then

echo "Password file exists."

fi

Here [ is actually a command (a builtin synonym for test) that checks if /etc/passwd is a file (-f flag).
You must close the bracket with ] (with spaces around it!). Common test conditions:

 File checks: -f file (regular file exists), -d file (directory exists), -e file (exists regardless of
type), -r/-w/-x file (readable, writable, executable by you), file1 -nt file2 (file1 newer than
file2), etc.

 String checks: -z string (zero-length string), -n string (non-empty string), str1 = str2 (strings
equal), str1 != str2 (not equal).

 Numeric comparison (within test or [): use -eq, -ne, -lt, -le, -gt, -ge. e.g., [ "$x" -gt 10 ].
So:

if [ $# -eq 0 ]; then

echo "No arguments provided."

elif [ "$1" = "hello" ]; then

echo "Hello to you too!"

else

echo "First arg is $1"

fi

That demonstrates integer comparison and string comparison.

case Statement: This is useful for multi-way branching based on a value matching patterns. Syntax:

case $variable in

pattern1)

commands
 ;;

pattern2)

commands

;;

*)

commands ;; # default case (matches anything else)

esac

For example:

read -p "Continue? (y/n): " answer

case "$answer" in

y|Y|yes|YES)

echo "Continuing...";;

n*|N*) # any input starting with n or N

echo "Quitting."

exit 0 ;;

*)

echo "Invalid response." ;;

esac

This will handle various forms of yes and no. Patterns in case allow the | to mean OR, and ) ends the
pattern. *) is the fallback (like default).

test and [ Commands: As noted, test (or its alias [) is used for conditions. Example usage: test -f file
&& echo "it is a file". In scripts, we almost always use if [ condition ]; then ... fi. Remember to put
spaces: [ and ] are separate tokens. Also, for string comparisons within [ ], you often quote variables
to avoid issues with spaces or empty values.

expr Command: expr is an older utility to evaluate expressions. It can do arithmetic or string
manipulation. Example:
 expr 1 + 2

outputs 3. Or

x=5

y=3

z=(expr $x \* y)

would set z to 15 (note the * must be escaped or quoted in expr because it's a shell wildcard
otherwise). expr can also do things like expr "$str" : '.*' to get length of string, or substring, or regex
match (it's somewhat archaic now).

In modern bash, we rarely need expr for arithmetic because bash has:

 Arithmetic expansion: $(( ... )) which can be used in assignment or echo.

 Example: sum=(( x + y )) or echo (( 5 * 3 )).

This is more convenient. expr still might be used in older scripts or when needing to do some
regex matching in old shells.

Arithmetic in shell: As mentioned, using (( )) is convenient. Also let is a builtin for arithmetic. e.g., let
i=i+1. In bash/zsh, arithmetic context treats variables as numbers automatically. Keep in mind shell
arithmetic handles only integers (no floating point; use bc or awk for that).

Example:

count=0

while [ $count -lt 5 ]; do

echo "Count is $count"

count=$((count + 1))

done

Loops in Shell:

 for loop (list): Iterate over a list of values (words). Syntax:

for var in list; do

 commands

done

Example:

for user in alice bob charlie; do

echo "Processing user $user..."

# commands here

done

If the list is not provided explicitly, for var in "@" will loop over script arguments, or you can do
simply for var; do ... done which implicitly uses "\@".

You can also do numeric loops by generating a list: e.g., for i in {1..10}; do echo i; done (bash brace
expansion) or using seq: for i in (seq 1 10); do ....

 C-style for loop (bash, ksh specific): Bash allows for ((i=0; i<10; i++)); do ... done which works
like C for loop.

 while loop: Repeats as long as a condition is true.

while condition; do

commands

done

The condition is evaluated like an if. Example:

while read line; do

echo "Got line: $line"

done < [Link]

This reads each line of a file until EOF.

Or a numeric example:

i=5
 while [ $i -gt 0 ]; do

echo "$i..."

i=$((i-1))

done

echo "Blastoff!"

 until loop: Opposite of while; it runs until the condition becomes true (i.e., it loops while
condition is false).

until ping -c1 [Link] >/dev/null 2>&1; do

echo "Host not up, sleeping..."

sleep 60

done

echo "Host is up!"

This will ping [Link] until it succeeds, checking every 60 seconds.

Loop control:

 You can use break to exit a loop early, and continue to skip to the next iteration of the loop.

 You can nest loops and use break n to break out of n levels, if needed.

Positional Parameters and @: As discussed, $1, $2, etc., are positional parameters for script
arguments. They are also used in shell functions similarly. $@ represents all arguments as separate
words. If quoted "$@", it expands to "$1" "$2" ... preserving spaces in each argument. *
concatenates them (not usually what you want).

Example in a script:

echo "Script name: $0"

echo "Number of args: $#"

 echo "Args: $@"

If run as ./myscript one "two two" three, you'd get:

 Script name: ./myscript

 Number of args: 3

 Args: one two two three

(notice "two two" appears as separate words unless quoted properly in output; if we did
echo "$@", it would print each argument on separate line perhaps, depending on IFS).

Exit Status and exit: Every command returns an exit status (0 means success). In scripts, you can
explicitly set exit status by exit N where N is 0-255. If a script ends without an explicit exit, its status is
that of the last command executed. You often use exit codes to indicate success or different error
types (e.g., exit 0 for success, exit 1 for general error, maybe exit 2 for usage error, etc., by
convention).

Also within a script or after a command, $? holds the last command's exit status.

Example Script Incorporating Several Constructs:

#!/bin/bash

# Example: Batch rename files given a suffix

suffix="$1"

if [ -z "$suffix" ]; then

echo "Usage: $0 <suffix>"

exit 1

fi

count=0

for file in *"$suffix"; do

if [ -f "$file" ]; then

base="{file%suffix}" # strip the suffix from filename

newname="${base}.bak"
 echo "Renaming file to newname"

mv "file" "newname"

count=$((count+1))

fi

done

echo "Renamed $count files."

This script takes one argument (a suffix to look for), then loops over all files in current directory
ending with that suffix, and renames them by replacing the suffix with .bak. It uses parameter
expansion {file%suffix} to remove the suffix (another shell feature) and uses an if to ensure it's a file.
It keeps a counter of how many it renames.

Shell scripting includes many more features (functions, error handling with set -e or traps, etc.), but
these basics (variables, if/then, loops, read, and such) allow writing fairly powerful scripts to
automate tasks.

11. System Administration

System administration in a UNIX context involves managing the system’s configuration, users,
resources, and ensuring the system operates smoothly. Here are some fundamental duties and
concepts for UNIX system administration:

Duties of a System Administrator: An administrator (often the "root" user or someone with elevated
privileges) is responsible for tasks such as:

 User Account Management: Creating, modifying, and deleting user accounts and groups.
This includes setting up home directories, setting initial passwords, managing user quotas or
privileges.

 Security Management: Setting file permissions appropriately, managing firewall rules,

applying OS patches for security, monitoring logs for suspicious activity, and enforcing
password policies.

 System Monitoring and Performance: Keeping an eye on system load, CPU/memory usage,
disk space, etc. Using tools like top, vmstat, iostat, and logs to ensure the system is healthy.
Tuning the system or processes (renice processes, kill runaway processes) when necessary.

 Process Management: As above, ensuring important services (daemons) are running,

handling crashes or hangs, and understanding the process tree. Possibly writing startup
scripts for services.
  Software Installation and Updates: Installing new software packages or compiling from
source, updating the OS and software packages, managing dependencies.

 Filesystem and Backups: Creating and maintaining file systems (partitioning disks, mounting
filesystems, perhaps LVM or RAID management), monitoring disk usage (cleaning up /tmp,
rotating logs), and setting up regular backups of important data. Recovery procedures (like
restoring from backup or troubleshooting disk failures) also fall here.

 Networking Configuration: Configuring network interfaces, hostnames, routing, DNS

settings, etc., to ensure the system is reachable and can reach others. Possibly NFS mounts or
other network services.

 System Startup/Shutdown: Making sure the system boots correctly into the desired runlevel
or target (on Linux, e.g., multi-user vs graphical), managing services that start at boot. And
performing clean shutdowns or reboots when required (alerting users, syncing filesystems,
etc.).

System Startup Process: When a UNIX system boots:

 First, the machine’s firmware/BIOS/UEFI loads the bootloader from disk (like GRUB). The
bootloader then loads the kernel into memory and hands off control.

 The kernel initializes hardware, memory, mounts the root filesystem, and then starts the first
user-space process, which is typically init (PID 1).

 init (on traditional System V systems) then runs a sequence of scripts as defined in
/etc/inittab and runlevel directories (/etc/rc.d/ or /etc/init.d/ scripts for each runlevel).
These scripts start system services (daemons) like networking, cron, getty on terminals, etc.
Modern Linux systems often use systemd as the init system, which uses .service unit files and
parallelizes startup.

 The system enters a default runlevel/target (e.g., multi-user text mode or graphical mode).
At that point, all key services (like SSH server, web server, database, etc., depending on the
system’s role) should be up.

 On startup, the admin's duty is often to ensure all necessary services start. If a service fails to
start, check its configuration or logs. System administrators may configure which services
start by default (like using chkconfig or systemd enable/disable).

During shutdown or reboot:

 The admin (or system) initiates a shutdown via commands like shutdown, halt, reboot, or in
systemd, systemctl reboot, etc. This sends out warnings (if scheduled) and ultimately signals
init or systemd to begin shutdown.

 The system will typically change to a runlevel that stops regular services (like going to single-
user or runlevel 0/6 for halt/reboot). It sends signals to processes (SIGTERM, then SIGKILL to
any that don't stop) to terminate them cleanly. Daemons stop via their stop scripts or
systemd unit instructions, unmount filesystems (or remount readonly), sync disks, and finally
either halt the CPU or reboot (depending on command).
  The admin must ensure that users are notified if it's a multi-user system (often shutdown -h
+5 "Going down for maintenance" sends a wall message). Also ensure that important data is
saved/flushed to disk and processes like databases have a chance to shut down gracefully.

User Account Management Basics:

 User accounts are defined in /etc/passwd (which contains fields: username, password
placeholder (now usually x if shadow is used), UID, GID, full name or comment, home
directory, shell).

 Passwords are actually stored (encrypted) in /etc/shadow in modern systems (shadow file is
only readable by root).

 Groups are defined in /etc/group (mapping group names to GIDs and listing members).

 To add a user, an admin can manually edit these files, but typically uses commands:

o useradd (or on some systems adduser) to create a new user. This sets up default
things like creating the home directory (often under /home), copying default config
files (from /etc/skel directory), assigning a default shell, etc. Options can set UID,
primary group, additional groups, comment (real name), etc.

o After adding, passwd username is used to set their password (interactive prompt to
enter new password).

o usermod to modify an existing user (change their info, add them to new groups, lock
account, etc.).

o userdel to delete an account (with -r option to remove their home directory if

desired).

o Similarly, groupadd, groupmod, groupdel for groups.

 Administrative accounts: The root user (UID 0) has full control. Sysadmins often also have a
normal user and use su or sudo to run commands as root when needed (to minimize
mistakes).

 Other system accounts: There are often service accounts (e.g., user daemon, www-data for
web server, etc.) that exist to run specific services with limited permissions. These usually
have nologin shells or locked passwords (so no interactive login).

 It's crucial to manage permissions and group memberships so that users have appropriate
access. E.g., maybe create a group projectX and add users to it for a shared directory.

 Login process recap: When a user logs in (via console, or ssh, or GUI), the system uses
/etc/passwd to verify username, then /etc/shadow to verify the password (hashed vs the
stored hash) – or via PAM (pluggable auth modules) nowadays. On success, it sets their UID,
GIDs, changes to their home directory, and starts their shell (from passwd entry).

System Administrator and Root Responsibilities:

  Root can do anything, so security of the root account is paramount (strong password, or
better, disable direct root login and use sudo for accountability).

 Installing system-wide software: depending on OS, using package manager (apt, yum, etc.)
for software, or compiling from source if needed.

 Managing scheduled tasks via cron (scheduled jobs) or systemd timers. Cron jobs for system
tasks are in /etc/crontab or /etc/cron.* directories and often user-specific in /var/spool/cron.

 Backups: setting up backup schedules using tools (tar, rsync, or backup software) to external
drives or tapes. Testing restore procedures.

 System logs: Checking logs in /var/log/ (e.g., syslog or messages, [Link], etc.) to identify
issues or intrusions. Possibly configuring log rotation via logrotate so logs don’t overgrow.

 Startup scripts/services: In System V init, knowing how to enable/disable a service (like

updating /etc/rc*.d links or using chkconfig). In systemd, using systemctl enable servicename
to start at boot, etc. Also writing service unit files if needed or editing them.

System Shutdown/Startup Commands:

 shutdown command: e.g., shutdown -h now (halt now), shutdown -r +5 "Rebooting for
update" (schedule a reboot in 5 minutes with a message). It notifies logged in users and
prevents new logins in the final minute typically.

 Alternatively, halt or poweroff commands to immediately stop, or reboot to reboot (these

often call shutdown under the hood or tell init to change runlevel).

 In modern systems, init 0 means go to runlevel 0 (halt), init 6 means reboot. Or systemctl
isolate [Link] etc. but shutdown is simpler.

 The admin should ensure no users are doing critical work or at least warn them (except in
emergencies).

Summary

The chmod command in UNIX/Linux is used to change file and directory permissions, allowing
control over read, write, and execute access via symbolic (e.g., u+rwx) or octal (e.g., 755) notation
(GeeksforGeeks). It also supports recursive mode (-R) to apply permissions throughout a directory
tree (Super User). The grep command (“global regular expression print”) searches files or streams for
text patterns using basic or extended regular expressions, with options for case-insensitive matching
(-i), whole-word search (-w), recursive search (-r), and more (Built In, GeeksforGeeks). Both
commands are fundamental for system administration and scripting.

chmod Command

Syntax
 chmod [options] mode file…

 mode can be symbolic (e.g., u+rwx) or octal (e.g., 755). (Warp)

 Common options:

o -R — change files and directories recursively (Super User)

o -v — verbose, display files as they’re processed

o -c — like -v, but report only when a change is made

Permission Notation

Symbolic Mode

 u = user (owner), g = group, o = others, a = all

 Operators:

o + add permission (e.g., g+w)

o - remove permission (e.g., o-x)

o = set exact permissions (e.g., u=rwx,g=rx,o=r)

 Permission letters:

o r = read, w = write, x = execute (GeeksforGeeks)

Octal Mode

 Three digits, each 0–7, corresponding to u, g, o

 Bit values: 4 (read), 2 (write), 1 (execute)

 Examples:

o chmod 755 file → rwxr-xr-x

o chmod 644 file → rw-r--r-- (Linux Handbook)

Examples

Command Description

chmod u+x [Link] Add execute permission for the file owner (The Geek Stuff)

chmod go-w [Link] Remove write permission for group and others (GeeksforGeeks)
 chmod u+rwx,go+r [Link] Owner: r/w/x; Group & Others: read only (GeeksforGeeks)

chmod a+x /usr/local/bin/app Add execute for everyone (a = all) (Admin\'s Choice)

chmod -R 750 /var/www/html Recursively set rwxr-x--- on all files/dirs (Super User)

chmod 600 ~/.ssh/id_rsa Owner read/write only (private key) (Linux Handbook)

grep Command

Syntax

grep [options] pattern [file…]

 pattern can be a simple string or a regular expression.

 Default: searches each file and prints matching lines.

Key Options

Option Description

-i Case-insensitive match

-w Match whole words only

-v Invert match (show lines that do not match)

-r / -R Recursively search directories

-n Prefix each matching line with its line number

-c Print a count of matching lines only

-l List only filenames with matches

 -E Use extended regular expressions (same as egrep)

--color=auto Highlight matching strings in output (GeeksforGeeks)

Examples

1. Simple Search

grep "error" /var/log/syslog

Prints lines containing “error” (Built In)

1. Case-Insensitive

grep -i "warning" *.log

Matches “Warning”, “WARNING”, etc. (DigitalOcean)

1. Whole-Word Match

grep -w "server" /etc/hosts

Avoids matching “preserver” (phoenixNAP | Global IT Services)

1. Count Matches

grep -c "TODO" *.c

Shows how many TODOs in each .c file (phoenixNAP | Global IT Services)

1. Recursive Search

grep -R "main" src/

Searches all files under src/ (Ask Ubuntu)

1. Show Line Numbers

 grep -n "Failed login" /var/log/[Link]

Prefixes matches with filename:line (phoenixNAP | Global IT Services)

1. Search Multiple Files

grep "timeout" [Link] [Link] [Link]

Outputs filename and matching lines (phoenixNAP | Global IT Services)

1. Invert Match

grep -v "DEBUG" [Link]

Shows all lines except those with DEBUG (DigitalOcean)

1. Extended Regex

grep -E "^(Error|Warning):" *.log

Matches lines starting with “Error:” or “Warning:” (GeeksforGeeks)

1. Piping and tee

2. dmesg | grep -i usb | tee [Link]

3. Filters dmesg for “usb” (case-insensitive) and saves to [Link] (Built In)