Module 3

File Attributes, File Permissions

Textbook 1: Chapter 6,11,12
2
Basic file attributes
Problems may arise when handling file or directory

UNIX file system is designed to allow users to access files that they do not own, but in a
controlled and secure manner(i.e without infringing on security)

The file has number of attributes that is stored in the inode

ls -l command with additional options are used to display the file attributes

The two basic attributes permissions and ownership are changeable by well defined
rules.

3
Basic file attributes
ls -l listing file attributes
-l option displays most attributes of file like its permissions, size, ownership
ls looks up the file’s inode to fetch its attributes.
ls -l lists the seven attributes of all files in the current directory

4
 Basic file attributes

The list is preceded by the words total 72 which indicates that a total of 72 blocks are
occupied by these files on disk

5
Basic file attributes $ ls –l Links
total 64

drwxr-xr-x 2 user group 4096 Sep 25 12:34 directory1

-rw-r--r-- 1 user group 84 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 5678 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 0 Jan 1 2020 [Link]
1. File type and permissions:
The first column shows the type and permissions associated with each file.
The first character – indicates the file is a ordinary file and it is d for directories
The file can have three types of permission read (r), write (w), execute(x)

2. Links
The number of link associated with the file
This is the number of file names associated with the file though there is a single file on the disk

6
Basic file attributes $ ls -l
total 64
drwxr-xr-x 2 user group 4096 Sep 25 12:34 directory1
-rw-r--r-- 1 user group 84 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 5678 Sep 25 12:34 [Link]
3. Ownership: -rw-r--r-- 1 user group 0 Jan 1 2020 [Link]
The one who creates the file becomes the owner to the file
The owner has the full authority to tamper with the file’s contents and permissions
User can create, modify and remove files in a directory if the user is the owner of the directory

4. Group Ownership
The fourth column represents the group owner of the file.
When opening a user account the administrator assigns the user to some group
Group of users owning a file has acquired importance as group members often need to work on the
same file.
For example, if the group is staff, all users who are members of the staff group will have the
permissions specified for the group.
7
 $ ls -l
Basic file attributes total 64
drwxr-xr-x 2 user group 4096 Sep 25 12:34 directory1
-rw-r--r-- 1 user group 84 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 5678 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 0 Jan 1 2020 [Link]
5. File size:
The fifth column shows the size of the file in bytes i.e. the amount of data it contains
It is only the character count of the file not the disk space that it occupies
The space occupied by file on disk is larger than this figure. (if block size is 1024 bytes
then to store a file of 84bytes 1024 byte is occupied by the file on the disk)

8
 $ ls -l
Basic file attributes total 64
drwxr-xr-x 2 user group 4096 Sep 25 12:34 directory1
-rw-r--r-- 1 user group 84 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 5678 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 0 Jan 1 2020 [Link]
6. Last modification time
The sixth, seventh and eighth column indicates the last modification time of the file
A file is said to be modified only if its contents are changed
The modification time remains unchanged if the ownership or permission of the file is changed
If the file is less than a year old since its last modification time then the year is not displayed.

9
 $ ls -l
Basic file attributes total 64
drwxr-xr-x 2 user group 4096 Sep 25 12:34 directory1
-rw-r--r-- 1 user group 84 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 5678 Sep 25 12:34 [Link]
-rw-r--r-- 1 user group 0 Jan 1 2020 [Link]
7. Filename
The last column gives the file name arranged in ASCII collating sequence (UNIX filenames vary
up to 255 characters).
If the important files to be displayed at the top then select at least the first letter of the file name in
uppercase

10
The –d option: Listing Directory attributes

To force the ls command to list the attributes of a directory, rather than its contents,
once can use –d option.

ls -ld dir1 dir2 dir3

Output:
drwxr-xr-x 2 user user 4096 Sep 25 21:08 dir1
drwxr-xr-x 2 user user 4096 Sep 25 21:08 dir2
drwxr-xr-x 2 user user 4096 Sep 25 21:08 dir3

11
File ownership
When the user creates the file, the user becomes the owner of the file (copy)

User cant create file in other users home directory because those directories are not owned by the user.

Several users may belong to single group.

The privileges of the group are set by the owner of the file and not by the group members-the file
owner has the authority to define what permissions the group members have on that file.

When the system administrator creates a user account, he has to assign these parameters to the user:

The user-id (UID) – both its name and numeric representation (/etc/passwd)

The group-id (GID) – both its name and numeric representation (/etc/group)

12
 File ownership
UNIX follows a three-tiered file protection system that determines a file’s access
rights. It is displayed in the following format:

Filetype owner (rwx) groupowner (rwx) others (rwx)

13
Basic file attributes

r indicates read permission – meaning cat can display the file.

w indicates write permission – meaning lets us edit the file.

x indicates execute permission – meaning the file can be executed as a program.

− shows absence of corresponding permission

14
 File ownership

The first group: owner – has all the permissions – (rwx) read, write, execute
The second group: group − has a hyphen in the middle slot, indicates the absence of write
permission to group owner of the file (r − x) read and execute permission only.

The third group: others – has write and execute bits absent (r - -) read permission only.

NOTE: Group permissions do not apply to the owner of the file though he belongs to the
group. The owner has its own set of permissions that override the group owners
permissions
15
chmod: Changing File Permissions

A file or a directory is created with a default set of permissions, which can be

determined by umask.
$cat /usr/bin/startx > xstart Actually copies the file xstart
$ls -l xstart
-rw-r--r-- 1 kumar ......
Using chmod command, change the file permissions and allow the owner to
execute his file. The command can be used in two ways:
1. In a relative manner by specifying the changes to the current permissions

2. In an absolute manner by specifying the final permissions

16
Relative permissions
chmod only changes the permissions specified in the command line and leaves the other
permissions unchanged.

The syntax is: chmod category operation permission filename(s)

chmod takes an expression as its argument which contains:

– user category (user, group, others)

– operation to be performed (assign or remove a permission)

– type of permission (read, write, execute)

By using suitable abbreviations for each of these components, frame a compact expression and
then use it as a argument to chmod.

User refers to owner.

17
18
 Basic file attributes

To assign execute permission to the user of the file xstart, we need to frame suitable expression by
using appropriate characters from each of the three columns

$ chmod u+x xstart

The command assigns (+) execute (x) permission to user (u) and other permission remains unchanged

19
 Relative permissions
To enable all of them to use the file , use multiple characters to represent the user category (ugo)
chmod ugo+x xstart
chmod a+x xstart //a imples ugo
chmod +x xstart // By default a is implied

-rwx r-x r-x 1 kumar metal 1906 sep 23:38 xstart

chmod accepts multiple file names in command line. The same set of permissions is assigned to a
group of files.
chmod u+x note note1 note3

20
 Relative permissions
• Permissions are removed with – operator

• Let initially,
-rwxr-xr-x 1 kumar metal 1906 sep 23:38 xstart
To remove the read permission from both group and others use the expression go -r

• Initially let us consider,

-rwx r-x r-x 1 kumar metal 1906 sep 23:38 xstart

chmod go-r xstart
Then, it becomes
-rwx--x--x 1 kumar metal 1906 sep 23:38 xstart
• More than one permission can also be set; u+rwx is a valid chmod expression
$chmod o+wx xstart

-rw-r---wx 1 kumar metal 1906 sep 23:38 xstart

21
Relative permissions

$ echo date > [Link] ; ls -l [Link]

-rw-r--r-- 1 kumar group 5 Aug 16 16:05 [Link]

All users have read permission, only the owner has write permission, but the file is
not executable by anyone:
$ ./[Link] Preferred way to run shell scripts
bash: ./[Link]: Permission denied

22
Relative permissions

$ chmod u+x [Link] Execute permission for user

$ ls -l [Link]
-rwxr--r-- 1 kumar group 5 Aug 16 16:05 [Link]
$ ./[Link] Now execute the shell script
Sat Aug 16 [Link] GMT 2003

23
Relative permissions
$ chmod u-x [Link] ; ls -l [Link]
-rw-r--r-- 1 kumar group 5 Aug 16 16:05 [Link]
$ chmod ugo+x [Link] ; ls -l [Link]
-rwxr-xr-x 1 kumar group 5 Aug 16 16:05 [Link]
$ chmod a+x [Link] ; ls -l [Link]
-rwxr-xr-x 1 kumar group 5 Aug 16 16:05 [Link]
$ chmod +x [Link] ; ls -l [Link]
-rwxr-xr-x 1 kumar group 5 Aug 16 16:05 [Link]
$ chmod go-rx [Link] ; ls -l [Link]
-rwx------ 1 kumar group 5 Aug 16 16:05 [Link]
$ chmod u-x,go+r [Link] ; ls -l [Link]
-rw-r--r-- 1 kumar group 5 Aug 16 16:05 [Link]

24
 Absolute Permissions
If the current file permissions are not known then set all nine permissions explicitly.

A string of three octal digits is used as an expression.

The permission can be represented by one octal digit for each category.

For each category, an octal digits is added.

If we represent the permissions of each category by one octal digit, this is how the permission can
be represented

Read permission – 4 (octal 100)

Write permission – 2 (octal 010)

Execute permission – 1 (octal 001)

25
Absolute Permissions

26
Absolute Permissions

If the default file permissions on our system are rw-r--r--. This is octal 644, so
let’s use it with chmod:
$ chmod 644 [Link] ; ls -l [Link]
-rw-r--r-- 1 kumar group 5 Aug 16 16:05 [Link]

If this file must be made unreadable to group and others. For this you need the
expression 600:
$ chmod 600 [Link] ; ls -l [Link]
-rw------- 1 kumar group 5 Aug 16 16:05 [Link]

27
Absolute Permissions

28
Absolute Permissions

A file’s permissions can only be changed by the owner (understood by chmod as

user) of the file.

One user can’t change the protection modes of files belonging to another user.

However, this restriction doesn’t apply to the privileged user, root.

29
Absolute Permissions
Initially, if xstart has the following permission:
$ ls -l xstart
-rw-r--r-- 1 kumar group 5 Aug 16 16:05 [Link]

Using relative permission, we have, chmod a+rw xstart

$ chmod a+rw xstart; ls -l xstart

-rw-rw-rw- 1 kumar group 5 Aug 16 16:05 xstart

Using absolute permission, chmod 666 xstart

$ chmod 666 xstart; ls -l xstart

-rw-rw-rw- 1 kumar group 5 Aug 16 16:05 xstart

30
Absolute Permissions
• Initially, if xstart has the following permission:
$ ls -l xstart
-rw-r-xr-x 1 kumar group 5 Aug 16 16:05 [Link]

Using absolute permission, chmod 644 xstart, we can restore

$ chmod go-x xstart; ls -l xstart

-rw-r-r- 1 kumar group 5 Aug 16 16:05 xstart

$ chmod 644 xstart; ls -l xstart

-rw-r-r- 1 kumar group 5 Aug 16 16:05 xstart

31
Absolute Permissions

chmod 761 xstart

will assign all permissions to the owner, read and write permissions for the group
and only execute permission to the others.

32
Security implications
Let the default permission for the file xstart is -rw-r--r--

chmod u-rw, go-r xstart or

chmod 000 xstart

---------- 1 kumar .......

This is simply useless but still the user can delete this file

We should not set to have all permissions enabled for all categories

chmod a+rwx xstart or

chmod 777 xstart

-rwxrwxrwx

• The UNIX system by default, never allows this situation as you can never have a secure system

33
Using chmod recursively (-R)

It is possible to make chmod descend a directory hierarchy and apply the expression to every
file and subdirectory it finds. This is done by –R option
chmod –R a+x shell_scripts
This makes all files and subdirectories as executables by all users that is found in a tree-walk that
commences from shell_scripts directory.

• To use chmod on your home directory tree, “cd” to it and use it in one of these ways:
chmod -R 755 . // Works on hidden files
chmod -R a+x * // Leaves out hidden files

34
 Directory permissions
Directories also have their own permissions.
The significance of permissions differ a great deal from those of ordinary files.
Read and write access to an ordinary file can also influenced by the permissions of the
directory housing them.
It is possible that a file cannot be accessed even though it has read permission, and can be
removed even when it is write protected.
Default directory permissions are rwxr-xr-x (755)
A directory must never be writable by group and others

35
Directory permissions
A directory stores the filename and inode number.
So the size of a directory is determined by the number of files housed by it and not by
the size of the files.
A directory also has its own set of permissions whose significance differs a great deal from
ordinary files.
A directory’s permissions also affect the access rights of its files.

mkdir progs ; ls -ld progs

drwxr-xr-x 2 kumar metal 512 Aug 16 09:24 progs

36
Read Permission
Read permission for a directory means that the list of filenames stored in that directory
is accessible.
Since ls reads the directory to display filenames, if a directory’s read permission is
removed, ls won’t work.
Consider removing the read permission first from the directory progs:
$ chmod u-r progs
$ ls progs
progs: Permission denied
However, this doesn’t prevent you from reading the files separately if you know their
names.

37
Write Permission
Only the kernel can write to the directory file.
Write permission for a directory implies that you are permitted to create or remove files
in it (that would make the kernel modify the directory entries).
Security issues are usually related to a directory’s write permission.

Now restore the read permission and remove the write permission from the directory before
you try to copy a file to it.
$chmod 555 progs ; ls –ld progs
dr-xr-xr-x 2 kumar metal 512 Aug 16 09:24 progs
$cp [Link] progs
cp: cannot create progs/[Link]: Permission denied

38
 Write Permission
• The write permission for a directory determines whether you can create or remove
files in it because these actions modify the directory.

• Whether you can modify a file depends solely on whether the file itself has write
permission. Changing a file doesn’t modify its directory entry in any way.

• The term “write-protected” has a limited meaning in the UNIX file system. A
write-protected file can’t be written, but it can be removed if the directory has write
permission.

• Danger arises when you mistakenly assign the permissions 775 or 777 to a directory. In
the present scenario, 775 allows any user of the metal group to create or remove files in
the directory. 777 extends this facility to the world. As a rule, you must never make
directories group- or world-writable unless you have definite reasons to do so.

39
Execute Permission
Executing a directory means that a user can “pass through” the directory in searching for
subdirectories.

When you use a pathname with any command:

$cat /home/kumar/progs/[Link]

you need to have execute permission for each of the directories in the pathname.

The directory home contains the entry for kumar, and the directory kumar contains the entry
for progs, and so forth.

If a single directory in this pathname doesn’t have execute permission, then it can’t be
searched for the name of the next directory. That’s why the execute privilege of a directory
is often referred to as the search permission.

40
Execute Permission
A directory has to be searched for the next directory, so the cd command won’t work if the
search permission for the directory is turned off:
$ chmod 666 progs ; ls -ld progs
drw-rw-rw- 2 kumar metal 512 Aug 16 10:11 progs
$ cd progs
bash: cd: progs: Permission denied

As for regular files, directory permissions are extremely important because system security
is heavily dependent upon them. If you tamper with the permissions of your directories,
then make sure you set them correctly.

41
umask: Default File and Directory Permissions
The UNIX system has the following default permissions for all files and directories:
rw-rw-rw- (octal 666) for regular files.
rwxrwxrwx (octal 777) for directories.

However, you don’t see these permissions when you create a file or a directory.

Actually, this default is transformed by subtracting the user mask from it to remove one or
more permissions.

42
umask: Default File and Directory Permissions
To understand what this means, let’s evaluate the current value of the mask by using
umask without arguments:
$ umask
022
This is an octal number which has to be subtracted from the system default to obtain
the actual default.
This becomes 644 (666 – 022) for ordinary files and 755 (777 – 022) for directories.
When you create a file on this system, it will have the permissions rw-r--r--.
A directory will have the permissions rwxr-xr-x.

43
umask: Default File and Directory Permissions
When umask gives an output of 002, it means the following:

Files: The default permissions will be 666 (read and write for everyone) minus 0002,
resulting in 664 (read and write for the owner and group, read-only for others).
Directories: The default permissions will be 777 (read, write, and execute for everyone)
minus 0002, resulting in 775 (read, write, and execute for the owner and group, read and
execute for others).

44
 umask: Default File and Directory Permissions
umask is a shell built-in command. A user can also use this command to set a new default.
Here’s an extreme setting:

umask 000 All read-write permissions on

A umask value of 000 means that you haven’t subtracted anything, and this could be dangerous.

The system’s default then applies (666 for files and 777 for directories).

All files and directories are then writable by all; nothing could be worse than that.

45
 umask: Default File and Directory Permissions
The important thing to remember is that no one—not even the administrator— can use umask
to turn on permissions not specified in the systemwide default settings.

However, you can always use chmod as and when required.

The systemwide umask setting is placed in one of the machine’s startup scripts and is
automatically made available to all users.

The effect of some of these settings on file and directory permissions is shown in Table in next
slide

46
umask: Default File and Directory Permissions

47
Changing File ownership
• When a user kumar of the metal group creates a file foo, kumar becomes the owner of
the file foo, and metal becomes the group owner.
• Only kumar can change the major file attributes like its permissions and group
ownership.

-rw-r--r-- 1 kumar metal 230 May 10 8.:30 foo

• No member of the the metal group (except kumar) can change these attributes.
• However, when sharma copies foo, the ownership of the copy is transferred to sharma,
and he can then manipulate the attributes of the copy.

48
 Changing File ownership
• Two commands meant to change the ownership of a file or directory- chown and chgrp.

• On BSD-based systems, only the system administrator can change file’s owner with
chown.

• On the same systems, the restrictions are less severe when it comes to changing groups
with chgrp. The user can change the group using chgrp to another group to which the
user belongs.

• On other systems, only the owner can change both - chown and chgrp.

49
chown: changing File owner
• Behavior of BSD-based chown that has been adopted by many systems including Solaris
and Linux. The command is as follows:

chown options owner [:group] file(s)

• Chown transfers ownership of a file to a user, and it seems that it can optionally change
the group as well. The command required user-id (UID) of the recipient, followed by one
or more filenames.

• The UID and Group-id(GID) are maintained in /etc/passwd.

50
chown: changing File owner
• Changing ownership requires superuser permissions. So, first change the status to that of
superuser with the su command:
$ su
Password: *******
#_

After the password is successfully entered, su returns a # prompt, the same prompt used by
root.

51
Use chown in following ways:
# ls –l note
-rwxr----x 1 kumar metal 347 May 10 20:30 note
# chown sharma note ; ls –l note
-rwxr----x 1 sharma metal 347 May 10 20:30 note
# exit Switches from super user’s shell to user’s login shell.
$_

• Once the ownership of the file has been given away to sharma, the user file permissions
that previously applied to kumar now applies to sharma.
• Thus, kumar can no longer edit note since there’s no write priveledge for groups and
others.
• He cant get back the ownership either, but he can copy the file to his own directory, in
which he becomes the owner of the copy.

52
 Chgrp: changing group owner
• By default, the group owner of a file is the group to which the owner belongs.
• chgrp command changes a file’s group owner.
• System that implement BSD version of chgrp (like solaris and linux), a user can change the
group owner of a file, but only to a group to which he/she also belongs.
• chgrp shares a similar syntax with chown.
• Kumar changes the group ownership of [Link] to dba (no superuser permission required):
$ ls –l [Link]
-rw-r--r-- 1 kumar metal 139 Jun 8 16:43 [Link]
$ chgrp dba [Link] ; ls –l [Link]
-rw-r--r-- 1 kumar dba 139 Jun 8 16:43 [Link]

53
 Chgrp: changing group owner
• Kumar changes the group ownership of [Link] to dba (no superuser permission required):
$ ls –l [Link]
-rw-r--r-- 1 kumar metal 139 Jun 8 16:43 [Link]
$ chgrp dba [Link] ; ls –l [Link]
-rw-r--r-- 1 kumar dba 139 Jun 8 16:43 [Link]

[Considering, a user can belong to more than one group, and the one shown in /etc/passwd is
the users main group.]
• This command will work on a BSD-based system if kumar is also a member of the dba
group. If not, then superuser can make the command work.
• Kumar can reverse the action and restore the previous group ownership to metal because he
is still owner of the file and consequently retains all rights related to it.

54
Using chown to do both the task
• As an added benefit, UNIX allows the administrator to use only chown to change not
owner and group.
• The syntax requires the two arguments to be separated by a :
$chown sharma:dba [Link] ownership to sharma, group to dba

Like chmod, both chown and chgrp use the –R option to perform their operations in a
recursive manner.

55
 When a file contains 1026 bytes and the disk block size is 1024 bytes, the file will occupy
more than one block.

[Link] Block: The first 1024 bytes of the file will occupy one block.

[Link] Block: The remaining 2 bytes will occupy another block.

Since each block is 1024 bytes, the file will occupy two blocks in total. Therefore, the total
disk space occupied by the file will be:
2×1024=2048 bytes
So, the file will occupy 2048 bytes of disk space.

56
 Show the octal representation of What will the permissions string look like for these
these permissions: (i) octal values? (i) 567,(ii) 623, (iii) 421
rwxr-xrw-(ii) rw-r----- , (iii)
--x-w-r--. 567:
[Link]-xrw-: Owner: 5 (r-x) = r-x
1. Owner: rwx (4+2+1 = 7) Group: 6 (rw-) = rw-
2. Group: r-x (4+0+1 = 5) Others: 7 (rwx) = rwx
3. Others: rw- (4+2+0 = 6) Permissions String: r-xrw-rwx
4. Octal: 756 623:
[Link]-r-----: Owner: 6 (rw-) = rw-
1. Owner: rw- (4+2+0 = 6) Group: 2 (-w-) = -w-
2. Group: r-- (4+0+0 = 4) Others: 3 (-wx) = -wx
3. Others: — (0+0+0 = 0) Permissions String: rw–w–wx
4. Octal: 640 421:
3.–x-w-r–: Owner: 4 (r–) = r–
1. Owner: --x (0+0+1 = 1) Group: 2 (-w-) = -w-
2. Group: -w- (0+2+0 = 2) Others: 1 (–x) = --x
3. Others: r-- (4+0+0 = 4) Permissions String: r—w—x
4. Octal: 124
57
What does a group member require to be able to remove a file?

To remove a file, a group member needs write (w) and execute (x) permissions on the
directory containing the file. This allows them to modify the directory’s contents, which
includes deleting files.

If a file’s permissions are 000, can the superuser still read and write it?

Yes, the superuser (root) can still read and write a file even if its permissions are set
to 000. The superuser has the ability to override file permissions.

58
Assuming that a file’s current permissions are rw-r-xr--, specify the chmod expression
required to change them to (i) rwxrwxrwx, (ii) r--r-----, (iii) ---r--r-- , (iv) ---------, using
both relative and absolute methods of assigning permissions

1. Change to rwxrwxrwx
Absolute Method: chmod 777 filename
Relative Method: chmod u+rwx,g+rwx,o+rwx filename

2. Change to r--r-----
Absolute Method: chmod 440 filename
Relative Method: chmod u-wx,g-wx,o-rwx filename

3. Change to ---r--r--
Absolute Method: chmod 044 filename
Relative Method: chmod u-rwx,g-rwx,o-rwx,u+r,g+r,o+r filename

4. Change to ---------
Absolute Method: chmod 000 filename
Relative Method: chmod u-rwx,g-rwx,o-rwx filename 59
Chapter 11 : More file attributes

60
• The directory stores the inode number along with the filename.

• When we use a command with a filename as argument, the kernel first locates the inode number of
the file from the directory and then reads the inode to fetch data relevant to the file.

• Every file system has a separate portion set aside for storing inodes, where they are laid out in a
contiguous manner. This area is accessible only to the kernel.

• The kernel can locate the inode number of any file using simple arithmetic.

• Since, a UNIX machine usually comprises multiple file systems we can conclude that the inode
number for a file is unique in a single file system.
• ls –i 🡨 gives a inode number of a file

$ls –il tulec05

9059 –rw-r—r-- 1 Kumar Metal 51813 Jan 31 11:15 tulec05

9059 is the inode number of the file tulec05.

No other file in the same file system can have this number unless the file is removed.
When that happens, the kernel will allocate this inode number to a new file.

62
Hard Link
Imagine you have a book in your library. You decide to make a photocopy of this book. Now,
you have two books: the original and the photocopy. Both contain the same content, and if
you lose the original, you still have the photocopy with all the information intact. In the
context of a file system, a hard link is like this photocopy. It is an exact replica of the original
file, sharing the same inode number and data blocks. If the original file is deleted, the hard
link still retains the data.

### Soft Link (Symbolic Link)

Now, think of a soft link as a bookmark or a reference card in a library catalog. This card
points to the location of the book on the shelf. If the book is moved or removed, the card
becomes useless because it only points to where the book was. In a file system, a soft link
is a pointer to the original file. It has a different inode number and only contains the path to
the original file. If the original file is deleted, the soft link becomes a broken link, pointing to
a non-existent file.

63
What are links?
• A connection between a filename and the actual data on the disk
• We can call it shortcut
• We can create a links or shortcut which will point to the actual data

What is the use of links?

• Ease to access
• To rename the files, delete the file etc.
Difference between soft links and hard links

Soft link (symbolic link)

• link will be removed if original file is removed or deleted.
• ln –s #command used to create the soft link

Hard link
• renaming, deleting or removing the file will not effect the link
• ln #command used to create the hard link
Hard links
Why is the filename not stored in the node?
So that the file can have multiple filenames. When that happens, we say the file has more than
one link. We can then access the file by any of its links.

All names provided to a single file have one thing in common; they all have the same inode
number.

The link count is displayed in the second column of the listing.

–rwxr-xr-- 2 kumar Metal 163 Jul 13 21:36 [Link]

–rwxr-xr-- 2 kumar Metal 163 Jul 13 21:36 [Link]
Hard links
All attributes seem to be identical, but the ‘files’ could still be copies (which have different inode
number).
It is the link count that seems to suggest that the “files” are linked to each other.

But this can only be confirmed by using the –i option to ls.

$ ls -li [Link] [Link]

478274 –rwxr-xr-- 2 kumar Metal 163 Jul 13 21:36 [Link]
478274 –rwxr-xr-- 2 kumar Metal 163 Jul 13 21:36 [Link]

Both “files” indeed have the same inode number, so there is actually only one file with a single
copy on disk.

This file simply has two aliases; changes made in one alias (link) are automatically available in the others.

67
Hard links
ln: creating hard links
File is linked with a ln (link) command, which takes two filenames as arguments.
We can create two links: hard and softlink and has similar syntax to the one used by cp.
ln [Link] employee 🡨 employee must not exist

The –i option to ls shows that they have a same inode number, meaning that they are actually
one and the same file.
$ ln [Link] employee; ls –li emp*
29518 –rwxr-xr-- 2 kumar Metal 915 May 4 09:58 [Link]
29518 –rwxr-xr-- 2 kumar Metal 915 May 4 09:58 employee

$ ln employee [Link]; ls –l emp*

29518 –rwxr-xr-- 3 kumar Metal 915 May 4 09:58 [Link]
29518 –rwxr-xr-- 3 kumar Metal 915 May 4 09:58 [Link]
29518 –rwxr-xr-- 3 kumar Metal 915 May 4 09:58 employee
Hard links
$ ln employee [Link];
Here ln won’t work if the destination filename exists, but you can force linking with the -f
option.
You can also use ln with multiple files (i.e., create a link for each), but then the destination
filename must be a directory.
Here’s how you create links for all shell scripts in the directory shell_scripts:

ln *.sh shell_scripts

If *.sh matches 27 filenames, then there will be 27 linked filenames in shell_scripts; i.e., there
will be 27 entries in that directory.
Hard links

The rm command removes a file by deleting its directory entry, so we expect the same
commands to remove a link below.

$ rm [Link]; ls –l [Link] employee

–rwxr-xr-- 2 kumar Metal 915 May 4 09:58 [Link]
–rwxr-xr-- 2 kumar Metal 915 May 4 09:58 employee

The link count has come down to two. A file is considered to be completely removed from the
system when its link count drops to zero.
Hard links
rm and ln are complementary, which is evident from the names of the system calls they
use—unlink and link. The effect they have on the inode and directory is depicted in Fig. 4.4.

After the invocation, the kernel performed two tasks:

• It added an entry to the directory for the filename [Link], but with the same inode number.
• It also updated the link count in the inode from one to two. Note that there’s actually one file and
71
that we can’t refer to them as two
Hard links
Where to Use Hard Links
Links are an interesting feature of the file system, but where does one use them? We can think of
three situations straightaway:

1. Let’s assume that you have written a number of programs that read a file [Link] in
$HOME/input_files. Later, you reorganize your directory structure and move [Link] to
$HOME/data instead. What happens to all the programs that look for [Link] at its original
location?

Simple, just link [Link] to the directory input_files:

ln data/[Link] input_files Creates link in directory input_files

With this link available, your existing programs will continue to find [Link] in the input_files
directory. It’s more convenient to do this than to modify all programs to point to the new path.

72
Hard links
2. Links provide some protection against accidental deletion, especially when they exist in
different directories. Referring to the previous application, even though there’s only a single file
[Link] on disk, you have effectively made a backup of this file. If you inadvertently delete
input_files/[Link], one link will still be available in data/[Link]; your file is not gone yet.

3. Because of links, we don’t need to maintain two programs as two separate disk files if
there is very little difference between them. A file’s name is available to a C program (as
argv[0]) and to a shell script (as $0). A single file with two links can have its program logic
make it behave in two different ways depending on the name by which it is called.

73
ln Again: Creating Symbolic Links
Two serious limitations of hard links:

• You can’t link a file across two file systems. In other words, if input_files and data are on
two separate file systems, you can’t connect a file in one file system with a link to the other.
This is quite obvious because a hard link takes only an inode number. There’s no way of
knowing which file system the inode number refers to.

• You can’t link a directory even within the same file system.

A symbolic link overcomes both problems. Until now, we have divided files into three
categories (ordinary, directory, and device); the symbolic link is the fourth file type.

74
ln Again: Creating Symbolic Links
A symbolic link is identified by the l (el) as the file type and the pointer notation, ->, that
follows the filename.
The ln command creates symbolic links also, but it needs the -s option. We can create a
symbolic link to [Link], but this time the listing tells you a different story:

$ ln -s [Link] [Link]
$ ls -li [Link] [Link]
254414 -rw-r--r-- 1 kumar metal 5 Aug 16 09:38 [Link]
254411 lrwxrwxrwx 1 kumar metal 7 Aug 18 06:52 [Link] -> [Link]

Here, [Link] is a symbolic link to [Link]. Unlike a hard link, a symbolic link is a separate
file with its own inode number.
[Link] simply contains the pathname [Link] as is evident from the file size ([Link] contains
seven characters).

The two files are not identical; it’s [Link] that actually has the contents.
A command like cat [Link] follows the symbolic link and displays the file the link points to.
75
ln Again: Creating Symbolic Links
To return to the problem of linking a hundred files in the directory data, you can use ln to
connect data to a symbolic link named input_files:

ln -s data input_files First argument is a directory

Think twice before you delete the file or directory that a symlink points to. For instance,
removing [Link] (considered in a previous example) won’t affect us much because we can
easily re-create the link. But if we remove [Link], we will lose the file containing the data:
$ rm [Link] ; cat [Link]
cat: cannot open [Link]

A disaster of greater magnitude could occur if we removed data instead of input_files. We

would then lose all one hundred files! In either case, [Link] and input_files would point to a
nonexistent file. These links are known as dangling symbolic links.
76
When the original file is deleted, the behavior of hard links and soft links (symbolic
links) differs significantly:

• Hard Link: • Soft Link (Symbolic Link):

• A hard link points directly to the inode • A soft link is a pointer to the pathname
of the file, which means it references the of the original file.
actual data on the disk.
• If the original file is deleted, the soft
• If the original file is deleted, the hard link becomes a “dangling” link,
link still retains access to the data meaning it points to a non-existent file.
because the data itself is not removed
until all hard links to it are deleted. • Attempting to access the file through the
soft link will result in an error, as the
• Essentially, the file content remains target file no longer exists
accessible through the hard link, and the
file appears as if it was never deleted.

77
Modification and access times
Three time stamps associated with UNIX files are:
• Time of last file modification ls –l
• Time of last access ls –lu
• Time of last inode modification ls -lc
Whenever you write to a file, the time of last modification is updated in the file’s inode.
A directory can be modified by changing its entries—by creating, removing and renaming files
in the directory.
Note: changing a files contents only changes its last modification time but not that of its
directory. ls –l displays the last modification time.
A file also has an access time., i.e, the last time someone read, wrote or executed the file.
For a directory, the access time is changed by a read operation only, creating or removing a file
or doing “cd” to a directory doesn’t change its access time. The access time is displayed when
ls –l is combined with the –u option.
Modification and access times(cont’d)
Even though ls –l and ls –lu show the time of last modification and access, respectively, The
sort order remains standard, i.e ASCII.
However when you add –t option to –l or –lu, the files are actually displayed in order of the
respective time stamps.

ls –lt displays listing in order of their modification time

ls –lut displays listing in order of their access time.
Touch: changing the time stamps
In Unix/Linux systems, files have several timestamps that record different types of activities.

Last Modification Time (mtime): This is the time when the contents of the file were last
modified. For example, if you edit a text file and save the changes, the modification time
will be updated.

Last Access Time (atime): This is the time when the file was last accessed, meaning it was
read or executed. Simply opening a file to view its contents will update the access time.
Touch: changing the time stamps
touch options expression filename (s)
• When touch is used without options or expression, both times are set to the current time.
The command creates the file if it doesn’t exist
touch [Link] creates a file If it doesn’t exist
• When touch is used without options but with expression, it changes both times.
Touch: changing the time stamps

• Setting Specific Timestamps:

• You can set a specific timestamp using the -t option:
touch -t 202410081230 [Link]

• This sets the timestamp to October 8, 2024, 12:30 PM.

• To update the access time to the current time:

• touch -a [Link]

• To update the modification time:

• touch -m [Link]

82
Find: locating files
• Find is one of the power tools of the UNIX system. It recursively examines a directory
tree to look for files matching some criteria and then takes some action on the selected
files.

• Find is easily tamed if you break up its arguments into three components:
find path_list selection_criteria action
Find: locating files

• The command completely examines a directory tree in this way:

• First, it recursively examines all files in the directories specified in path_list.
Here, it begins the search from /home.
• It then matches each file for one or more selection_criteria. This always consists
of an expression in the form -operator argument (-name [Link]). Here, find
selects the file if it has the name [Link]. Finally, it takes some action on
those selected files.
• The action -print simply displays the find output on the terminal. All find
operators (are also referred to as options)

84
Find: locating files
Three components: find path_list selection_criteria action
path_list comprises one or more subdirectories separated by whitespace.
selection_criteria is used to match a file
Multiple actions to dispose of the file.

$ find / -name [Link] –print

The path_list (/) indicates that the search should start from the root directory. Each file is
matched against the selection criteria (-name [Link]), which always consists of an expression in
the form
-operator argument. If the expression matches the file (the file has a name [Link]), the file is
selected. The third section specifies the action (-print) to be taken on the files; in this case, a
simple display on the terminal. All find operators start with a - , and the path_list can never
contain one

85
Find: locating files
• We can use relative names (like . )in the path list, and find will then output a list of relative
pathnames.
• When find is used to match a group of filenames with a wild-card pattern, the pattern
should be quoted to prevent the shell from interpreting it before find gets a chance to
process it.
find . -name “*.c” –print All files with extension .c

find . -name ‘[A-Z]*’ –print single quotes will also do

The first command looks for all C program source files in the current directory tree. The
second one searches for all files whose names begin with an uppercase letter.
-name is not the only operator used in framing the selection criteria; there are many others
(table 11.1(refer next slide))
88
Selection criteria

Locating a File by Inode Number (-inum)

Find allows us to locate files by their inode number. Use the -inum option to find all filenames
that have the same inode number:

$ find / -inum 29518 –print

29518 –rwxr-xr-- 3 kumar Metal 915 May 4 09:58 [Link]

29518 –rwxr-xr-- 3 kumar Metal 915 May 4 09:58 [Link]

89
Selection criteria
File Type and Permissions (-type and -perm) The -type option followed by the letter f, d,
or l selects files of the ordinary, directory, and symbolic link type.
Here’s how you locate all directories of your home directory tree:
$ cd ; find . -type d -print 2>/dev/null
.
./.netscape
./java_progs
./c_progs
./c_progs/include
./.ssh
Here’s a breakdown of the command:
find .: Starts the search from the current directory (.).
-type d: Restricts the search to directories (d).
-print: Prints the full path of each directory found.
2>/dev/null: Redirects any error messages (file descriptor 2) to /dev/null, effectively
silencing them.
90
Selection criteria
The -perm option specifies the permissions to match. For instance, -perm 666 selects files having
read and write permission for all user categories.
You can use two options in combination to restrict the search to only directories:
find $HOME -perm 777 -type d –print

find uses an AND condition (an implied -a operator between -perm and -type) to select directories
that provide all access rights to everyone.

It selects files only if both selection criteria (-perm and -type) are fulfilled.

91
Selection criteria
Finding Unused Files (-mtime and -atime) Files tend to build up incessantly on disk. Some
of them remain unaccessed or unmodified for months—even years.
find’s options can easily match a file’s modification (-mtime) and access (-atime) times to
select them.
The -mtime option helps in backup operations by providing a list of those files that have been
modified, say, in less than two days:
find . -mtime -2 -print
Here, -2 means less than two days.
To select from the /home directory all files that have not been accessed for more than a year, a
positive value has to be used with -atime:
find /home -atime +365 -print
+365 means greater than 365 days; -365 means less than 365 days. For specifying exactly
365, use 365.

92
Selection criteria
Files Modified Exactly 3 Days Ago:
find . -type f -mtime 3 This command finds files that were modified exactly 3 days ago.

Files Modified More Than 7 Days Ago:

find . -type f -mtime +7 This command finds files that were modified more than 7 days ago.
Files Modified Within the Last 2 Days:
find . -type f -mtime -2 This command finds files that were modified within the last 2 days.

Combining -mtime Options

You can combine -mtime options to specify a range of days. For example:

find . -type f -mtime +2 -mtime -5

This command finds files that were modified more than 2 days ago but less than 5 days ago.

93
Find operators (!, -o and –a)
The find Operators (!, -o, and -a)
There are three operators that are commonly used with find.
The ! operator is used before an option to negate its meaning. So,
find . ! -name “*.c” –print selects all but the C program files.

To look for both shell and perl scripts, use the –o operator, which represents an OR condition.
We need to use an escaped pair of parentheses here:
find /home \( -name “*.sh” -o -name “*.pl” \) -print

The ( and ) are special characters that are interpreted by the shell to run commands in a group.
The same characters are used by find to group expressions using the -o and -a operators, the
reason why they need to be escaped.
The -a operator represents an AND condition, and is implied by default whenever two selection
criteria are placed together.
94
Option available in action component
Displaying the Listing (-ls) The -print option belongs to the action component of the find syntax.
In real life, you’ll often want to take some action on the selected files and not just display the
filenames.
For instance, you may want to view the listing with the -ls option:
$ find . -type f -mtime +2 -mtime -5 -ls
475336 1 -rw-r--r-- 1 romeo users 716 Aug 17 10:31 ./c_progs/fileinout.c

find . : Starts the search from the current directory (.)

-type f: Restricts the search to regular files (f).
-mtime +2: Finds files modified more than 2 days ago.
-mtime -5: Finds files modified less than 5 days ago.
-ls: Lists the details of each file found, similar to the ls -l command.

Example Scenario
If today is October 9th, 2024, this command will find files that were modified between October 4th,
2024, and October 7th, 2024. In this example, we see two options in the selection criteria (both
-mtime) simulating an AND condition. It’s the same as using \( -mtime +2 -a -mtime -5 \).
95
Option available in action component
Taking Action on Selected Files (-exec and -ok) The -exec option allows you to run any
UNIX command on the selected files.
-exec takes the command to execute as its own argument, followed by {} and finally the
rather cryptic symbols \; (backslash and semicolon).
This is how you can reuse a previous find command quite meaningfully:
find $HOME -type f -atime +365 -exec rm {} \;
find $HOME: Starts the search from your home directory.
-type f: Restricts the search to regular files.
-atime +365: Finds files that were last accessed more than 365 days ago.
-exec rm {} \; : Executes the rm command to delete each file found. The {} is a placeholder
for the current file, and \; indicates the end of the -exec command.
This will use rm to remove all ordinary files unaccessed for more than a year. This can be a
risky thing to do, so you can consider using rm’s -i option.

96
Option available in action component

But not all commands have interactive options, in which case you should use find’s -ok
option:
$ find $HOME -type f -atime +365 -ok mv {} $HOME/safe \;
< mv ... ./[Link] > ? y
< mv ... ./[Link] > ? n
< mv ... ./[Link] > ? y

97
Selection criteria
Selection criteria (cont’d)
Find operators (!, -o and –a)
Option available in action component
Option available in action component (cont’d)
Chapter 12: Simple filters
 Filters
• Filters are text manipulation tools available in UNIX.
• Filters—commands that use both standard input and standard output.
• Filters are different from other commands in that their power lies, not in the standalone
mode, but when used in combination with other tools.
• A filter has the capability of reading from standard input and writing to standard output. By
default a filter writes to standard output. It reads from standard input when used without a
filename as argument, and from the file otherwise.
• The piping mechanism of the shell lets the standard output of one filter act as the standard
input of another. This feature lets us design pipelines containing a series of filters.

104
Consider a file [Link]
• Each line of this file has six fields separated by five delimiters. The details of an employee
are stored in one single line. This text file designed in fixed format and containing a
personnel database. There are 15 lines, where each field is separated by the delimiter |

$ cat [Link]
1006 | chanchal singhvi | director | sales | 03/09/38 | 6700
6213 | karuna ganguly | g.m. | accounts | 05/06/62 | 6300
1265 | s.n. dasgupta | manager | sales | 12/09/63 | 5600
4290 | jayant choudhury | executive | production | 07/09/50 | 6000
2476 | anil aggarwal | manager | sales | 01/05/59 | 5000
6521 | lalit chowdury | directir | marketing | 26/09/45 | 8200

105
pr: Paginating Files
• The pr command prepares a file for printing by adding suitable headers, footers, and
formatted text.
• When used with a filename as argument, pr doesn’t behave like a filter.
$cat [Link]
01|accounts|6213
02|progs|5423
03|marketing|6521
04|personnel|2365
05|production|9876
06|sales|1006

106
 pr: Paginating Files
• pr command adds suitable headers, footers and formatted text. pr adds five lines of margin
at the top and bottom. The header shows the date and time of last modification of the file
along with the filename and page number.

$pr [Link]
May 06 10:38 1997 [Link] 1

01| accounts | 6213

02 | progs | 5423
03 | marketing | 6521
04 | personnel | 2365
05 | production | 9876
06 | sales | 1006
…blank lines…
107
pr options

108
pr options
• The -k option (where k is an integer) prints in k columns.
• If a program outputs a series of 20 numbers, one in each line, then this option can make good
use of the screen’s empty spaces.
• Let’s use pr as a filter this time by supplying its input from the standard output of another
program

• The -t option suppresses headers and footers. If you are not using this option, then you can
use the -h option (followed by a string) to have a header of your choice

109
pr options
$pr +10 chap01 starts printing from 10
$pr -l 54 chap01 Page set to 54 lines
$ pr -t -n -d -o 10 group1 -t option suppresses headers and footers,
-d Double-spaces input, reduces clutter,
-n Numbers lines, which helps in debugging code.
-o n Offsets lines by n spaces, increases left margin of page.

110
• The tee command in Unix is used to read from standard input and write to both standard output and
one or more files simultaneously.
• It allows you to redirect the output of a command or a series of commands to a file while still
displaying it on the terminal.
• The basic syntax of the tee command is as follows:

command | tee [OPTION]... [FILE]...

command: The command whose output you want to redirect.

tee: The command itself.
[OPTION]: Optional flags that modify the behavior of tee.
[FILE]: Optional file names where the output should be written. If no files are specified, tee will only
write to the standard output.

ls | tee [Link]
-a (or --append): Appends the output to the specified files rather than overwriting them.
ls | tee -a [Link]
head: Displaying the Beginning of a File
• The head command displays the top of the file. By default, it displays the first 10 lines:
$ head group1 Shows first 10 lines

• You can use the -n option (POSIX mandated) to specify a line count and display, say, the
first three lines of the file:
$ head -n 3 [Link] Or head -3 group1 on some systems

112
head: Displaying the Beginning of a File

• head can be used in imaginative ways. Consider that you are resuming an editing session
the next day and find that you cannot recall the name of the file you last edited.
• Since ls -t displays files in order of their modification time, picking up the first file from
the list and using it as an argument to the vi editor should do the job.
• This requires command substitution:
• vi `ls -t | head -n 1` Opens last modified file for editing

113
 tail: Displaying the End of a File
• Complementing its head counterpart, the tail command displays the end of the file.
• It provides an additional method of addressing lines, and like head it displays the last 10 lines
when used without arguments.
• The last three lines are displayed in this way:
$ tail -n 3 [Link] Or use tail -3 [Link]

• Some versions of UNIX limit the size of the segment that can be extracted by tail with the -n
option. To get over this problem, you can address lines from the beginning of the file instead of
the end.
• The +count option allows you to do that, where count represents the line number from where the
selection should begin. If a file contains 1000 lines, selecting the last 200 implies using

• $ tail +801 foo 801th line onwards, possible with + symbol

114
tail: Displaying the End of a File
• Monitoring File Growth (-f) Many UNIX programs constantly write to the system’s log
files as long as they are running. The tail -f option in UNIX is a powerful tool for
monitoring file growth in real-time.

• When you use tail -f filename, it continuously displays the last part of the file and updates
the output as new data is appended to the file. It's particularly useful for keeping an eye on
log files and debugging live issues as they happen.

• System administrators need to monitor the growth of these files to view the latest messages.

• tail offers the -f (follow) option for this purpose.

• Use tail –f when we are running a program that continuously writes to a file, and we want
to see how the file is growing. We have to terminate this command with the interrupt key

115
tail: Displaying the End of a File
Extracting Bytes Rather Than Lines (-c):
• When you use tail -c -512 foo, you're essentially telling the system to read and display the
last 512 bytes of the file named "foo".

• This is especially useful when you need to check the latest portion of a file without scrolling
through the entire thing, which can be a lifesaver for massive logs or data files.

$tail -c -512 foo Copies last 512 bytes from foo

$tail -c +512 foo Copies everything after skipping 511 bytes

116
cut: Slitting a File Vertically
• While head and tail are used to slice a file horizontally, you can slice a file vertically with the
cut command. cut identifies both columns and fields.

• Cutting Columns (-c) To extract specific columns, you need to follow the -c option with a list
of column numbers, delimited by a comma. Ranges can also be specified using the hyphen.
Here’s how we extract the first four columns of the group file:

$ cut -c1-4 emp -c or -f option always required

• Note that there should be no whitespace in the column list. Moreover, cut uses a special form
for selecting a column from the beginning and up to the end of a line:
$ cut -c -3,6-22,28-34,55- foo Must be an ascending list

• The expression 55- indicates column number 55 to the end of the line. Similarly, -3 is the same
as 1-3.
117
cut: Slitting a File Vertically
• The cut command with the -f option in UNIX is used for cutting out specific fields from lines
of a file, typically used with tabular data where fields are separated by delimiters (commonly
tabs or spaces).
• For example, if you have a file [Link] with tab-separated values, and you want to extract the
first and third fields, you’d use:

$cut -f 1,3 [Link]

• You can specify the delimiter with the -d option if it’s something other than a tab. For
instance, if your fields are comma-separated:

$cut -d ',' -f 1,3 [Link]

• This command helps you isolate and extract specific parts of each line, making it a powerful
tool for data processing.

118
cut: Slitting a File Vertically
• Extracting User List from who Output
• cut can be used to extract the first word of a line by specifying the space as the delimiter.
• The command who displays information about the users currently logged into the system.

$ who | cut -d“ ” -f1 Space is the delimiter

• When combined with cut –d“ ” -f1, it extracts the first field (the username) from each line
of the who output.

119
paste: Pasting Files
• What you cut with cut can be pasted back with paste—but vertically rather than horizontally.
You can view two files side-by-side by pasting them:
$paste foo1 foo2

• Like cut, paste also uses the -d option to specify the delimiter, which by default is also the tab.
paste has fewer applications than cut.

120
paste: Pasting Files
• If cut was used to create the two files cutlist1 and cutlist2 containing two cut-out portions of
the same file.
$paste cutlist1 cutlist2
• We can specify one or more delimiters with -d
$paste -d “|” cutlist1 cutlist2
• here each field will be separated by the delimiter |. Even though paste uses at least two files for
concatenating lines, the data for one file can be supplied through the standard input.

121
paste: Pasting Files
Option (-s), can be used to join lines.
Consider this address book that contains details of three persons, with three lines for each:

$ cat addressbook
barry wood
woodb@[Link]
245-690-4004
charles harris
charles_harris@[Link]
345-865-3209
james wilcocks
$ paste -s addressbook would join all of [Link]@[Link]
these nine lines to form a single line. 190-349-0743

122
paste: Pasting Files
• If we specify the delimiter string as ::\n with -d, the delimiters are used in a circular
manner.
• The first and second lines would be joined with the : as delimiter, and the same would be
true for the second and third lines. The third and fourth lines would be separated by a
newline.
• After the list is exhausted it is reused. This is exactly what we want:

$ paste -s –d“::\n” addressbook

barry wood:woodb@[Link]-690-4004
charles harris:charles_harris@[Link]-865-3209
james wilcocks:[Link]@[Link]-349-0743

123
sort: Ordering a File

• This is a text file containing five lines of a personnel database:

$ cat shortlist

2233:charles harris :g.m. :sales :12/12/52: 90000

9876:bill johnson :director :production :03/12/50: 130000
5678:robert dylan :d.g.m. :marketing :04/19/43: 85000
2365:john woodcock :director :personnel :05/11/47: 120000
5423:barry wood :chairman :admin :08/30/56: 160000

124
sort: Ordering a File
• Sorting is the ordering of data in ascending or descending sequence. In UNIX, we use the sort
command to sort complete lines or parts of them by specifying one or more keys.

• Consider the important sort options by sorting the file shortlist. This is a text file containing
five lines of a personnel database:

• Each line has six fields delimited by a :. The details of an employee are stored in each line. A
person is identified by emp-id, name, designation, department, date of birth, and salary (in the
same order).

125
sort: Ordering a File
By default, sort reorders
lines in ASCII collating
sequence—whitespace first,
then numerals, uppercase letters,
and finally lowercase letters:

• Here, sorting starts with the first character of each line and proceeds to the next character only
when the characters in two lines are identical. Using options, you can alter the default
ordering sequence and sort on multiple keys (fields).
126
sort: Ordering a File

127
sort: Ordering a File
sort Options

• Sorting on Primary Key (-k) Let’s now use the -k option to sort on the second field
(name). The option should be -k 2:
$ sort -t: -k 2 shortlist

128
sort: Ordering a File

The sort order can be reversed

with the -r (reverse) option.
The following sequence reverses
the previous sorting order:

129
Sorting on Secondary Key
• You can sort on more than one key; i.e., you can provide a secondary key to sort. If the
primary key is the third field, and the secondary key the second field, then you need to
specify for every -k option where the sort ends.

• This is done in this way:

This sorts the file by designation and

name.
The -k 3,3 option indicates that sorting
starts on the third field and ends on the
same field.

130
Sorting on Columns
• You can also specify a character position within a field to be the beginning of a sort.
• If you are to sort the file according to the year of birth, then you need to sort on the seventh
and eighth column positions within the fifth field:

• The -k option also uses the form -k m.n where n is the character position in the mth field.
131
So, -k 5.7,5.8 means that sorting starts on column 7 of the fifth field and ends on column 8.
Numeric Sort (-n)
• When sort acts on numericals, strange things can happen.
• When we sort a file containing only numbers, we get a curious result. This can be
overridden by –n (numeric) option.

$ sort -t":" -k1,3 [Link] cat > [Link]

1:A 1:A
201:E 2:B
22:D 30:C
2:B 22:D
30:C 201:E

$ sort -t":" -k1,3 -n [Link]

1:A
2:B
22:D
30:C
132
201:E
Removing Repeated Lines (-u)
• The -u (unique) option lets you remove repeated lines from a file. To find out the unique
designations that occur in the file, cut out the designation field and pipe it to sort:

133
Merge-sort (-m)
• When sort is used with multiple filenames as arguments, it concatenates them and sorts
them collectively.

• When large files are sorted in this way, performance often suffers.

• The -m (merge) option can merge two or more files that are sorted individually:

sort -m foo1 foo2 foo3

• This command will run faster than the one used without the -m option only if the three
files are sorted.

134
Saving sort Output (-o)
• Even though sort’s output can be redirected to a file, we can use its -o option to specify the
output filename. Curiously, the input and output filenames can even be the same:

sort -o sortedlist -k 3 shortlist Output stored in sortedlist

sort -o shortlist shortlist Output stored in same file

The -c option with sort checks whether a file is sorted.

sort –c shortlist

If the file is sorted, there’s no output; if it’s not, sort -c reports the first instance where the
sorting order is violated.

135
uniq: Locate Repeated and Nonrepeated Lines
• When you concatenate or merge files, you’ll face the problem of duplicate entries creeping
in. You saw how sort removes them with the -u option.
• UNIX offers a special tool to handle these lines—the uniq command. Consider a sorted file
[Link] that includes repeated lines:

$ cat [Link]
01:accounts:6213
01:accounts:6213
02:admin:5423
03:marketing:6521
03:marketing:6521
03:marketing:6521
04:personnel:2365
05:production:9876
06:sales:1006 136
uniq: Locate Repeated and Nonrepeated Lines
• uniq simply fetches one copy of each line and writes it to the standard output:
$ uniq [Link]

137
uniq: Locate Repeated and Nonrepeated Lines
• Since uniq requires a sorted file as input, the general procedure is to sort a file and pipe its
output to uniq.
• The following pipeline also produces the same output, except that the output is saved in a
file:
sort [Link] | uniq > uniqlist

• uniq is indeed unique; when provided with two filenames as arguments, uniq reads the
first file and writes to the second.

• Here, it reads from standard input and writes to uniqlist.

138
uniq Options
• To select unique lines, it’s preferable to use sort -u, which does the job with a single
command.
• But uniq has a few useful options that can be used to make simple database queries.

• Selecting the Nonrepeated Lines (-u) The -u option selects the unique lines in
input—lines that are not repeated:

139
uniq Options
• Selecting the Duplicate Lines (-d) The -d (duplicate) option selects only one copy of the
repeated lines:

140
uniq Options
• Counting Frequency of Occurrence (-c) The -c (count) option displays the frequency of
occurrence of all lines, along with the lines:

• This is an extremely useful option, and it is often used in tandem with sort to count
occurrences.

141
uniq Options
• Consider the file shortlist where the third field represents the designation.
• To determine the number of people having the same designation, first cut out the third field
with cut, sort it with sort, and then run uniq -c to produce a count:

142
uniq Options
• Like sort, uniq also accepts the output filename as an argument, but without using an
option (unlike -o in sort).
• If you use uniq foo1 foo2, uniq simply processes foo1 and overwrites foo2 with its
output. Never use uniq with two filenames unless you know what you are doing.

143
tr: Translating Characters
• The tr (translate) filter manipulates individual characters in a line. More specifically, it
translates charactersusing one or two compact expressions:

tr options expression1 expression2 standard input

• Note that tr takes input only from standard input; it doesn’t take a filename as argument.
• By default, it translates each character in expression1 to its mapped counterpart in expression2.
• The first character in the first expression is replaced with the first character in the second
expression, and similarly for the other characters.

144
tr: Translating Characters
• Let’s use tr to replace the : with a ~ (tilde) and the / with a -. Simply specify two expressions
containing these characters in the proper sequence:

• Note that the lengths of the two expressions should be equal. If they are not, the longer
expression will have unmapped characters (not in Linux).
• Single quotes are used here because no varia ble evaluation or command substitution is
involved.

145
tr: Translating Characters
• It’s just as easy to define the two expressions as two separate variables, and then evaluate
them in double quotes
exp1=’:/’ ; exp2=’~-’
tr “$exp1” “$exp2” < shortlist

• Since tr does not accept a filename as argument, the input has to be redirected from a file or
a pipe.
• The following sequence changes the case of the first three lines from lower to upper:
head –n 3 [Link] | tr ‘[a-z]’ ‘[A-Z]’

Or tr '[a-z]' '[A-Z]' < [Link] | head -n 3

146
tr: Translating Characters
• Using ASCII Octal Values and Escape Sequences Like echo, tr also supports the use of
octal values and escape sequences to represent characters.
• This facility allows us to use nonprintable characters (like LF) in the expression. So, to have
each field on a separate line, replace the : with the LF character (octal value 012):

147
tr: Translating Characters
tr Options
• Deleting Characters (-d) The file shortlist has delimited fields and the date formatted in
readable form with a /.
• In nondatabase setups, delimiters are not used, and the date is generally represented as a
six-character field in the format mmddyy.
• To convert this file to the traditional format, use the -d (delete) option to delete the
characters : and /from the file.
• The following command does it for the first three lines:

148
tr: Translating Characters
• Compressing Multiple Consecutive Characters (-s) UNIX tools work best with fields
rather than columns (as above), so it’s preferable to use files with delimited fields.
• In that case, lines need not be of fixed length; you can eliminate all redundant spaces with
the -s (squeeze) option, which squeezes multiple consecutive occurrences of its argument to
a single character.
• We can then have compressed output with lines in free format:

149
tr: Translating Characters
• Complementing Values of Expression (-c) Finally, the -c (complement) option complements
the set of characters in the expression.
tr -c ‘[ A-Za-z0-9./ ]’ ‘$’ < shortlist
• 2233$charles harris $g.m. $sales $12/12/52$ 90000$9876$bill johnson $director $production
$03/12/50$ 130000$5678$robert dylan $d.g.m. $ marketing $04/19/43$ 85000$2365$john
woodcock $director $ personnel $05/11/47$ 120000$5423$barry wood $chairman $admin $
08/30/56$ 160000$
• To delete all characters except the : and /, you can combine the -d and -c options

• tr has deleted all characters except the : and the / from its input. The appearance of the prompt
at the immediate end of output shows that the newline character has also not been spared.

150
End of Module 3