Unit-III

System tools
OS and Sys modules
Directory Traversal tools
Lab7: process standard streams.
Parallel System tools
Threading and queue
Program Exits
Lab 8 :Command-line arguments, shell variables
System interfaces by focusing on tools and techniques
Binary files, tree walkers
Python’s library support for running programs in parallel.
Lab 9: Python scripts here perform real tasks.
System tools

The [Link] to Knowledge

System tools/programs sometimes called command-line utilities, shell scripts, system

administration, systems programming, and other permutations of such words.

System Scripting Overview

./[Link]
Python 3.1.1 (r311:74480, Feb 20 2010, 10:16:52)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os
>>> len(dir(sys))
64
>>> len(dir(os))
217
>>> len(dir([Link]))
51

Python System Modules

Most system-level interfaces in Python are shipped in just two modules: sys and os.
That’s somewhat oversimplified; other standard modules belong to this domain too.
Among them are the following:

glob
For filename expansion

socket
For network connections and Inter-Process Communication (IPC)

threading, _thread, queue

For running and synchronizing concurrent threads

time, timeit
For accessing system time details

subprocess, multiprocessing
For launching and controlling parallel processes

signal, select, shutil, tempfile, and others

For various other system-related tasks

OS Module in python
The name function gives the name of the operating system dependent module imported. The
following names have currently been registered: ‘posix’, ‘nt’, ‘os2’, ‘ce’, ‘java’ and ‘riscos’.
import os
print([Link])
 Handling the Current Working Directory

Current Working Directory(CWD) as a folder, where Python is operating. Whenever the

files are called only by their name, Python assumes that it starts in the CWD which means
that name-only reference will be successful only if the file is in the Python’s CWD.

import os
cwd = [Link]()
print("Current working directory:", cwd)

To change the current working directory (CWD) [Link]() method is used. This method
changes the CWD to a specified path. It only takes a single argument as a new directory
path.

import os
def current_path():
print("Current working directory before")
print([Link]())
print()
current_path()
[Link]('../')
current_path()

 Creating a Directory

[Link]() method in Python is used to create a directory named path with the specified
numeric mode. This method raises FileExistsError if the directory to be created already
exists.
 import os
directory = "GeeksforGeeks"
parent_dir = "D:/Pycharm projects/"
path = [Link](parent_dir, directory)

[Link](path)
print("Directory '% s' created" % directory)
directory = "Geeks"
parent_dir = "D:/Pycharm projects"
mode = 0o666
path = [Link](parent_dir, directory)
[Link](path, mode)
print("Directory '% s' created" % directory)

output:
Directory 'GeeksforGeeks' created
Directory 'Geeks' created

 Listing out Files and Directories with Python

[Link]() method in Python is used to get the list of all files and directories in the
specified directory. If we don’t specify any directory, then the list of files and directories in
the current working directory will be returned.

import os
path = "/"
dir_list = [Link](path)
print("Files and directories in '", path, "' :")
print(dir_list)

 Deleting Directory or Files using Python

OS module proves different methods for removing directories and files in Python.
These are –
  Using [Link]()
 Using [Link]()
import os
file = '[Link]'
location = "D:/Pycharm projects/GeeksforGeeks/Authors/Nikhil/"
path = [Link](location, file)
[Link](path)

import os
directory = "Geeks"
parent = "D:/Pycharm projects/"
path = [Link](parent, directory)
[Link](path)

Sys Module in Python

[Link] is used which returns a string containing the version of Python Interpreter with
some additional information. This shows how the sys module interacts with the interpreter. Let
us dive into the article to get more information about the sys module.
import sys
print([Link])

output:
3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]

Read from stdin in Python

import sys
for line in [Link]:
if 'q' == [Link]():
break
 print(f'Input : {line}')

print("Exit")

Python [Link] Method

import sys
[Link]('Geeks')

stderr function in Python

import sys
def print_to_stderr(*a):
print(*a, file = [Link])

print_to_stderr("Hello World")

Exiting the Program

[Link]([arg]) can be used to exit the program. The optional argument arg can be an integer
giving the exit or another type of object. If it is an integer, zero is considered “successful
termination”.

import sys
age = 17
if age < 18:
[Link]("Age less than 18")
else:
print("Age is not less than 18")

Directory Traversal tools

[Link]() method of the OS module can be used for listing out all the directories. This method
basically generates the file names in the directory tree either top-down or bottom-up. For each
directory in the tree rooted at directory top (including top itself), it yields a 3-tuple (dirpath,
dirnames, filenames).
 dirpath: A string that is the path to the directory
 dirnames: All the sub-directories from root.
 filenames: All the files from root and directories.

Syntax:

[Link](top, topdown=True, onerror=None, followlinks=False)

Parameters:

top: Starting directory for [Link]().

topdown: If this optional argument is True then the directories are scanned from top-down
otherwise from bottom-up. This is True by default.

onerror: It is a function that handles errors that may occur.

followlinks: This visits directories pointed to by symlinks, if set to True.

Return Type: For each directory in the tree rooted at directory top (including top itself), it
yields a 3-tuple (dirpath, dirnames, filenames).

# Python program to list out all the sub-directories and files

import os

# List to store all directories

L=[]

# Traversing through Test

for root, dirs, files in [Link]('Test'):

# Adding the empty directory to list

[Link]((root, dirs, files))

print("List of all sub-directories and files:")

for i in L:
print(i)
output:
List of all sub-directories and files:
('Test', ['B', 'C', 'D', 'A'], [])
('Test/B', [], [])
('Test/C', [], ['[Link]'])
('Test/D', ['E'], [])
('Test/D/E', [], [])
('Test/A', ['A2', 'A1'], [])
('Test/A/A2', [], [])
('Test/A/A1', [], ['[Link]'])

Parallel System tools

Early on in computing, programmers realized that they could tap into such unused processing
power by running more than one program at the same time. By dividing the CPU’s attention
among a set of tasks, its capacity need not go to waste while any given task is waiting for an
external event to occur. The technique is usually called parallel processing (and sometimes
“multiprocessing” or even “multitasking”) because many tasks seem to be performed at once,
overlapping and parallel in time

There are two fundamental ways to get tasks running at the same time in Python

 Process forks
 Spawned threads.

Functionally, both rely on underlying operating system services to run bits of Python code in
parallel. Procedurally, they are very different in terms of interface, portability, and
communication. For instance, at this writing direct process forks are not supported on Windows
under standard Python (though they are under Cygwin Python on Windows). By contrast,
Python’s thread support works on all major platforms. Moreover, the [Link] family of calls
provides additional ways to launch programs in a platform neutral way that is similar to forks

Forking Processes

Forked processes are a traditional way to structure parallel tasks, and they are a fun damental
part of the Unix tool set. Forking is a straightforward way to start an inde pendent program,
whether it is different from the calling program or not. Forking is based on the notion of copying
programs: when a program calls the fork routine, the operating system makes a new copy of that
program and its process in memory and starts running that copy in parallel with the original.

After a fork operation, the original copy of the program is called the parent process, and the copy
created by [Link] is called the child process.

Output
$python [Link]

Hello from parent 7296 7920

Hello from child 7920
Hello from parent 7296 3988
Hello from child 3988
Hello from parent 7296 6796
Hello from child 6796
q

The fork/exec Combination

The below example shows that forks new processes until we type q again, but child processes
run a brand-new program instead of calling a function in the same file.

Spawned child program

Just as when typed at a shell, the string of arguments passed to [Link] by the fork exec script
in the above example starts another Python program file, as shown in below example.

$[Link]

[Link] import os, sys

print('Hello from child', [Link](), [Link][1])

Threading and queue

Threads are another way to start activities running at the same time. In short, they run a call to a
function in parallel with the rest of the program. Threads are sometimes called “lightweight
processes,” because they run in parallel like forked processes, but all of them run within the same
single process. While processes are commonly used to start independent programs.

The _thread Module

Since the basic _thread module is a bit simpler than the more advanced threading module
covered later in this section, let’s look at some of its interfaces first. This module provides a
portable interface to whatever threading system is available in your platform.

#spawn threads until you type 'q'

import _thread

def child(tid):
print('Hello from thread', tid)
def parent():
i=0
while True:
i += 1
_thread.start_new_thread(child, (i,))
if input() == 'q': break
parent()

the _thread.start_new_thread call itself returns immediately with no useful value, and the thread
it spawns silently exits when the function being run returns (the return value of the threaded
function call is simply ignored).

Output:
Hello from thread 1
Hello from thread 2
Hello from thread 3
Hello from thread 4
q

Threading module

Step 1: Import Module

First, import the threading module.
import threading

Step 2: Create a Thread

To create a new thread, we create an object of the Thread class. It takes the ‘target’ and ‘args’
as the parameters. The target is the function to be executed by the thread whereas the args
is the arguments to be passed to the target function.
t1 = [Link](target, args)
t2 = [Link](target, args)

Step 3: Start a Thread

To start a thread, we use the start() method of the Thread class.
[Link]()
[Link]()

Step 4: End the thread Execution

Once the threads start, the current program (you can think of it like a main thread) also keeps
on executing. In order to stop the execution of the current program until a thread is complete,
we use the join() method.
[Link]()
[Link]()

As a result, the current program will first wait for the completion of t1 and then t2. Once, they
are finished, the remaining statements of the current program are executed.

Example:

import threading

def print_cube(num):
print("Cube: {}" .format(num * num * num))

def print_square(num):
print("Square: {}" .format(num * num))

# __name__ == '__main__' statement checks whether the current script is being run as the
if __name__ =="__main__": main program, and if it is, it calls the main() function to execute the code.

t1 = [Link](target=print_square, args=(10,))
t2 = [Link](target=print_cube, args=(10,))

[Link]()
[Link]()

[Link]()
[Link]()

print("Completed!")

Output:
Square: 100
Cube: 1000
Completed!

The queue Module

You can synchronize your threads’ access to shared resources with locks, but you often don’t
have to. As mentioned, realistically scaled threaded programs are often structured as a set of
producer and consumer threads, which communicate by placing data on, and taking it off of, a
shared queue. As long as the queue synchronizes access to itself, this automatically synchronizes
the threads’ interactions.

The Python queue module implements this storage device. It provides a standard queue data
structure—a first-in first-out (fifo) list of Python objects, in which items are added on one end
and removed from the other. Like normal lists, the queues provided by this module may contain
any type of Python object.

There are various functions available in this module:

 maxsize – Number of items allowed in the queue.

 empty() – Return True if the queue is empty, False otherwise.
 full() – Return True if there are maxsize items in the queue. If the queue was initialized
with maxsize=0 (the default), then full() never returns True.
 get() – Remove and return an item from the queue. If queue is empty, wait until an item is
available.
 get_nowait() – Return an item if one is immediately available, else raise QueueEmpty.
 put(item) – Put an item into the queue. If the queue is full, wait until a free slot is available
before adding the item.
 put_nowait(item) – Put an item into the queue without blocking. If no free slot is
immediately available, raise QueueFull.
 qsize() – Return the number of items in the queue.

Example:
from queue import Queue
q = Queue(maxsize = 3)
print([Link]())
[Link]('a')
[Link]('b')
[Link]('c')
print("\nFull: ", [Link]())
print("\nElements dequeued from the queue")
print([Link]())
print([Link]())
print([Link]())
 print("\nEmpty: ", [Link]())
[Link](1)
print("\nEmpty: ", [Link]())
print("Full: ", [Link]())

Output:
0
Full: True
Elements dequeued from the queue
a
b
c
Empty: True
Empty: False
Full: False

Another Example

thread.start_new_thread(producer, (i,))
[Link](((numproducers-1) * nummessages) + 1)
print('Main thread exit.')
Program Exits
As we’ve seen, unlike C, there is no “main” function in Python. When we run a program, we
simply execute all of the code in the top-level file, from top to bottom (i.e., in the filename we
listed in the command line, clicked in a file explorer, and so on). Scripts normally exit when
Python falls off the end of the file, but we may also call for program exit explicitly with tools in
the sys and os modules.

sys Module Exits

For example, the built-in [Link] function ends a program when called, and earlier than normal:
>>> [Link](N) # exit with status N, else exits on end of script

>>> import sys

>>> try:
... [Link]() # see also: os._exit, Tk().quit()
... except SystemExit:
...

os Module Exits

It’s possible to exit Python in other ways, too. For instance, within a forked child process on
Unix, we typically call the os._exit function rather than [Link]; threads may exit with a
_thread.exit call; and tkinter GUI applications often end by calling something named Tk().quit().
We’ll meet the tkinter module later in this book; let’s take a look at os exits here.

Command-line arguments
The arguments that are given after the name of the program in the command line shell of the
operating system are known as Command Line Arguments. Python provides various ways of
dealing with these types of arguments. The three most common are:
 Using [Link]
 Using getopt module
 Using argparse module

Using [Link]

The sys module provides functions and variables used to manipulate different parts of the
Python runtime environment. This module provides access to some variables used or
maintained by the interpreter and to functions that interact strongly with the interpreter.
One such variable is [Link] which is a simple list structure. It’s main purpose are:
 It is a list of command line arguments.
 len([Link]) provides the number of command line arguments.
 [Link][0] is the name of the current Python script.

# Python program to demonstrate

# command line arguments
import sys

# total arguments
n = len([Link])
print("Total arguments passed:", n)

# Arguments passed
print("\nName of Python script:", [Link][0])

print("\nArguments passed:", end = " ")

for i in range(1, n):
print([Link][i], end = " ")

# Addition of numbers
Sum = 0
# Using argparse module
for i in range(1, n):
Sum+=int(input([Link][i]))

print("\n\nResult:", Sum)

System interfaces by focusing on tools

File Tools

External files are at the heart of much of what we do with system utilities. For instance, a testing
system may read its inputs from one file, store program results in another file, and check
expected results by loading yet another file.

In Python, the built-in open function is the primary tool scripts use to access the files on the
underlying computer system

When called, the open

function returns a new file object that is connected to the external file; the file object has methods
that transfer data to and from the file and perform a variety of file-related operations.
The open function also provides a portable interface to the underlying file system it works the
same way on every platform on which Python runs.

The File Object Model in Python

• Text files contain Unicode text. In your script, text file content is always a str string—a
sequence of characters (technically, Unicode “code points”). Text files perform the automatic
line-end translations described in this chapter by default and automatically apply Unicode
encodings to file content: they encode to and decode from raw binary bytes on transfers to and
from the file, according to a provided or default encoding name. Encoding is trivial for ASCII
text, but may be sophisticated in other cases.

• Binary files contain raw 8-bit bytes. In your script, binary file content is always a byte string,
usually a bytes object—a sequence of small integers, which supports most str operations and
displays as ASCII characters whenever possible. Binary files perform no translations of data
when it is transferred to and from files: no line end translations or Unicode encodings are
performed.

Binary files
Python scripts can also open and process files containing binary data—JPEG images, audio
clips, packed binary data produced by FORTRAN and C programs, encoded text, and anything
else that can be stored in files as bytes. The primary difference in terms of your code is the mode
argument
passed to the built-in open function:

>>> file = open('[Link]', 'wb') # open binary output file

>>> file = open('[Link]', 'rb') # open binary input file
Once you’ve opened binary files in this way, you may read and write their contents using the
same methods just illustrated: read, write, and so on. The readline and readlines methods as well
as the file’s line iterator still work here for text files opened in binary mode, but they don’t make
sense for truly binary data that isn’t line oriented (end-of-line bytes are meaningless, if they
appear at all).

In all cases, data transferred between files and your programs is represented as Python
strings within scripts, even if it is binary data. For binary mode files, though, file content
is represented as byte strings. Continuing with our text file from preceding examples:

>>> open('[Link]').read() # text mode: str

'Hello file world!\nBye file world.\nThe Life of Brian'

>>> open('[Link]', 'rb').read() # binary mode: bytes

b'Hello file world!\r\nBye file world.\r\nThe Life of Brian'

>>> file = open('[Link]', 'rb')

>>> for line in file: print(line)

...
b'Hello file world!\r\n'
b'Bye file world.\r\n'
b'The Life of Brian'

Tree walkers
what if you want to apply an operation to every file in every directory and subdirectory in an
entire directory tree?

The [Link] visitor

[Link] is a generator function—at each directory in the tree, it yields a three-item tuple,
containing the name of the current directory as well as lists of both all the files and all the
subdirectories in the current directory. Because it’s a generator, its walk is usually run by a for
loop (or other iteration tool); on each iteration, the walker advances to the next subdirectory, and
the loop runs its code for the next level of the tree (for instance, opening and searching all the
files at that level).

>>> import os
>>> for (dirname, subshere, fileshere) in [Link]('.'):
... print('[' + dirname + ']')
... for fname in fileshere:
... print([Link](dirname, fname)) # handle one file
...
[.]
.\[Link]
.\[Link]
.\[Link]
.\[Link]
[.\parts]
.\parts\part0001
.\parts\part0002
.\parts\part0003
.\parts\part0004