Module 5

OS & SYS MODULES

 OS Module

● provides functions for interacting with the Operating System.

● OS, comes under python’s standard utility modules

[Link]
>>>import os
>>>[Link]
'nt'
[Link]()
>>> [Link]()
'C:\\Users\\binuvp\\AppData\\Local\\Programs\\Python\\Python38-32
'
OS Module
Make
Directory:

Get current directory and change directory

 OS Module
Removing Directory: Cannot remove CWD and directory should be empty

List Directories: List of files in

CWD
OS Module
In order to set the current directory to the parent directory use “..” as the
argument in the chdir() function
OS Module

Walking through directory or subdirectory either top-down or bottom-up

sys Module
Provides functions and variable

Returns list of command line arguments,

sys Module

Return largest integer a variable can

take,
sys Module
Search path for all python modules,

Version number of current

interpreter,
Install NumPy
$sudo apt-get install python3-numpy
● NumPy is a library consisting of multidimensional array
objects and a collection of routines for processing those
arrays.
● Using NumPy, mathematical and logical operations on
arrays can be performed.
Using NumPy, a developer can perform the following
operations :
● Mathematical and logical operations on arrays.
● Fourier transforms and routines for shape manipulation.
● Operations related to linear algebra.
● NumPy has in-built functions for linear algebra and
random number generation.
 ndarray Object

The most important object defined in NumPy is an N-dimensional array

type called ndarray.
It describes the collection of items of the same type. Items in the
collection can be accessed using a zero-based index.
Every item in an ndarray takes the same size of block in the memory.
Each element in ndarray is an object of data-type object (called dtype).
The basic ndarray is created using an array function in NumPy as follows
− [Link]
 Creating Arrays
import numpy as np Output:
a = [Link]([1,2,3,4])
print(a) [1 2 3 4]

b = [Link]([(1,2,3),(4,5,6)], dtype = float)

[[1.2.3]
print(b) 4.5.6]]

c = [Link]([(1,2,3),(4,5,6),(7,8,9)]) [[1 2 3]
print(c) [4 5 6]
[7 8 9]]
 ndarray Object – Parameters
Some important attributes of ndarray object
[Link]
ndim represents the number of dimensions (axes) of the
ndarray.
[Link]
shape is a tuple of integers representing the size of the ndarray
in each dimension.
[Link]
size is the total number of elements in the ndarray. It is equal to
the product of elements of the shape
[Link]
dtype tells the data type of the elements of a NumPy
array. In NumPy array, all the elements have the same
data type.

[Link]
itemsize returns the size (in bytes) of each element of a
NumPy array.
Example:

import numpy as np
a = [Link]([[[1,2,3],[4,3,5]],[[3,6,7],[2,1,0]]])
print("The dimension of array a is:", [Link])
print("The size of the array a is: ", [Link])
print("The total no: of elements in array a is: ", [Link])
print("The datatype of elements in array a is: ", [Link])
print("The size of each element in array a is: ", [Link])
Output:
The dimension of array a is: 3
The size of the array a is: (2, 2, 3)
The total no: of elements in array a is: 12
The datatype of elements in array a is: int64
The size of each element in array a is: 8
 Indexing and slicing
One-dimensional arrays can be indexed, sliced and iterated over, much
like lists and other Python sequences.
import numpy as np
A=[Link](10)
print(A)
>>[0 1 2 3 4 5 6 7 8 9]
print(A[0])
>>0
print(A[-1])
>>9
print(A[0:3])
>>[0 1 2]
A[0:3]=100
A[3]=200
print(A)
>>[100 100 100 200 4 5 6 7 8 9]

slice=A[5:9]
print(slice)
>>[5 6 7 8]

slice[:]=200
B=[Link](10)
print(B[0:8:2])
>>[0 2 4 6]

print(B[8:0:-2])
>>[8 6 4 2]
print(B[:4])

>>[0 1 2 3]
print(B[5:])
>>[5 6 7 8 9]
print(B[::-1])
>>[9 8 7 6 5 4 3 2 1 0]
Arithmetic Operations with NumPy Array
Basic operations : with scalars
import numpy as np
a = [Link]([1,2,3,4,5])
b = a+1
print(b)
c = 2**a
print(c) Output:
[2 3 4 5 6]
[ 2 4 8 16 32]
Matrix operations – numpy functions
Matrices a and b are two arrays.
Addition: a+b or [Link](a,b)
Subtraction: a-b or [Link](a,b)

Multiplication: C = [Link](b) or [Link](a, b)

Transpose(): [Link]()
Inverse() = [Link]()
Determinant = [Link]()
Sample questions
1. Write a Python program to add two matrices and also find the
transpose of the resultant matrix.
2. Write a Python program to multiply two matrices.
Matrix
addition
Matrix Multiplication
from numpy import array
# define first matrix
A = array([[1, 2],[3, 4],[5, 6]])
print(A)

# define second matrix

B = array([[1, 2],[3, 4]])
print(B)

# multiply matrices C = [Link](B)

print(C)
Numpy - Random Numbers
NumPy offers the random module to work with random numbers.

Generate a random integer from 0 to 100:

 Numpy - Random Numbers
The random module's rand() method returns a random float between 0 and 1.
Numpy - Random
Numbers
The randint() method takes a size parameter where you can specify the shape of
an array.
Numpy - Random
Numbers
Generate a 2-D array with 3 rows, each row containing 5 random
integers from 0 to 100:
Numpy - Random
Numbers
•The rand() method also allows you to specify the shape of the array.
•Generate a 1-D array containing 5 random floats:
Numpy - Random Numbers
Generate a 2-D array with 3 rows, each row containing 5 random
numbers:
Numpy - Random
Numbers
• The choice() method allows you to generate a random value
based on an array of values.
• The choice() method takes an array as a parameter and
randomly returns one of the values.
Numpy - Random Numbers
The choice() method also allows you to return an array of values.
Add a size parameter to specify the shape of the array.
matplotlib

Installation:
$sudo apt-get install python3-matplotlib
Pyplot:
Most of the Matplotlib utilities lies under the pyplot submodule, and
are usually imported under the plt alias:
import [Link] as plt
Matplotlib Plotting
The plot() function is used to draw points (markers) in a diagram.
By default, the plot() function draws a line from point to point.

The function takes parameters for specifying points in the diagram.

◦Parameter 1 is an array containing the points on the x-axis.
◦Parameter 2 is an array containing the points on the y-axis.
Example – basic plot
Specifying line and symbol types and colors
Examples
plot(x, y, 'ro') # red circles
plot(x, y, 'ks-') # black squares connected by black lines
plot(x, y, 'g^') # green triangles pointing up
plot(x, y, 'k-') # black line
plot(x, y, 'C1s') # orange(ish) squares
 Matplotlib Markers

You can use the keyword argument

marker to emphasize each point with a
specified marker:
 Matplotlib Line

You can use the keyword argument

linestyle, or shorter ls, to change the
style of the plotted line:
 Matplotlib Labels and Title
With Pyplot, you can use the xlabel() and ylabel() functions to set a label
for the x- and y-axis.
 Matplotlib Adding Grid Lines
With Pyplot, you can use the grid() function to add grid lines to the plot.
 Matplotlib Subplot

With the subplot() function

you can draw multiple plots in
one figure:
Parameters are number of
rows, number
of columns, index of
the current plot.
 Matplotlib Scatter

The scatter() function plots one dot for each observation. It needs
two arrays of the same length, one for the values of the x-axis, and
one for values on the y-axis:
Matplotlib
Bars
With Pyplot, you can use the bar() function to draw bar graphs:
Matplotlib Histograms

A histogram is a graph showing frequency distributions.

It is a graph showing the number of observations within each given
interval.
Example: Say you ask for the height of 250 people, you might end up
with a histogram like this:
In Matplotlib, we use the hist() function to create histograms.
Matplotlib Histograms
Matplotlib Pie
Charts
With Pyplot, you can use the pie() function to draw pie charts:
Labels
Add labels to the pie chart with the label
parameter. The label parameter must be an
array with one label for each wedge:
Start Angle
As mentioned the default start angle is at the x-axis, but you can change the start
angle by specifying a startangle parameter.
The startangle parameter is defined with an angle in degrees, default angle is 0:
For more: Reference
[Link]
 Ticks and Tick Labels

•Ticks are the markers denoting data points on axes.

• The xticks() and yticks() function takes a list object as argument. The
elements in the list denote the positions on corresponding action where
ticks will be displayed.
•labels corresponding to tick marks can be set
by set_xlabels() and set_ylabels() functions respectively.
• [Link](*args, emit=True, **kwargs): For setting the axes for our
plot with parameter rect as [left,bottom,width,height]for setting axes
position. none: It gives a new full window axes.
•ax.set_title('sine') : To give title as Sine
Example
Legends
A legend is an area describing the elements of the graph. In the
matplotlib library, there’s a function called legend() which is used to
Place a legend on the axes.
The attribute Loc in legend() is used to specify the location of the
legend.
Default value of loc is loc=”best” (upper left). The strings ‘upper left’,
‘upper right’, ‘lower left’, ‘lower right’ place the legend at the
corresponding corner of the axes/figure.
 Legend
The Following are some more attributes of function legend() :
shadow: [None or bool] Whether to draw a shadow behind the [Link]’s Default
value is None.
markerscale: [None or int or float] The relative size of legend markers compared with
the originally drawn [Link] Default is None.
numpoints: [None or int] The number of marker points in the legend when creating a
legend entry for a Line2D (line).The Default is None.
fontsize: The font size of the [Link] the value is numeric the size will be the
absolute font size in points.
facecolor: [None or “inherit” or color] The legend’s background color.
edgecolor: [None or “inherit” or color] The legend’s background patch edge color.
Example
Example
CSV file format
• CSV (Comma Separated Values) is a simple file format used to
store tabular data, such as a spreadsheet or database.
•A CSV file stores tabular data (numbers and text) in plain text.
•Each line of the file is a data record.
•Each record consists of one or more fields, separated by commas.
• The use of the comma as a field separator is the source of the
name for this file format.
 CSV file characteristics
● One line for each record
● Comma separated fields
● Space characters adjacent to comma are ignored
● Fields with in built commas are separated by double quote characters
● Field with double quote characters must be surrounded by double
quotes.
Working with CSV files- pandas
Install pandas:
$sudo apt install python3-pandas
A simple way to store big data sets is to use CSV files (comma
separated files).
 pandas - DataFrame
Pandas DataFrame: It is two-dimensional size-mutable, potentially
heterogeneous tabular data structure with labeled axes (rows and
columns). Pandas DataFrame consists of three principal components,
the data, rows, and columns.
 Creating a dataframe using List:
DataFrame can be created using a single list or a list of lists.
Creating DataFrame from dict of
ndarray/lists:
Column Selection:
Row Selection:
Pandas provide a unique method to retrieve rows
from a Data frame. [Link][] method is used
to retrieve rows from Pandas DataFrame.
Rows can also be selected by passing integer location
to an iloc[] function.
DataFrame functions
abs() : Return a Series/DataFrame with absolute numeric value of
each element.
DataFrame functions
agg([func, axis]) : Aggregate using one or more operations over the
specified axis.
DataFrame functions
append(other[, ignore_index, ...]): Append rows of other to the end
of caller, returning a new object.
DataFrame functions

count([axis, level, numeric_only]) : Count non-NA cells for each

column or row.
DataFrame functions

count([axis, level, numeric_only]) : Count non-NA cells for each

column or row.
DataFrame functions

[Link]([by, axis, level, ...]): Group DataFrame using a

mapper or by a Series of columns.
DataFrame functions
DataFrame functions
[Link]([axis, skipna, level, ...]): Return the maximum of the
values over the requested axis.
DataFrame functions
[Link]([axis, skipna, level, ...]): Return the minimum of the
values over the requested axis.
DataFrame functions

[Link]([axis, skipna, level, ...]): Return the mean of the

values over the requested axis.
For more functions:
Reference: [Link]
Working with CSV files
Load CSV files to Python
Pandas

read_csv() : parameters
1. filepath_or_buffer: It is the location of the file which is to be retrieved using
this function. It accepts any string path or URL of the file.
2. sep: It stands for separator, default is ‘, ‘ as in csv(comma separated values).
read_csv() : parameters(continue)

3. header: It accepts int, list of int, row numbers to use as the column names
and start of the data. If no names are passed, i.e., header=None, then, it
will display first column as 0, second as 1, and so on.
4. usecols: It is used to retrieve only selected columns from the csv file.
5. nrows: It means number of rows to be displayed from the dataset.
6. index_col: If None, there are no index numbers displayed along with records.
7. squeeze: If true and only one column is passed, returns pandas series.
8. skiprows: Skips passed rows in new data frame.
9. names: It allows to retrieve columns with new names.
read_csv() : parameters(continue)
read_csv() : parameters(continue)

Read the csv file sep='|' :

Read the csv file with header parameter:

The row 0 seems to be a better fit for the header.
Note: Row numbering starts from 0 including column header
Renaming column headers
 Writing to CSV file
Pandas DataFrame provides to_csv() method to write/export DataFrame
to CSV comma-separated delimiter file along with header and index.
df.to_csv('[Link]')
Write DataFrame to CSV without Header
df.to_csv('[Link]', header=False)
Writing Using Custom Delimiter
df.to_csv('[Link]', header=False, sep='|')
Writing to CSV ignoring Index
df.to_csv('[Link]', index=False)
 Writing to CSV file
Writing to CSV ignoring Index
df.to_csv('[Link]', index=False)
Export Selected Columns to CSV File
column_names = ['Courses', 'Fee','Discount']
df.to_csv('[Link]',index=False, columns=column_names)

Change Header Column Names While Writing

column_names = ['Courses', 'Course_Fee','Course_Duration','Course_Discount']
df.to_csv('[Link]',index=False, header=column_names)
 Writing to CSV file

Write DataFrame to CSV by Encoding

df.to_csv(file_name, sep='\t', encoding='utf-8')
Append DataFrame to existing CSV File
df.to_csv("c:/tmp/[Link]", header=False, sep='|', index=False, mode='a')
Writing to CSV file
 Clean and Update the CSV file
CSV Data Cleaning Checks
Missing Values
Outliers
Duplicate Values

1. Cleaning Missing Values in CSV File:

In Pandas, a missing value is usually denoted by NaN , since it is based on the
NumPy package it is the special floating-point NaN value particular to NumPy.
 Clean and Update the CSV file

1. Dropping Missing Values:

[Link]() – Drop all rows that have any NaN values
[Link](how=’all’) – Drop only if ALL columns are NaN
[Link](thresh=2) – Drop row if it does not have at least two values that are
not NaN
[Link](subset=[1]) – Drop only if NaN in specific column
Dropping Missing Values:
Clean and Update the CSV file

2. Replacing Missing values:

Pandas module has the .fillna() method, which accepts a value that
we want to replace in place of NaN values. We just calculated the
mean of the column and passed it as an input argument to fillna()
method.
.sum() uses to find sum of all values along the index axis. We are
going to skip the NaN values in the calculation of the sum.
Sample questions
Q. Given a file “[Link]” of automobile data with the fields index,
company, body-style, wheel-base, length, engine-type, num-of-
cylinders, horsepower, average-mileage, and price, write Python
codes using Pandas to
1) Clean and Update the CSV file
2) Print total cars of all companies
3) Find the average mileage of all companies
4) Find the highest priced car of all companies.
Data set
1) Clean and Update the CSV file
Output
2) Print total cars of all companies
Print total cars of all companies
3) Find the average mileage of all
companies
4) Find the highest priced car of all
companies.
Sample questions
Q. Write Python program to write the data given below to a CSV file.

Q. What are the important characteristics of CSV file format.

Sample questions
Q. Given the sales information of a company as CSV file with the following
fields month_number, facecream, facewash, toothpaste,
bathingsoap, shampoo, moisturizer, total_units, total_profit. Write Python
codes to visualize the data as follows
1) Toothpaste sales data of each month and show it using a scatter plot
2) Face cream and face wash product sales data and show it using the bar
chart
3) Calculate total sale data for last year for each product and show it using a
Pie chart.
Data set
Toothpaste sales data of each month and
show it using a scatter plot
Output
Face cream and face wash product sales
data and show it using the bar chart
Output
Calculate total sale data for last year for
each product and show it using a Pie
chart.
Output
 Flask
• Flask is a micro web framework written in Python. It is classified as a
microframework because it does not require particular tools or libraries.
• It has no database abstraction layer, form validation, or any other
components where pre-existing third-party libraries provide
common functions.
• Flask is a web application framework written in Python. Armin
Ronacher, who leads an international group of Python enthusiasts named
Pocco, develops it.
• Flask is based on Werkzeug WSGI toolkit and Jinja2 template
engine. Both are Pocco projects.
Install virtualenv for development
environment
$sudo apt-get install virtualenv
To activate corresponding environment, on Linux/OS X, use
the following −

venv/bin/activate
[Link]
 Flask – Application

Importing flask module in the project is mandatory. An object of Flask class is

our WSGI application.
Flask constructor takes the name of current module ( name ) as argument.
The route() function of the Flask class is a decorator, which tells
application
the which URL should call the associated function.
[Link](rule, options)

The rule parameter represents URL binding with the function.

The options is a list of parameters to be forwarded to the
underlying Rule object.
 Flask – Application
In the above example, ‘/’ URL is bound with hello_world() function. Hence, when the home
page of web server is opened in browser, the output of this function will be rendered.
Finally the run() method of Flask class runs the application on the local development server.
[Link](host, port, debug, options)
All parameters are optional,
host: Hostname to listen on. Defaults to [Link] (localhost). Set to ‘[Link]’ to have server
available externally and port: Defaults to 5000 (TCP/UDP)
debug: Defaults to false. If set to true, provides a debug information
options: To be forwarded to underlying Werkzeug server.
Micro services using Flask
The framework very easy to work with for developing RESTful services
working with JSON.
The simplicity of both the framework and the language itself allows you to
write small, concise request handler functions.
Micro services using Flask

You can also easily add extra request processing logic around your
endpoints. For example, if you have one doing some expensive operation,
you could memoize the results for some time instead of repeating it on
every call.
Micro services using Flask
Caching View Functions
To cache view functions you will use the cached() decorator. This
decorator will use [Link] by default for the cache_key.: