Avanthi Degree & P.

G College
Narayanaguda –Hyderabad

[Link]. (Data Science) II Year III Semester

Data Engineering with Python

( Paper – III )

Name : ________________________________

Roll No : ________________________________

Group : ________________________________

The expert in anything was once a beginner

 Paper – III : Data Engineering with Python
Unit – I
Data Science:
Data Analysis Sequence, Data Acquisition Pipeline, Report Structure
[Reference 1 (Chapter 1-Unit 1 to Unit 3)]
Files and Working with Text Data:
Types of Files, Creating and Reading Text Data, File Methods to Read and Write Data, Reading
and Writing Binary Files, The Pickle Module, Reading and Writing CSV Files, Python os and
[Link] Modules. [Reference 2, Chapter 9)]
Working with Text Data:
JSON and XML in Python. [Reference 2, Section12.2]
Unit – II
Working with Text Data:
Processing HTML Files, Processing Texts in Natural Languages
[Reference 1(Chapter3 –Unit 13, and Unit16)
Regular Expression Operations:
Using Special Characters, Regular Expression Methods, Named Groups in Python Regular
Expressions, Regular Expression with glob Module [Reference 2-Chapter 10]
Unit – III
Working with Databases:
Setting Up a MySQL Database, Using a MySQL Database: Command Line, Using a MySQL
Database, Taming Document Stores: MongoDB
[Reference 1 (Chapter 4-Unit 17 to Unit 20)]
Working with Tabular Numeric Data ( Numpy with Python):
NumPy Arrays Creation Using array() Function, Array Attributes, NumPy Arrays Creation with
Initial Placeholder Content, Integer Indexing, Array Indexing, Boolean ArrayIndexing, Slicing and
Iterating in Arrays, Basic Arithmetic Operations on NumPy Arrays, Mathematical Functions in
NumPy, Changing the Shape of an Array, Stacking and Splitting of Arrays, Broadcasting in Arrays.
[Reference 2: Section 12.3)]
Unit – IV
Working with Data Series and Frames:
Pandas Data Structures, Reshaping Data, Handling Missing Data, Combining Data, Ordering and
Describing Data, Transforming Data, Taming Pandas File I/O
[Reference 1 (Chapter 6-Unit 31 to Unit 37)]
Plotting:
Basic Plotting with PyPlot, Getting to Know Other Plot Types, Mastering Embellishments, Plotting
with Pandas [Reference 1 (Chapter8-Unit 41 to Unit 44)]
1
References :
1. Data Science Essentials in Python: Collect, Organize, Explore, Predict, Value. Dmitry
Zinoriev, The Pragmatic Programmers LLC, 2016
2. Introduction to Python Programming. Gowrishankar S., Veena A. CRC Press, Taylor &
Francis Group, 2019

Practical : Data Engineering with Python (Lab)

Exercises :

1. Write programs to parse text files, CSV, HTML, XML and JSON documents and extract
relevant data. After retrieving data check any anomalies in the data, missing values etc.
2. Write programs for reading and writing binary files
3. Write programs for searching, splitting, and replacing strings based on pattern matching
using regular expressions
4. Design a relational database for a small application and populate the database. Using SQL
do the CRUD (create, read, update and delete) operations.
5. Create a Python MongoDB client using the Python module pymongo. Using a collection
object practice functions for inserting, searching, removing, updating, replacing, and
aggregating documents, as well as for creating indexes.
6. Write programs to create numpy arrays of different shapes and from different sources,
reshape and slice arrays, add array indexes, and apply arithmetic, logic, and aggregation
functions to some or all array elements.
7. Write programs to use the pandas data structures: Frames and series as storage containers
and for a variety of data-wrangling operations, such as:
• Single-level and hierarchical indexing
• Handling missing data
• Arithmetic and Boolean operations on entire columns and tables
• Database-type operations (such as merging and aggregation)
• Plotting individual columns and whole tables
• Reading data from files and writing data to files

2
 Chapter 1
Data Science

Data Science is a combination of mathematics, statistics, machine learning, and computer

science. Data Science is collecting, analyzing and interpreting data to gather insights into the
data that can help decision-makers make informed decisions.
Data Science is used in almost every industry today that can predict customer behavior and
trends and identify new opportunities. Businesses can use it to make informed decisions about
product development and marketing.
In simple terms, Data Science helps to analyze data and extract meaningful insights from it by
combining statistics & mathematics, programming skills, and subject expertise.

1.1 Data Analysis Sequence

What is Data Analysis?

• Data analysis is defined as a process of cleaning, transforming, and modeling data to
discover useful information for business decision-making.
• The purpose of Data Analysis is to extract useful information from data and taking the
decision based upon the data analysis.
A simple example of Data analysis is whenever we take any decision in our day-to-day ,
analyzing our past or future and making decisions based on it. For that, we gather memories
of our past or dreams of our future

Why Data Analysis?

To grow your business even to grow in your life, sometimes all you need to do is Analysis.

Data Analysis Sequence :

The process of data analysis is a systematic approach that involves several stages, each crucial
to ensuring the accuracy and usefulness of the results.

• The first step in the data analysis process is to define the objectives , understanding the
problem , identifying the data needed to address it, and defining the metrics to measure
the outcomes.
• The next step is to collect the relevant data. This can be done through various methods
such as surveys, interviews, observations, or extracting from existing databases. The data
collected can be quantitative (numerical) or qualitative (non-numerical)
3
 • Data cleaning, It involves checking the data for errors and inconsistencies, and correcting
or removing them.
• Once the data is cleaned, it's time for the actual analysis. This involves applying statistical
or mathematical techniques to the data to discover patterns, relationships, or trends.
There are various tools and software available for this purpose, such as Python, R, Excel,
etc.
• The next step is to interpret the results and visualize them in a way that is easy to
understand.
• The final step in the data analysis process is data storytelling. It is crucial for
communicating the results to non-technical audiences and for making data-driven
decisions.
Types of Data Analysis :

Data analysis can be categorized into four main types, each serving a unique purpose and
providing different insights. These are descriptive, diagnostic, predictive, and prescriptive
analyses.

4
1.2 Data Acquisition Pipeline

What is Data Acquisition?

Data Acquisition is process of obtaining data from sources & Converting into representations
suitable for further processing.
What is Data Pipeline?
A Data pipeline is set of actions that takes / ingest raw data from different sources moves
data to destination for storage and processing.
Or
A Data Pipeline is a network system that allows data to be moved from source to target
location.

The 3 main sources of data are Databases, Internet and Local files. These data obtained may be
structured or unstructured.
The most popular data formats are :
➢ Unstructured plain text in a natural language (Such as English or Chinese)
➢ Structured data includes Tabular data in CSV (Comma separated values) files, Tabular
data from databases, tagged data in HTML (Hyper Text Markup Language), XML
(eXtensible Markup Language), Tagged data in JavaScript Object Notation (JSON).
➢ Original structure of the extracted data is represented using native python data
structures (lists and dictionaries) or advanced data structures that support specialized
(arrays and data frames).
➢ Data pipeline automation converts data from various sources into a specific format to
be stored or for other analysis tools.

5
1.3 Report Structure
The project report data scientists submit to the data sponsor (the customer) . The
report typically includes the following :
➢ Abstract (a brief description of the project)
➢ Introduction
➢ Methods that were used for data acquisition and processing
➢ Results that were obtained while processing
➢ Conclusion
➢ Appendix
In addition to the non-essential results and graphics, the appendix contains all
reproducible code used to process the data.

6
 Chapter 2
Files and Working with Text Data
2.1 Introduction to Files
• A file is a named location on a secondary storage media where data are permanently
stored for later access.
• All the data on hard drive consists of files and directories. Files store data, Directories
store files and other directories.
• The folders often referred to as directories, are used to organize files on computer.
• The Directories themselves take up virtually no space on the hard drive. Files, on the
other hand, can range from a few bytes to several gigabytes.

2.2 Types of Files

Computers store every file as a collection of 0s and 1s i.e., in binary form. Therefore, every
file is basically just a series of bytes stored one after the other
There are mainly two types of data files – Text files and Binary files. These two file types
may look the same on the surface but they encode data differently.
2.2.1 Text File
• A text file consists of human readable characters, Text files don’t have any specific
encoding and it can be opened in normal text editor itself.
• Each line of a text file is terminated by a special character, called the End of Line (EOL).
For example, the default EOL character in Python is the newline (\n).
• Contents in a text file are usually separated by whitespace, but comma (,) and tab (\t) are
also commonly used to separate values in a text file.

Example:
• Web standards: html, XML, CSS, JSON, etc.
• Source code: c, cpp, js, py, java, etc.
• Documents: txt, tex, rtf,asciidoc, etc.
• Tabular data: csv, tsv , etc.

2.2.2 Binary File

• Binary files are made up of non-human readable characters and symbols, which require
specific programs to access its contents.
• Binary files are also stored in terms of bytes (0s and 1s), but unlike text files, these bytes
do not represent the ASCII values of characters.

7
Example:
• Document files: .pdf, .doc, .xls ,.ppt, etc.
• Image files: .png, .jpg, .gif, .bmp etc.
• Video files: .mp4, .3gp, .mkv, .avi etc.
• Audio files: .mp3, .wav, .mka, .aac etc.
• Database files: .mdb, .accde, .frm, .sqlite etc.
• Archive files: .zip, .rar, .iso, .7z etc.
• Executable files: .exe, .dll, .class etc.

All binary files follow a specific format. We can open some binary files in the normal text
editor but we can’t read the content present inside the file. That’s because all the binary files
will be encoded in the binary format, which can be understood only by a computer or
machine.

For handling such binary files we need a specific type of software to open it.

For Example, You need Microsoft word software to open .doc binary files. Likewise, you need
a pdf reader software to open .pdf binary files and you need a photo editor software to read
the image files and so on.

2.2.3 File Paths

• Each and every operating system follows naming conventions for an individual file.
• In Windows operating system, the maximum length for a path is 260 characters whereas
in Linux Operating system it is 4096 characters.
The following fundamental rules enable applications to create and process valid names for
files and directories, regardless of the file system.
• Each and every file has extension, using period character (.) base file name and file
extension will separate.
• The file path contains one or more components, In Windows use backslash (\) and Linux
use forward slash (/) to separate the components of a path.
• File and Directory names in windows are not case sensitive while in Linux it is case
sensitive.
• Volume designators (drive letters) are similarly case-insensitive. For example, "D:\" and
"d:\" refer to the same volume.
• In windows Operating system reserved words like
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8,
COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9
should not be used to name files and directories.

8
2.2.3 Fully Qualified Path and Relative Path
• A file path can be fully qualified path name or relative path.
• The fully qualified path name is also called an Absolute path. Absolute path is the exact
address of the file in the file system, starting from the root.
Example : c:\Students\MyFolder\[Link] refers to file name [Link] in
under root directory c:\
• A Relative path starts with respect to present working directory.
Example : \MyFolder\[Link] refers to file name [Link] in under present
working directory MyFolder

2.3 Creating and Reading Text Data

• In real world applications, computer programs deal with data coming from different
sources like databases, CSV files, HTML, XML, JSON, etc.
• We broadly access files either to write or read data from it. But operations on files include
creating and opening a file, writing data in a file, traversing a file, reading data from a file
and so on.

2.3.1 Creating and Opening Text Files

To open a file in Python, we use the open() function. The syntax of open() is as follows
file_object= open(file_name, access_mode)

Here, file_name is the name of the file or the location of the file that you want to open,
and file_name should have the file extension included as well.
Which means in [Link] – the term test is the name of the file and .txt is the extension
of the file.
The mode in the open function syntax will tell Python as what operation you want to do on a
file.

• ‘r’ – Read Mode: Read mode is used only to read data from the file.
• ‘w’ – Write Mode: This mode is used when you want to write data into the file or modify
it. Remember write mode overwrites the data present in the file.
• ‘a’ – Append Mode: Append mode is used to append data to the file. Remember data
will be appended at the end of the file pointer.
• ‘r+’ – Read or Write Mode: This mode is used when we want to write or read the data
from the same file.
• ‘a+’ – Append or Read Mode: This mode is used when we want to read data from the file
or append the data into the same file.
9
2.3.2 Closing a File
In order to close a file, we must first open the file. In python, we have an in-built method
called close() to close the file which is opened.

Whenever you open a file, it is important to close it, especially, with write method. Because
if we don’t call the close function after the write method then whatever data we have written
to a file will not be saved into the file.

The syntax of close() is:

file_object.close()

2.3.3 File Methods to Read and Write Data

Description
Method Syntax

The read() method of file object

reads whole data from the open
file. But, optionally we can also
read() file_object.read(size)
specify the number of characters
(size) to be read.

The readline() method

of file object reads single line in a
file. But, optionally we can also
readline() file_object.readline(size)
specify the number of characters to
be read.

The readlines() method is used

readlines() file_object.readlines() to read all the lines of file .

The write() method Writes a

write() file_object.write(string) string of data to the file

The writelines() method

file_object. writes a sequence of strings to the
writelines()
writelines(sequence) file.

10
 The tell() method returns the
tell() file_object.tell()
current position within the file.

The seek() method is used to

position the file object at a
particular position in a file.
offset is the number of bytes by
which the file object is to be
moved.
file_object.seek(offset, reference_point have any of
seek() the following values :
reference_point)
0 - beginning of the file
1 - current position of the file
2 - end of file

By default, the value of

reference_point is 0.

Example Program on write and read method :

f=open("E:/[Link]","w")
[Link]("Hello World.\nHello Python")
[Link]()
f=open("E:/[Link]","r")
print([Link]())
[Link]()

11
Example Program on write and read method with specific size :

f=open("E:/[Link]","w")
[Link]("Hello World.\nHello Python")
[Link]()
f=open("E:/[Link]","r")
print([Link](5))
[Link]()

Example Program on writelines and readline method :

f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline strings\n",
"This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
print([Link]())
[Link]()

Example Program on writelines and readline method with specific size :

f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline strings\n",
"This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
print([Link](7))
[Link]()

12
Example Program on writelines and readlines method :
f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline
strings\n", "This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
print([Link]())
[Link]()

Example Program on split method :

split() function /method is used to display each word of line separately as an element of a
list.

f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline strings\n",
"This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
d=[Link]()
for line in d:

words=[Link]()
print(words)
[Link]()

13
Example Program on splitlines method :
splitlines() function /method is used to display each line separately as an element of a lsit.

f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline strings\n",
"This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
d=[Link]()
for line in d:
words=[Link]()
print(words)
[Link]()

Example Program on tell() and seek() method :

f=open("E:/[Link]","w")
lines = ["Hello everyone\n", "Writing multiline strings\n",
"This is the third line"]
[Link](lines)
[Link]()
f=open("E:/[Link]","r")
print([Link]())
print("\nInitially, the position of the file object is: ",[Link]())
[Link](0)
print("\nNow the file object is at the beginning of the file:
",[Link]())
[Link](6)
print("\nThe position of the file object is at: ",[Link]())
print()
print([Link]())

14
2.3.4 Opening a file using with clause / Statements
In Python, we can also open a file using with clause . The syntax of with clause is:
with open (file_name, access_mode) as file_ object:
• The advantage of using with clause is that any file that is opened using this clause is
closed automatically, once the control comes outside the with clause.
• In case the user forgets to close the file explicitly or if an exception occurs, the file is
closed automatically.

def writefile():
with open("E:/[Link]", "w") as f:
[Link]("Python is a general-purpose high-level
language")
def readfile():
with open("E:/[Link]", "r") as f:
print([Link]())
def main():
writefile()
readfile()
if __name__=="__main__":
main()

15
2.4 Reading and Writing Binary Files
Access modes in Binary files.
• ‘wb’ – Open a file for write only mode in the binary format.
• ‘rb’ – Open a file for the read-only mode in the binary format.
• ‘ab’ – Open a file for appending only mode in the binary format.
• ‘rb+’ – Open a file for read and write only mode in the binary format.
• ‘ab+’ – Open a file for appending and read-only mode in the binary format.

Example 1:

f = open("E:/[Link]", "wb")
num=[10,20,30,40,50]
arr=bytearray(num)
[Link](arr)
f = open("E:/[Link]", "rb")
print([Link]())

Example 2:

f = open("E:/[Link]", "wb+")
message = "Hello Python"
file_encode = [Link]("ASCII")
[Link](file_encode)
[Link](0)
bindata = [Link]()
print("Binary Data:",bindata)
txtdata = [Link]("ASCII")
print("Normal data:", txtdata)

16
2.5 The Pickle Module
• Python objects (list, tuple, dictionary, etc ) can be serialized into binary form and
deserialized back to Python objects using the methods and classes available from the
Python module pickle.
• Serialization or pickling is the process of converting an object in memory to a byte stream
that can be stored on disk or sent over a network.
• De-serialization or unpickling is the inverse of pickling process where a byte stream is
converted back to Python object.
• The pickle module deals with binary files. The pickle module provides two methods -
dump() and load() to work with binary files for pickling and unpickling,
respectively.

2.5.1 The dump() method (Pickling)

• This method is used to convert (pickling) Python objects for writing data in a binary file.
• The file in which data are to be dumped, needs to be opened in binary write mode (wb).

Syntax of dump() is as follows:

dump(data_object, file_object)

where data_object is the object that has to be dumped to the file with the file handle
named file_object.

2.5.2 The load() method (Unpickling)

• load() method is used to read data from a binary file.
• The file to be loaded is opened in binary read (rb) mode.

Syntax of load() is as follows:

Store_object = load(file_object)
Here, the pickled Python object is loaded from the file having a file handle named
file_object and is stored in a new file handle called store_object.

17
Example program on dump() and load():

import pickle
with open("[Link]","wb") as f:
list=[10,"Santhosh",1992]
dic={"Subject":"Python","Exp":10}
[Link](list,f)
[Link](dic,f)
with open("[Link]", "rb") as f:
lst=[Link](f)
d=[Link](f)
print(lst)
print(d)

2.6 Reading and Writing CSV Files

What is CSV File?
• A Comma Separated Values (CSV) file is a plain text file that contains the comma-
separated data.
• These files are often used for exchanging data between different applications.
• CSV files are usually created by programs that handle huge amounts of data. They are
used to export data from spreadsheets (ex:- excel file) and databases (Ex:- Oracle,
MySQL). It can be used to import data into a spreadsheet or a database.
• CSV can be easily read and processed by Python. CSV files have .csv extension.

Some advantages of CSV files are :

• It is in a human-readable format and is easier to edit manually
• It is simple to generate, parse and handle
• It is standard format and is supported by many applications.

18
CSV File Structure

Python CSV Module

• CSV Module is available in Python Standard Library.
• The CSV module contains classes that are used to read and write tabular form of data into
CSV format.
• To work with CSV Files, programmer have to import CSV Module
• Methods of csv module : writer() and reader() , Both the methods return an Object
of writer or reader class

2.6.1 writer()
• To write to a CSV file in Python, we can use the [Link]() function.
• The [Link]() function returns a writer object that converts the user's data into
a delimited string.
• The writer class has following methods:
writerow() :- This function writes items in a sequence (list, tuple or string) separating
them by comma character (i.e. Write a single line)

writerows() :- This function writes each sequence in a list as a comma separated line
of items in the file (i.e. Write Multiple lines)

Note :
Problem with writer() method is extra newline will appear after each line in csv file.
To remove this we have to open the csv file with newline parameter as empty string (‘ ‘).

2.6.2 reader()
• This function returns a reader object which is an iterator of lines in the csv file. We can
use a for loop to display lines in the file. The file should be opened in 'r' mode.

19
Example program on writerow() and reader()

import csv
row=['Santhosh','Faculty','10']
f=open("[Link]",'w', newline='')
wobj=[Link](f)
[Link](row)
[Link]()
f=open("[Link]",'r')
robj=[Link](f)
for data in robj:
print(data)
[Link]()

Example program on writerows() and reader()

import csv
rows=[['Santhosh','Java','5'],
['Santhosh','Data Structure','3'],
['Santhosh','Python','2']]
f=open("[Link]",'w',newline='')
wobj=[Link](f)
[Link](rows)
[Link]()
f=open("[Link]",'r')
robj=[Link](f)
for data in robj:
print(data)
[Link]()

20
Example program on csv file :

import csv
f=open("[Link]","w",newline='')
swriter=[Link](f)
[Link](['Rollno','Name','Marks'])
rec=[]
while True:
r=int(input("Enter Roll no : "))
n=input("Enter Name : ")
m=int(input("Enter Marks : "))
list=[r,n,m]
[Link](list)
ch=input("Do you want to enter more records : (y/n)")
if ch=='n':
break
for i in rec:
[Link](i)
[Link]()
f=open("[Link]","r")
sreader=[Link](f)
print("Student Details are : ")
for i in sreader:
print(i)
[Link]()

21
2.6.3 writing and reading into a file using Dictionary

• The dictionary data can be read and written to a csv file using DictReader() and
DictWriter() classes.

Syntax of DictReader :

[Link](fileobj,fieldnames=None,restkey=None)

Syntax of DictWriter :

[Link](fileobj,fieldnames)

Example Program using DictReader() and DictWriter()

import csv
my_dictionary=[{'name':'santhosh','subject':'java','Semester':5},
{'name':'santhosh','subject':'Python','Semester':2},
{'name':'santhosh','subject':'C++','Semester':2}]

my_attributes=['name','subject','Semester']

with open('[Link]','w',newline='') as f:
wobj=[Link](f,fieldnames=my_attributes)
[Link]()
[Link](my_dictionary)

with open('[Link]','r') as f:
robj=[Link](f)
for i in robj:
print(i)

22
Example Program to display specific key values using DictReader() and
DictWriter() .

import csv
my_dictionary=[{'name':'santhosh','subject':'java','Semester':5},
{'name':'santhosh','subject':'Python','Semester':2},
{'name':'santhosh','subject':'C++','Semester':2}]

my_attributes=['name','subject','Semester']

with open('[Link]','w',newline='') as f:
wobj=[Link](f,fieldnames=my_attributes)
[Link]()
[Link](my_dictionary)

with open('[Link]','r') as f:
robj=[Link](f)
for i in robj:
print(i['subject'])

2.7 Python os and [Link] Modules

• Python os module provides easy functions that allow us to interact and get Operating
System information.
• The os module in Python provides functions for creating and removing a directory
(folder), fetching its contents, changing and identifying the current directory, etc
• The [Link] module contains many helpful functions related to filenames and file paths.
[Link] is a module inside the os module.

23
2.7.1 Various methods of os Module

Methods Syntax Description

chdir() [Link](path) This method changes the current working directory to
the path we specify
getcwd() [Link]() This method returns the current working directory of a
process
mkdir [Link](path) This method creates the directory named path. If the
directory already exits, FileExistsError is raised.
remove() [Link](path) This method removes the specified file path. If that
path is a directory, it raises an OSError.
rmdir() [Link](path) This method removes the directory path specified. If
the directory isn’t empty, however, it raises an
OSError.
[Link] allows us to generate file names in a
directory tree by “walking the tree” – (we can
get folders, subfolders, and files at every
level). A tuple of three elements is returned for
each directory in the tree.

• dirPath – this is path to the current

directory
• dirNames – these are subdirectories in
the current directory(for dirPath)
• fileNames – this is a list of the names of
the non-directory files in the current
walk() directory(for dirPath)
[Link]
It has the following parameters:

1. topdown − By default, directories are

searched from the top-to-down sequence,
and topdown is set to true.
Directories are scanned from the bottom to
up sequence if set to False.
This is optional.
2. onerror − onerror can either throw an
exception to end the walk or show an error
to continue.
This is optional. By default, it is set
to None..

24
 3. followlinks − If followlinks is true, this
navigates to directories that our mentioned
links point to.
This is optional. By default, it is set
to false.

4. Top - Each directory rooted at directory

rename() [Link] This method is used rename the file from

(oldname oldname to newname.
,newname)
listdir() [Link](path) This method returns the list of all files and
directories in the specified directory.

25
Example Program on os module methods

import os
print([Link]())
[Link]("E:\Avanthi")
[Link]("E:\Avanthi")
print([Link]())
print([Link]("E:"))
[Link]("..")
[Link]("E:\Avanthi")

2.7.2 Various methods of [Link] Module

Methods Syntax Description

join() [Link](path, This method is used to join one or more path
*paths) components intelligently. The return value is
the concatenation of path and any
members of *paths with exactly one
directory.
exists() [Link](path) This method returns True if path refers to an
existing path else returns False for broken
links.
isfile() [Link](path) This method returns True if path is an existing
regular file.
isdir() [Link](path) This method returns True if path is an existing
directory
getmtime() [Link](path) This method returns the time of last
modification of path.
abspath() [Link](path) This method returns a normalized absolutized
version of the pathname path.
[Link]() [Link](path) This method returns True if path is an
absolute pathname

26
 relpath() [Link](path, This method returns a relative filepath to path
start=[Link]) either from the current directory or from an
optional start directory.
dirname() [Link](path) This method returns the directory name of
the pathname path.
basename() [Link](path) This method returns the base name of
pathname path.
split() [Link](path) This method splits the pathname path into a
pair, (head, tail)
splitext() [Link](path) This method splits the pathname path into a
pair (root, ext) such that
root + ext == path where ext begins with a
period and contains at most one period and
root is everything leading up to that.
getsize() [Link](path) This method returns the size, in bytes, of
path.

Example Program on [Link] module methods

import os
filename = 'BSC/MSDS/[Link]'
print("split :",[Link](filename))
print("splitext :",[Link](filename))
print("Directory name :",[Link](filename))
print("Base name :",[Link](filename))
print("join",[Link]([Link](filename),
[Link](filename)))
print("Relative Path: ",[Link](filename))
print("Absoulute Path: ",[Link](filename))
print("isdir : ",[Link](filename))

27
Consider the File Structure Given Below. Write Python Program to Delete All the Files and
Subdirectories from the main (i.e. Data Science) Directory.

import os
def deletefiles(path):
for root,dirs,files in [Link](path):
for file in files:
file_path=[Link](root,file)
print(file_path," is deleted")
[Link](file_path)

def main():
path=input("Enter the directory path you want to delete files")
deletefiles(path)
if __name__=="__main__":
main()

28
 Chapter 3
Working with Text Data
3.1 JSON and XML in Python
• JSON (JavaScript Object Notation) and XML (Extensible Markup Language) standards are
commonly used for transmitting data in web applications.
• The web is based on very basic client/server architecture , client (usually a web browser )
sends a request to a server, using Hypertext Transfer Protocol (HTTP). The server answers
the request using the same protocol.

HTTP(S)

3.1 Using JSON with Python

• JSON (JavaScript Object Notation) is a lightweight text-based data-interchange format.
• It is simple for humans to read and write and easy enough for machines to parse and
generate.
• Douglas Crockford specified the JSON format .
• It stores data as a quoted string in a key: value pair within curly brackets. Python provides
support for JSON objects through a built-in package called “json”.
• Using json python package , we can convert JSON files to Python Objects and vice versa,
to share data on Python web apps.
Rules for JSON syntax are:

• Data should be in key : value pairs

• Data should be separated by commas
• Curly braces should hold objects
• Square brackets hold arrays

{ key : value, key : value…..}

JSON is very similar to Python dictionary. Python has an inbuilt library as a JSON.

29
JSON serialization and deserialization are the processes of converting JSON data to
and from other formats, such as Python objects or strings, to transmit or store the data.
3.1.1 Python to JSON (Encoding / Serializing)
• Serialization / Encoding is the process of converting an object or data structure into a
JSON string.
• Here are some common functions from json library that are used for serialization :
dumps() and dump()

[Link]()

This function is used to serialize a Python object into a JSON string. The dumps() function
takes a single argument, the Python object, and returns a JSON string

Syntax :
json_string = [Link](python_obj)

[Link]()

This function is used to serialize a Python object and write it to a JSON file. The dump()
function takes two arguments, the Python object and the file object.
Syntax :
[Link](python_obj,file_object)

3.1.2 JSON to Python (Decoding / DeSerializing)

• Deserialization / Decoding is the process of converting a JSON string back into an
object or data structure.
• Here are some common functions from json library that are used for deserialization :
loads() and load()

[Link]()

This function is used to parse a JSON string into a Python object. The loads() function
takes a single argument, the JSON string, and returns a Python object.
Syntax :
python_obj = [Link](json_string)
[Link]()
This function is used to read a JSON file and parse its contents into a Python object. The
load() function takes a single argument, the file object, and returns a Python object.
Syntax :
python_obj = [Link](file_object)

30
Program to demonstrate Python Serializing using JSON dumps() method and
Deserializing using JSON loads() method.

import json
python_obj = {"name":"Santhosh",
"Subject":"Data Engineering with Python",
"Semester":3}
json_string = [Link](python_obj)
python_obj = [Link](json_string)
print(python_obj)

Program to demonstrate Python Serializing using JSON dump() method and

Deserializing using JSON load() method.

import json
python_obj = {"name":"Santhosh",
"Subject":"Data Engineering with Python",
"Semester":3}
with open("[Link]",'w') as file:
[Link](python_obj, file)
with open('[Link]', 'r') as file:
python_obj = [Link](file)
print(python_obj)

31
Formatting JSON Data
In Python, the [Link]() function provides options for formatting and ordering the JSON
output. Here are some common options:

1. Indent
This option specifies the number of spaces to use for indentation in the output JSON string.
2. sort_keys
This option specifies whether the keys in the output JSON string should be sorted in
alphabetical order

import json
python_obj = {"name":"Avanthi",
"age":1991,
"semester":"Three"}
json_string =
[Link](python_obj,indent=2,sort_keys=True)
print(json_string)

32
3.1.3 Using Requests Module
• The requests module allows you to send HTTP requests using Python.
• The system that sends requests is known as the client, and the system that holds the
webserver is known as a server.
• The HTTP request returns a Response Object with all the response data (content,
encoding, status, etc).
• While working with the requests, we will come across the following methods.
get - It is used to request data from a server.
post - It is used to submit some data to the server for processing it.
• The response object will store the information. Below are the essential properties.
content - It returns the content of the data for the responses.
status_code - It returns the status of our request. For example - 200 OK means you
made successful request, 404 NOT FOUND means resource is not
found.
To install Requests, simply run this simple command in command prompt
C:\>pip install requests
To import module in pycharm
Type the name of the package (import requests) and hit Alt-Enter , then choose Install
and import package

Program to get Text Response content, status_code and url response using
requests module

import requests
resp = [Link]('[Link]
print(resp.status_code)
print([Link])
print([Link])

33
Program to get JSON response content using requests module

import requests
response = [Link]('[Link]
print(response)
print([Link]()) #returns a JSON object of the result

3.2 Using XML with Python

• XML stands for eXtensible Markup Language. It was designed to store and exchange wide
variety of data on the web and elsewhere.
• An XML document is a universal format for data on the web.
• XML documents have an .xml extension.
• Developers use XML format for Reuse, Portability (i.e. Platform-independent) ,
Interchange and Self-describing (i.e. Human-readable format).
Syntax rules of XML document :
• All XML elements must have a closing tag.
Incorrect : <faculty>Santhosh
Correct : <faculty>Santhosh</faculty>
• XML tags are case sensitive.
Incorrect : <Faculty>Santhosh</faculty>
Correct : <faculty>Santhosh</faculty>
• All XML elements must be properly nested.
Incorrect:
<department>
<faculty> Santhosh </department>
</faculty>
Correct:
<department>
<faculty> Santhosh </faculty>
</department>
34
 • All XML documents must have a root element.
Example :
<root>
<child>
<subchild>….</subchild>
</child>
</root>

• Attribute values must always be quoted.

Incorrect:
<subject name=DataEngineering> Semester3 </subject>
Correct:
<subject name=”DataEngineering”> Semester3 </subject>

Introduction to ElementTree
Python has a built in library, ElementTree, that has functions to read and manipulate XMLs (and
other similarly structured files). ElementTree module available in [Link] package.
First, import ElementTree. It's a common practice to use the alias of ET:
import [Link] as ET

Parsing XML Data

The module provides two primary methods for parsing XML
data: parse() and fromstring().The parse() function reads an XML file, while
the fromstring() function reads an XML string.

Syntax of parse() :

ElementTree_Object = [Link]('[Link]')

[Link]() function takes the file name as an argument and returns an ElementTree object.

35
Program 1 : Construct an XML File and Write Python Program to
parse the XML Data

[Link]

<student>
<name type="item1">Avanthi</name>
<Rollno type="item2">1991</Rollno>
</student>

[Link]

import [Link] as ET
tree = [Link]('[Link]')
root = [Link]()
print(root)

Syntax of fromstring() :

root = [Link](xml_string)

The fromstring() function returns the root Element of the parsed XML tree.

Program 2 : Construct an XML Formatted Data and write Python Program to

parse the XML Data

import [Link] as ET
root=[Link]('''<Student>
<name>Avanthi</name>
<rollno>1991</rollno>
</Student>''')
print("Root element name : ",[Link])
print("Sub element contents are :")
for elem in root:
print([Link])
print("Sub element contents are :")
for content in root:
print([Link])
36
Program 3 : Write a Python program to Generate XML formatted Data and
Save it as XML Document and displays tag names.

import [Link] as ET
root=[Link]("subjects")
child=[Link](root,"semester")
subchild_1=[Link](child,"first",{"name":"sub1"})
subchild_2=[Link](child,"second",{"name":"sub2"})
subchild_1.text="FIT"
subchild_2.text="Python"
tree=[Link](root)
[Link]("[Link]")
content=[Link]("[Link]")
data=[Link]()
print(data) #returns root element
print([Link]) #returns root tag name
print(data[0].tag) #returns child tag name
print(data[0][0].tag) # returns subchild1 tag name
print(data[0][1].tag) # returns subchild2 tag name

37
 [Link]

Unit-I Questions
1. Explain about data analysis sequence
2. What is Data Acquisition? Explain about data acquisition pipeline
3. Write a short note on Data Science and report structure
4. What is File? Explain different types of files
5. Write short note on File paths and File methods to reading and writing data
in Python.
6. Working with Text files with example programs
7. Working with Binary files with example programs
8. Explain about pickle module with an example program.
9. Working with CSV files with example programs.
10. os Module & [Link] module methods with an example programs.
11. Working with JSON file with example programs.
12. Working with XML files with example programs.

38