Python for Data Science (3150713) 240183107018

A
Laboratory Manual
for

Python for Data Science (3150713)

B.E. Semester 5
(Computer Engineering)

Directorate of Technical Education, Gandhinagar,

Gujarat

1
Python for Data Science (3150713) 240183107018

Vishwakarma Government Engineering College, Chandkheda

Computer Engineering Department

Certificate

This is to certify that Ms. Charmi K. Panchal Enrollment No. 240173107018

of B.E. Semester 5 Computer Engineering of this Institute (GTU Code: 017)
has satisfactorily completed the Practical / Tutorial work for the subject
Python for Data Science (3150713) for the academic year 2025-26.

Place:

Date:

Sign of Faculty member Head of the Department

2
 Preface

Main motto of any laboratory/practical/field work is for enhancing required skills as well as
creating ability amongst students to solve real time problem by developing relevant
competencies in psychomotor domain. By keeping in view, GTU has designed competency
focused outcome-based curriculum for engineering degree programs where sufficient weightage
is given to practical work. It shows importance of enhancement of skills amongst the students
and it pays attention to utilize every second of time allotted for practical amongst students,
instructors and faculty members to achieve relevant outcomes by performing the experiments
rather than having merely study type experiments. It is must for effective implementation of
competency focused outcome-based curriculum that every practical is keenly designed to serve
as a tool to develop and enhance relevant competency required by the various industry among
every student. These psychomotor skills are very difficult to develop through traditional chalk
and board content delivery method in the classroom. Accordingly, this lab manual is designed
to focus on the industry defined relevant outcomes, rather than old practice of conducting
practical to prove concept and theory.

By using this lab manual students can go through the relevant theory and procedure in advance
before the actual performance which creates an interest and students can have basic idea prior to
performance. This in turn enhances pre-determined outcomes amongst students. Each
experiment in this manual begins with competency, industry relevant skills, course outcomes as
well as practical outcomes (objectives). The students will also achieve safety and necessary
precautions to be taken while performing practical.

This manual also provides guidelines to faculty members to facilitate student centric lab
activities through each experiment by arranging and managing necessary resources in order that
the students follow the procedures with required safety and necessary precautions to achieve the
outcomes. It also gives an idea that how students will be assessed by providing rubrics.

Data Science is about data gathering, analysis and decision-making. Data Science is about
finding patterns in data, through analysis, and make future predictions. By using Data Science,
companies are able to make:

 Better decisions (should we choose A or B)

 Predictive analysis (what will happen next?)
 Pattern discoveries (find pattern, or maybe hidden information in the data)

Data Science is used in many industries in the world today, e.g. banking, consultancy,
healthcare, and manufacturing. Python is an open-source, interpreted, high-level language and
provides a great approach to data science, machine learning, and research purposes. It is one of
the best languages for data science to use for various applications & projects. When it comes to
dealing with mathematical, statistical, and scientific functions, Python has great utility.

Utmost care has been taken while preparing this lab manual however always there is chances of
improvement. Therefore, we welcome constructive suggestions for improvement and removal
of errors if any.

3
Python for Data Science (3150713) 240173107018

Practical – Course Outcome matrix

Course Outcomes (COs):

1. Apply various Python data structures to effectively manage various types of data.
2. Explore various steps of data science pipeline with role of Python.
3. Design applications applying various operations for data cleansing and transformation.
4. Use various data visualization tools for effective interpretations and insights of data.
5. Perform data Wrangling with Scikit-learn applying exploratory data analysis.

Sr.
Objective(s) of Experiment CO1 CO2 CO3 CO4 CO5
No.
Write python script for following
a) To understand the control structures of python.
1. b) To learn different types of data structures (list, √
dictionary, tuples) in python.

Develop a program that reads a .csv dataset file using Pandas

library and display the following content of the dataset.
a) First five rows of the dataset
2. b) Complete data of the dataset √ √
c) Summary or metadata of the dataset.

Develop a program to perform following operation on the

dataset.
3. a) To shows application of slicing and dicing over the √ √
rows and columns of the dataset.
b) To apply split and merge operations on the dataset.

Develop a program that shows usage of aggregate function

over the input dataset.
4. a) describe b) max c) min d) mean e) median √ √
f) count g) std h) Corr

Develop a program that shows the various data cleaning

tasks over the dataset.
5. a) Identifying the null values √ √ √
b) Identifying the empty values
c) Identifying the incorrect timestamp

Develop a program that shows usage of following NumPy

array operations: a) any() b) all() c) isnan() d) isinf() e)
isfinite() f) isinf() g) zeros() h) isreal() i) iscomplex()
6. j) isscalar() k) less() l) greater() m) less_equal() n) √
greater_equal() and vector functions: a) arrange() b)
reshape() c) linspace() d) randint() e) dot()

Write python script using matplotlib library for following:

a) To display line plot for given values of X:
[1,2,3,...,49] and values of Y (thrice of X):
7. [3,6,9,12,...,144,147] √
b) To display bar plot for given value Languages =
['Java', 'Python', 'PHP', 'JavaScript', 'C#', 'C++'] and
popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7]

4
Python for Data Science (3150713) 240173107018

c) To display pie plot for given value Languages =

['Java', 'Python', 'PHP', 'JavaScript', 'C#', 'C++'] and
popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7] using colors
= ["#1f77b4", "#ff7f0e", "#2ca02c", "#d62728",
"#9467bd", "#8c564b"]
d) To display scatter plot for 200 random points for
both X and Y axis.

Develop a program that reads .csv file from the url:

([Link]
8. mple [Link]?raw=true) and plot the data of the dataset √ √
stored in the .csv file.

Write a text classification pipeline using a custom

preprocessor and CharNGramAnalyzer using data from
Wikipedia articles as a training set.
9. √ √ √
a) Evaluate the performance on some held out test sets

Write a text classification pipeline to classify movie reviews

as either positive or negative.
10. √ √ √
a) Find a good set of parameters using grid search.
b) Evaluate the performance on a held out test set.

5
Python for Data Science (3150713) 240173107018

Industry Relevant Skills

The following industry relevant competencies are expected to be developed in the student by
undertaking the practical work of this laboratory.
1. Programming Languages
2. Mathematics, Statistical Analysis, and Probability
3. Data Mining
4. Machine Learning and AI
5. Data Visualization
Guidelines for Faculty members
1. Teacher should provide the guideline with demonstration of practical to the students
with all features.
2. Teacher shall explain basic concepts/theory related to the experiment to the students
before starting of each practical
3. Involve all the students in performance of each experiment.
4. Teacher is expected to share the skills and competencies to be developed in the
students and ensure that the respective skills and competencies are developed in the
students after the completion of the experimentation.
5. Teachers should give opportunity to students for hands-on experience after the
demonstration.
6. Teacher may provide additional knowledge and skills to the students even though not
covered in the manual but are expected from the students by concerned industry.
7. Give practical assignment and assess the performance of students based on task
assigned to check whether it is as per the instructions or not.
8. Teacher is expected to refer complete curriculum of the course and follow the
guidelines for implementation.
Instructions for Students
1. Students are expected to carefully listen to all the theory classes delivered by the faculty
members and understand the COs, content of the course, teaching and examination scheme,
skill set to be developed etc.
2. Students shall organize the work in the group and make record of all observations.
3. Students shall develop maintenance skill as expected by industries.
4. Student shall attempt to develop related hand-on skills and build confidence.
5. Students shall make a small project/application in Python.
6. Student shall develop the habits of evolving more ideas, innovations, skills etc. apart from
those included in scope of manual.
7. Student shall refer technical magazines and data books.

Common Safety Instructions

Students are expected to
1. Switch on the PC carefully (not to use wet hands)
2. Shutdown the PC properly at the end of your Lab
3. Carefully Handle the peripherals (Mouse, Keyboard, Network cable etc)
4. Use Laptop in lab after getting permission from Teacher

6
Python for Data Science (3150713) 240173107018

Index
(Progressive Assessment Sheet)

Sr. Objective(s) of Experiment Page Date Date Assess Sign. of Rema

No. No. of of ment Teacher rks
perfor submi Marks with
mance ssion date
Write python script for following
a) To understand the control structures of
python.
1. b) To learn different types of data structures
(list, dictionary, tuples) in python.

Develop a program that reads a .csv dataset file

using Pandas library and display the following
content of the dataset.
2. a) First five rows of the dataset
b) Complete data of the dataset
c) Summary or metadata of the dataset.

Develop a program to perform following

operation on the dataset.
a) To shows application of slicing and dicing
3. over the rows and columns of the dataset.
b) To apply split and merge operations on the
dataset.

Develop a program that shows usage of

aggregate function over the input dataset.
4. a) describe b) max c) min d) mean e)
median f) count g) std h) Corr

Develop a program that shows the various data

cleaning tasks over the dataset.
5. a) Identifying the null values
b) Identifying the empty values
c) Identifying the incorrect timestamp

Develop a program that shows usage of

following NumPy array operations: a) any() b)
all() c) isnan() d) isinf() e) isfinite() f)
isinf() g) zeros() h) isreal() i) iscomplex()
6. j) isscalar() k) less() l) greater() m)
less_equal() n) greater_equal() and vector
functions: a) arrange() b) reshape() c) linspace()
d) randint() e) dot()

Write python script using matplotlib library

for following:
7. a) To display line plot for given values of
X: [1,2,3,...,49] and values of Y (thrice of
X): [3,6,9,12,...,144,147]

7
Python for Data Science (3150713) 240173107018

b) To display bar plot for given value

Languages = ['Java', 'Python', 'PHP',
'JavaScript', 'C#', 'C++'] and popularity =
[22.2, 17.6, 8.8, 8, 7.7, 6.7]
c) To display pie plot for given value
Languages = ['Java', 'Python', 'PHP',
'JavaScript', 'C#', 'C++'] and popularity =
[22.2, 17.6, 8.8, 8, 7.7, 6.7] using colors =
["#1f77b4", "#ff7f0e", "#2ca02c",
"#d62728", "#9467bd", "#8c564b"]
d) To display scatter plot for 200 random
points for both X and Y axis.

Develop a program that reads .csv file from the

url:
8. ([Link]
aster/data/sample [Link]?raw=true) and
plot the data of the dataset stored in the .csv file.

Write a text classification pipeline using a

custom preprocessor and CharNGramAnalyzer
using data from Wikipedia articles as a training
set.
9.
a) Evaluate the performance on some held out
test sets

Write a text classification pipeline to classify

movie reviews as either positive or negative.
a) Find a good set of parameters using grid
10.
search.
b) Evaluate the performance on a held out test
set.

Total

8
Python for Data Science (3150713) 240173107018

Experiment No: 1

Write python script for following

a) To understand the control structures of python.
b) To learn different types of data structures (list, dictionary, tuples) in
python.

Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces (GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Basic programming concepts: You should have a good grasp of basic programming
concepts such as variables, data types, conditional statements, loops, and functions.
 Python programming language: You should have a good understanding of Python syntax,
data structures, and standard library functions.
 Sequences: Sequences are ordered collections of elements that can be accessed by their
index or key. You should have a good understanding of the different types of sequences
such as string, tuple, list, dictionary, and set, and their respective properties.
 String manipulation: You should know how to manipulate them using methods such as
slicing, concatenation, and formatting.
 Collection manipulation: Collections such as lists, tuples, dictionaries, and sets can be
manipulated using methods such as append, insert, remove, pop, and sort.
 Iteration: You should know how to use for loops and list comprehensions to iterate over
sequences.
 Conditional statements: You should know how to use conditional statements to check for
specific conditions in sequences.
 Functions: You should know how to define functions that operate on sequences and return
values.

Relevant CO: CO1

Objectives:

(a) To learn and understand the different control structures in Python, such as loops,
conditional statements, and functions.
(b) To learn how to manipulate and access their elements, iterate over them, perform
conditional operations on them, and use them in functions.
(c) To learn how to select the appropriate sequence type for a given task based on its
properties and performance characteristics.

Equipment/Instruments: Personal Computer, Internet, Python

9
Python for Data Science (3150713) 240173107018

Theory:
Conditional statements in Python allow you to execute certain blocks of code based on whether a
certain condition is true or false. The two main types of conditional statements in Python are "if"
statements and "if-else" statements.

Loops in Python allow you to repeat a block of code multiple times, either for a fixed number of
times or until a certain condition is met. The two main types of loops in Python are "for" loops
and "while" loops.

Functions in Python allow you to encapsulate blocks of code and reuse them throughout your
program. Functions can accept parameters and return values, making them a powerful tool for
organizing and structuring your code.

Scope in Python refers to the region of your program where a variable or function is visible and
accessible. Understanding scope is critical for avoiding errors and ensuring that your code is
organized and easy to maintain.

Error handling in Python involves detecting and responding to errors that may occur during
program execution. Proper error handling can help you avoid crashes and ensure that your
program continues to run smoothly.

String data type in Python represents a sequence of characters and is immutable, meaning its
contents cannot be changed once it is created. Strings can be manipulated using various methods
such as slicing, concatenation, and formatting.

Lists and tuples are similar in many ways, but tuples are immutable, whereas lists are mutable.
Lists and tuples can hold elements of any data type and can be indexed and sliced like strings.
However, lists offer additional methods such as append, insert, remove, and pop that allow for
manipulation of the list's contents.

Dictionaries are another important sequence type in Python and are implemented as unordered
collections of key-value pairs. Each element in a dictionary consists of a key and a corresponding
value. Dictionaries can be used to store and retrieve data quickly based on the key.

Sets are collections of unique elements that are unordered and mutable. Sets are often used to
perform set operations such as union, intersection, and difference.

Safety and necessary Precautions:

1. Data validation.
2. Check the data types.
3. Input sanitization.
4. Error Handling and Secure coding practices.
5. Use comments.
6. Test your code.

10
Python for Data Science (3150713) 240173107018

Procedure:
1. Plan the program structure and flow: Develop a plan for the program structure, including
the control structures that will be included, and the flow of the program logic.

2. Implement the control structures in Python: Write the code to implement the different
control structures in Python, including conditional statements, loops, and functions.

3. Create a string variable using single or double quotes. Use string methods like upper(),
lower(), strip(), split(), join(), and replace() to manipulate the string as needed. Use
indexing and slicing to access specific characters or substrings within the string.

4. Create a tuple variable using parentheses. Use indexing and slicing to access specific
elements or subsets within the tuple. Tuples are immutable, so you cannot add, remove or
modify elements once created.

5. Create a list variable using square brackets. Use indexing and slicing to access specific
elements or subsets within the list. Use list methods like append(), insert(), remove(),
pop(), extend(), and sort() to modify the list as needed. Lists are mutable, so you can add,
remove or modify elements once created.

6. Create a dictionary variable using curly braces or the dict() constructor. Use keys to access
values within the dictionary. Use dictionary methods like keys(), values(), and items() to
access different parts of the dictionary. Use del or pop() to remove elements from the
dictionary. Use assignment to add or modify elements in the dictionary.

7. Create a set variable using curly braces or the set() constructor. Use set methods like add(),
remove(), pop(), union(), and intersection() to modify or perform operations on the set.
Sets do not allow duplicate elements, so adding the same element multiple times will only
add it once.

Observations:
Code implementation:

# conntrol structures
age = 18
if age >= 18:
print("You are eligible to vote")
else:
print("You are not eligible to vote")

for i in range(1, 10):

print(i, end=" ")

def add(a=10, b=5):

return a + b

# strings in python
str1 = " kAraN "
print([Link]())
print([Link]())
print([Link]())
print([Link]("r", "V"))
print([Link]("a"))
11
Python for Data Science (3150713) 240173107018

print([Link]("123"))
print(str[0:3])

# tuples in python
tup = (1, 2, 3, 4, 5)
print(tup[0])
print(tup[1:3])

# lists in python
list1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, -1]
print(list1[0])
print(list1[1:3])
[Link](6)
[Link](2, 7)
[Link](3)
[Link]()
[Link]([11, 12, 13])
[Link]()

# dictionaries in python
dictionary = {"name": "Alice", "age": 20, "marks": 90}
print([Link]())
print([Link]())
print([Link]())
del dictionary["name"]
print([Link]("age"))

# sets in python
set1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
set2 = {1, 2, 3, 4, 5}
print([Link](11))
print([Link](1))
print([Link]())
print([Link](set2))
print([Link](set2))

12
Python for Data Science (3150713) 240173107018

Conclusion:
Understanding Python's control structures and data structures is crucial for effective programming.
Control structures manage flow, while data structures store and manipulate data efficiently.
Mastery of these concepts enables the development of robust, maintainable Python applications.

Suggested Reference:
1. [Link]
2. [Link]
3. [Link]
4. [Link]
5. [Link]

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

13
Python for Data Science (3150713) 240183107018

Experiment No: 2
Develop a program that reads a .csv dataset file using Pandas library and
display the following content of the dataset.
a) First five rows of the dataset
b) Complete data of the dataset
c) Summary or metadata of the dataset.

Date:
Competency and Practical Skills:

Competency skills:

 Knowledge of Python programming language and its libraries, particularly the Pandas
library.
 Understanding of the structure of .csv files and how to read and manipulate them using
Pandas.
 Familiarity with the different methods and functions available in Pandas, such as "head()",
"print()", "display()", "info()", and "describe()".
 Ability to write and debug code, and troubleshoot errors that may arise when working with
datasets.
 Experience in working with datasets, including data cleaning, data wrangling, and data
analysis.
 Ability to understand the content and structure of datasets, and use them to derive insights
and information.

Practical skills:

 Writing code to load a .csv dataset file into a Pandas DataFrame using the "read_csv()"
function.
 Using the "head()" method to display the first five rows of the dataset.
 Using the "print()" function or "display()" method to display the complete data of the
dataset.
 Using the "info()" method or "describe()" method to display the summary or metadata of
the dataset.
 Handling errors and exceptions that may arise when working with datasets.
 Writing clean and efficient code that is easy to read and maintain.
 Testing the program with different datasets to ensure its accuracy and reliability.

Relevant CO: CO1, CO2

Objectives: (a) To read and load the .csv dataset file into a Pandas DataFrame.
(b) To display the first five rows of the dataset using the "head()" method.
(c) To display the complete data of the dataset using the "print()" function or "display()" method.
(d) To display the summary or metadata of the dataset using the "info()" method or "describe()"
method.

Equipment/Instruments: Personal Computer, Internet, Python

Jhv v

14
Python for Data Science (3150713) 240183107018
Theory:
Pandas is a popular data manipulation library for Python, widely used in data science and machine
learning. It provides a powerful and flexible toolset for working with structured data, including
loading, manipulating, and analyzing datasets in various formats, including .csv files

Safety and necessary Precautions:

1. Data security, quality and privacy.

2. Memory and performance optimization.
3. Error handling and exception handling.
4. Use comments.
5. Test your code.

Procedure:
1. Import the Pandas library: To use the Pandas library in Python, it is essential to import it
into your program. You can do this by using the "import pandas as pd" statement.

2. Load the dataset: The next step is to load the dataset into a Pandas DataFrame using the
"read_csv()" function. This function takes the path to the .csv file as an argument and
returns a DataFrame object that contains the data from the file.

3. Display the first five rows: To display the first five rows of the dataset, you can use the
"head()" method. This method returns the first five rows of the DataFrame by default, but
you can specify the number of rows you want to display as an argument.

4. Display the complete data: To display the complete data of the dataset, you can use the
"print()" function or "display()" method. This will output the entire DataFrame to the
console or Jupyter Notebook.

5. Display summary or metadata: To display the summary or metadata of the dataset, you can
use the "info()" method or "describe()" method. The "info()" method provides information
about the DataFrame, including the number of rows and columns, data types, and memory
usage. The "describe()" method provides statistical summary of the dataset, including
count, mean, standard deviation, minimum, maximum, and quartiles for each column.
Observation:
# import the pandas library
import pandas as pd

# load the data using read_csv function

dataframe = pd.read_csv("generated_data.csv")

# print the first 5 rows of the dataframe

print([Link]())

# display first 10 rows of the dataframe

print([Link](10))

# displat last 5 rows of the dataframe

print([Link]())

# display the complete information of the dataframe

print(display(dataframe))
Jhv v

15
Python for Data Science (3150713) 240183107018
# display information of the dataframe
print([Link]())

Result:

Conclusion:
Using Pandas simplifies data handling by efficiently reading and summarizing large datasets. It
provides essential tools to quickly assess data quality and structure, facilitating deeper exploration
and preprocessing

Jhv v

16
Python for Data Science (3150713) 240183107018
Suggested Reference:
1. Official Pandas documentation: [Link]
2. "Python for Data Analysis" by Wes McKinney:
[Link]
3. "Python Data Science Handbook" by Jake VanderPlas:
[Link]
4. Pandas tutorial by DataCamp: [Link]
tutorial-dataframe-python

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

17
Python for Data Science (3150713) 240183107018

Experiment No: 3

Develop a program to perform following operation on the dataset.

a) To shows application of slicing and dicing over the rows and columns of the
dataset.
b) To apply split and merge operations on the dataset.

Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces (GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Knowledge of Python programming language.

 Familiarity with Pandas library.
 Ability to read and load dataset files.
 Familiarity with slicing and dicing operations.
 Understanding of data indexing.
 Familiarity with data cleaning and preprocessing.
 Knowledge of data visualization.
 Problem-solving skills.
 Strong analytical and statistical skills.

Relevant CO: CO1, CO2

Objectives:
a) To gain insights into the dataset and extract meaningful information from it.
b) To split large datasets into smaller ones for ease of handling and processing.
c) To consolidate information and make it easier to analyze.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
Slicing and dicing are powerful operations that allow data analysts to manipulate data by selecting
specific subsets of data from a larger dataset. These operations are widely used in data analysis
and are a crucial aspect of data manipulation.

In the context of Python, slicing refers to extracting specific portions of data from a larger data
structure, such as a list, tuple, or DataFrame. Slicing is performed by specifying the start and end
indices of the portion of data to be extracted. For example, in a list of numbers, slicing can be
used to extract the first three numbers or the last five numbers. In a DataFrame, slicing can be
used to extract specific rows or columns based on specific conditions or criteria.

Dicing, on the other hand, refers to grouping and aggregating data based on specific criteria. This
involves dividing the data into smaller subsets based on specific categories or conditions and
Jhv v

18
Python for Data Science (3150713) 240183107018
performing aggregation functions on each subset. For example, in a dataset containing sales data,
dicing can be used to group the data by product type, region, or time period and calculate the total
sales for each group.

In Python, the Pandas library provides powerful tools for slicing and dicing data in a DataFrame.
The .loc and .iloc methods are used for slicing rows and columns based on specific conditions or
criteria. The .groupby method is used for grouping data based on specific categories, and
aggregation functions such as .sum(), .mean(), and .count() can be used to perform calculations on
each group. The .pivot_table method is used for creating pivot tables, which provide a
summarized view of the data by grouping and aggregating data based on specific categories.

Splitting a Dataset: The split() method is a built-in function in Python that can be used to split a
string into a list of substrings based on a specified delimiter. This can be useful for splitting a
dataset into smaller chunks. The numpy.array_split() function can be used to split a numpy array
into smaller arrays of equal or nearly equal size.

Merging Datasets: The [Link]() function can be used to concatenate pandas dataframes
along a specified axis. The concatenate() function can be used to merge two or more arrays into a
single array.

Safety and necessary Precautions:

1. Backup the original data.
2. Validate the data.
3. Check the output.
4. Secure the data.
5. Use appropriate tools: the Pandas library provides powerful tools for data manipulation
and analysis.

Procedure:
1. Load the dataset: Load the dataset into Python using the Pandas library's read_csv function.

2. Explore the dataset: Use the head, tail, and info functions to explore the dataset and get a
sense of its structure and contents.

3. Slice and dice the data: Use the Pandas DataFrame's indexing and slicing operations to
select specific rows and columns of the dataset. Examples of slicing operations include loc,
iloc, and [ ].

4. Apply filtering: Use Boolean indexing to filter rows of the dataset based on specific
criteria.

5. Aggregate the data: Use the groupby function to group the data by specific columns and
apply aggregation functions such as sum, mean, and count.

6. Visualize the data: Use visualization libraries such as Matplotlib or Seaborn to create
visualizations of the sliced and diced data.

7. Refine and iterate: Refine the analysis and iterate as needed based on the insights gained
from the analysis.

8. Define the input datasets: Determine the input datasets and their format. It could be CSV
files, Excel files, or other file types. Also, define the delimiter or separator character for
splitting the data.

Jhv v

19
Python for Data Science (3150713) 240183107018

9. Load the datasets: Load the datasets into the program using the appropriate libraries and
functions. Check that the data is loaded correctly and perform any necessary data cleaning
or formatting.

10. Split the datasets: Use the appropriate function or library to split the datasets into smaller
chunks. Specify the size or number of chunks to create and ensure that the resulting
datasets are consistent and valid.

11. Merge the datasets: Use the appropriate function or library to merge the datasets into a
single dataset. Specify the method of merging and ensure that the resulting dataset is
consistent and valid.

12. Handle missing or duplicate data: Check for any missing or duplicate data in the merged
dataset and handle them appropriately. You can choose to remove the records with missing
data or impute the missing values.

13. Perform calculations or analysis: Once the datasets are merged, you can perform any
necessary calculations or analysis on the resulting dataset. This could include aggregating
data, calculating averages, or performing statistical analysis.

Observation:
# import pandas library
import pandas as pd
import [Link] as plt
import numpy as np

# load the data

dataframe = pd.read_csv('sample_data_100_records.csv')

# use of head, tail and info functions

print([Link]())
print([Link]())
print([Link]())

# slice and dice the data

print([Link][0:5, 'Name'])

# filter the data using boolean indexing

print(dataframe[dataframe["Name"] == "Jane Anderson"])

# use of groupby function

print([Link]('Name').count())
print([Link]('Age').sum())

# visualize the data

[Link]([Link]('Age').count())
[Link]()

# split the data into training and test data

from sklearn.model_selection import train_test_split
train, test = train_test_split(dataframe, test_size=0.2)
print(train)
Jhv v

20
Python for Data Science (3150713) 240183107018

# merge the data

dataframe1 = pd.read_csv('sample_data_100_records.csv')
dataframe2 = pd.read_csv('sample_data_100_records.csv')
print([Link](dataframe1, dataframe2, on='Name'))

# dictionary of lists
dict = {
"First Score": [100, 90, [Link], 95],
"Second Score": [30, 45, 56, [Link]],
"Third Score": [[Link], 40, 80, 98],
}

# creating a dataframe from list

df = [Link](dict)

# using isnull() function

[Link]()

Graph:

Conclusion:
Slicing and dicing enable efficient data extraction for detailed insights, while split and merge
operations allow manipulation of different segments of data. These operations enhance data
analysis, helping to isolate relevant portions and combine data for a comprehensive understanding.

Suggested Reference:
1. "Python for Data Analysis" by Wes McKinney
2. "Python Data Science Handbook" by Jake VanderPlas
3. "Pandas User Guide" on the Pandas documentation website
4. "Data Wrangling with Pandas" course on DataCamp
5. "Data Manipulation with Pandas" course on Coursera
Jhv v

21
Python for Data Science (3150713) 240183107018

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

22
Python for Data Science (3150713) 240183107018

Experiment No: 4
Develop a program that shows usage of aggregate function over the input
dataset. a) describe b) max c) min d) mean e) median f) count g) std h) Corr

Date:
Competency and Practical Skills:

Competency skills:

 Knowledge of the input dataset format (e.g. CSV, Excel, JSON) and how to load it into a
data structure in Python using libraries like Pandas.
 Understanding of the different aggregate functions available in Pandas, such as describe,
max, min, mean, median, count, std, and corr.
 Familiarity with the syntax of Pandas functions for applying aggregate functions, such as
groupby, apply, and agg.
 Ability to interpret and analyze the results of the aggregate functions to gain insights about
the dataset.

Practical skills:

 Loading the input dataset into a Pandas DataFrame object.

 Applying the desired aggregate functions to the DataFrame using the appropriate syntax.
 Displaying the results of the aggregate functions in a user-friendly format, such as a table
or chart.
 Handling any errors or exceptions that may arise during the data manipulation process.

Relevant CO: CO1, CO2

Objectives: (a) To understand the concept of aggregate functions and their usage in data
analysis.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
In data analysis, aggregate functions are used to calculate summary statistics over a dataset. These
functions are applied to columns or rows of a dataset to calculate values like the maximum,
minimum, mean, median, count, standard deviation, and correlation.

Here is a brief overview of the aggregate functions:

a) describe: This function generates descriptive statistics that summarize the central tendency,
dispersion, and shape of a dataset's distribution.

b) max: This function is used to find the maximum value of a column or row.

c) min: This function is used to find the minimum value of a column or row.

d) mean: This function is used to find the average value of a column or row.

e) median: This function is used to find the median value of a column or row.

Jhv v

23
Python for Data Science (3150713) 240183107018

f) std: This function is used to calculate the standard deviation of a column or row.

g) Corr: This function is used to calculate the correlation between columns or rows of a dataset.

In Python, these aggregate functions can be applied using the Pandas library. The groupby()
function is used to group data based on a specified column, and the aggregate functions can then
be applied to the grouped data.

Safety and necessary Precautions:

1. Make sure that the input dataset is clean and well-formatted.

2. Check the data types of the columns in the dataset.
3. Be careful when working with large datasets, as some aggregate functions may require a
lot of computational power and memory.
4. Double-check the output of the aggregate functions to ensure that they make sense and
match the expected results.

Procedure:
1. Import necessary libraries: You will need to import Pandas library to load the dataset and
perform various operations on it.
2. Load the dataset: Load the dataset in a Pandas dataframe using the read_csv() function.
Make sure the dataset is in a CSV format and is saved in your working directory.
3. Check the dataset: Print the first few rows of the dataset using the head() function to check
if the dataset is loaded correctly.
4. Describe the dataset: Use the describe() function to get the summary statistics of the
dataset, such as count, mean, standard deviation, minimum, and maximum values.
5. Apply aggregate functions: Apply the aggregate functions such as max(), min(), mean(),
median(), count(), std(), and corr() on the dataset.
6. Display the results: Display the results of the aggregate functions to the user.

Observation:
# import necessary libraries
import pandas as pd

# load the dataset using pandas

df = pd.read_csv("[Link]")

# print first few rows of the dataset

print([Link](10))

# descibe the dataset

print([Link]())

# apply aggregation functions on the dataset

print("\nDisplay the results of aggregation functions")
print("Maximum", df["Marks"].max())
print("Minimum", df["Marks"].min())
print("Average", df["Marks"].mean())
print("Mediam", df["Marks"].median())
print("Standard devaition", df["Marks"].std())
print("Number of non-null values", df["Marks"].count())
Jhv v

24
Python for Data Science (3150713) 240183107018
print("Correlation", df["Marks"].corr(df["Age"]))

Results:

Conclusion:
Aggregate functions are powerful tools for statistical analysis of datasets. They provide a quick
and comprehensive view of the data's central tendencies, spread, and relationships, helping in
understanding key patterns and making data-driven decisions.

Suggested Reference:
1. [Link]
2. [Link]

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

25
Python for Data Science (3150713) 240183107018

Experiment No: 5

Develop a program that shows the various data cleaning tasks over the dataset.
a) Identifying the null values. b) Identifying the empty values c) Identifying the
incorrect timestamp
Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces (GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Basic understanding of Python programming language

 Familiarity with data cleaning techniques, including identifying null and empty values,
handling incorrect timestamps, and removing outliers.
 Knowledge of statistical methods and data visualization techniques to identify anomalies
and outliers in the data.
 Familiarity with data cleaning libraries and tools, such as Pandas and NumPy in Python
 Problem-solving skills

Relevant CO: CO1, CO2, CO3

Objectives: (a) To identify and handle missing or incomplete data in the dataset.
(b) To identify and handle invalid or incorrect data in the dataset.
(c) To remove duplicate data in the dataset.
(d) To standardize data formats and values to ensure consistency across the dataset.
(e) To handle outliers and extreme values that may skew data analysis results.
(f) To ensure data accuracy and completeness for reliable data analysis.
(g) To improve data quality by reducing errors and inconsistencies in the dataset.
(h) To prepare the dataset for further analysis and modeling..

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
Data cleaning is an essential step in the data preparation process that involves identifying and
handling missing, incorrect, or inconsistent data in the dataset. In Python, data cleaning is
typically performed using libraries such as NumPy and Pandas, which provide functions for data
manipulation and analysis.

The theory behind data cleaning in Python involves several key steps:

Importing data: The first step in data cleaning is to import the data into Python using the
appropriate library and data format. Common data formats include CSV, Excel, and JSON.

Identifying missing data: Once the data is imported, the next step is to identify missing data in the
dataset. This can be done using the isnull() function in Pandas, which returns a Boolean value
indicating whether a value is missing or not.
Jhv v

26
Python for Data Science (3150713) 240183107018

Handling missing data: Once missing data is identified, the next step is to handle it appropriately.

This can be done by either removing the rows or columns with missing values or imputing the
missing values with a suitable value such as the mean or median of the column.

Identifying incorrect data: After handling missing data, the next step is to identify incorrect data in
the dataset, such as values that are outside the expected range or format. This can be done using
statistical techniques such as data visualization and analysis.

Handling incorrect data: Once incorrect data is identified, the next step is to handle it
appropriately. This can be done by removing the outliers or replacing the incorrect values with a
suitable value such as the median or mode of the column.

Standardizing data formats and values: To ensure consistency across the dataset, it is often
necessary to standardize data formats and values. This can be done by converting data types,
renaming columns, or applying formatting rules.

Removing duplicates: Duplicate data can skew analysis results and should be removed from the
dataset. This can be done using the drop_duplicates() function in Pandas.

Quality control: The final step in data cleaning is to perform quality control checks to ensure that
the data is accurate, complete, and consistent. This involves comparing the cleaned dataset to the
original dataset and verifying that the data has been cleaned appropriately.

Safety and necessary Precautions:

1. Backup data.
2. Use secure and updated software.
3. Access control.
4. Data privacy.
5. Data encryption
6. Error handling.
7. Test and validate.

Procedure:
1. Import the required libraries: Import the necessary libraries such as pandas, numpy, and
matplotlib to read, manipulate and visualize the dataset.

2. Load the dataset: Load the dataset into the program using a pandas dataframe.

3. Identify null values: Use the isnull() function to identify null values in the dataset. If any
null values are found, decide on a strategy to handle them. This could involve replacing
null values with a mean or median value, dropping the null values or imputing them with a
different value.

4. Identify empty values: Use the empty() function to identify empty values in the dataset.
Empty values are those values that contain nothing (not even null). If any empty values are
found, decide on a strategy to handle them. This could involve replacing empty values
with a mean or median value, dropping the empty values or imputing them with a different
value.

5. Identify incorrect timestamp: Use the to_datetime() function to convert the timestamp
column to a datetime object. This will identify any incorrect timestamp values. If any
Jhv v

27
Python for Data Science (3150713) 240183107018
incorrect timestamp values are found, decide on a strategy to handle them. This could
involve dropping the rows with incorrect timestamp values or imputing them with a
different value.

6. Remove duplicates: Use the drop_duplicates() function to remove any duplicate rows in
the dataset.

7. Data normalization: Use the normalization technique to transform the data into a standard
format to make it more consistent and easier to analyze.

8. Data standardization: Use the standardization technique to transform the data into a
standard scale to make it more consistent and easier to analyze.

9. Save the cleaned dataset: Save the cleaned dataset to a new file for future use.

10. Visualize the cleaned dataset: Use matplotlib or other visualization libraries to create
visualizations of the cleaned dataset to better understand the data and identify any further
cleaning that may be required.

Observation:
# Step 1: Import the required libraries
import pandas as pd
import numpy as np
import [Link] as plt
from [Link] import MinMaxScaler, StandardScaler

# Step 2: Load the dataset

df = pd.read_csv('sample_data.csv')

# Step 3: Identify and handle null values

print("Null values in each column:\n", [Link]().sum())

# Option 1: Replace null values with the column mean for numeric columns only
numeric_cols = df.select_dtypes(include=[[Link]]).columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].mean())

# Step 4: Identify and handle empty values

print("Empty values in each column:\n", (df == "").sum())

# Option 2: Replace empty values with the column median

# Replace empty values with NaN for easier handling
[Link]("", [Link], inplace=True)

# Fill NaN values with the median for numeric columns only
numeric_cols = df.select_dtypes(include=[[Link]]).columns
df[numeric_cols] = df[numeric_cols].fillna(df[numeric_cols].median())

# Step 5: Identify incorrect timestamps and handle them

if "timestamp_column" in [Link]: # Change 'timestamp_column' to the actual timestamp
column name
df["timestamp_column"] = pd.to_datetime(df["timestamp_column"], errors="coerce") #
Convert to datetime
incorrect_timestamps = df["timestamp_column"].isnull().sum() # Count incorrect timestamps
print(f"Number of incorrect timestamps: {incorrect_timestamps}")
[Link](subset=["timestamp_column"], inplace=True) # Option to drop rows with incorrect
Jhv v

28
Python for Data Science (3150713) 240183107018
timestamps

# Step 6: Remove duplicates

df.drop_duplicates(inplace=True)
print("Data after removing duplicates:", [Link])

# Step 7: Data normalization (MinMax scaling)

scaler = MinMaxScaler()
if (
"numeric_column" in [Link]
): # Change 'numeric_column' to a relevant numeric column
df[["numeric_column"]] = scaler.fit_transform(df[["numeric_column"]])

# Step 8: Data standardization (Standard scaling)

std_scaler = StandardScaler()
if (
"numeric_column" in [Link]
): # Change 'numeric_column' to a relevant numeric column
df[["numeric_column"]] = std_scaler.fit_transform(df[["numeric_column"]])

# Step 9: Save the cleaned dataset

df.to_csv("cleaned_dataset.csv", index=False)
print("Cleaned dataset saved as 'cleaned_dataset.csv'.")

# Step 10: Visualize the cleaned dataset

# Example: Visualizing the distribution of a numeric column
if "numeric_column" in [Link]: # Change 'numeric_column' to a relevant column
[Link](df["numeric_column"], bins=20, color="skyblue")
[Link]("Distribution of Numeric Column")
[Link]("Value")
[Link]("Frequency")
[Link]()

# Example: Visualizing a scatter plot of two numeric columns

if (
"numeric_column1" in [Link] and "numeric_column2" in [Link]
): # Change 'numeric_column1' and 'numeric_column2'
[Link](df["numeric_column1"], df["numeric_column2"], color="blue")
[Link]("Scatter Plot of Numeric Columns")
[Link]("Column 1")
[Link]("Column 2")
[Link]()

Sample_data.csv
id,name,age,salary,timestamp_column
1,Alice,25,50000,2023-01-10
2,Bob,,60000,2023-02-15
3,Charlie,30,,not_a_date
4,,28,55000,2023-03-01
Jhv v

29
Python for Data Science (3150713) 240183107018
5,Eve,25,50000,2023-01-10
6,Alice,25,50000,2023-01-10
7,Frank,,,""
8,Gina,32,65000,2023-04-05

Output:
Null values in each column:
id 0
name 1
age 2
salary 2
timestamp_column 0
dtype: int64

Empty values in each column:

id 0
name 0
age 0
salary 0
timestamp_column 1
dtype: int64

Number of incorrect timestamps: 2

Data after removing duplicates: (6, 5)

Cleaned dataset saved as 'cleaned_dataset.csv'.

id name age salary timestamp_column

0 1 Alice -0.825573 0.000000 2023-01-10
1 2 Bob 0.165115 0.666667 2023-02-15
3 4 NaN 0.363252 0.333333 2023-03-01
4 5 Eve -0.825573 0.000000 2023-01-10
5 6 Alice -0.825573 0.000000 2023-01-10

Jhv v

30
Python for Data Science (3150713) 240183107018

Conclusion:
Data cleaning is an essential step in data analysis to ensure data quality and accuracy. Identifying and
addressing missing values, empty values, and incorrect timestamps are crucial for reliable analysis and
decision-making.

Suggested Reference:
1. Data Cleaning with Python" course on DataCamp.
2. "Data Cleaning in Python: A Complete Guide" on Towards Data Science.
3. "Data Cleaning with Python and Pandas: Detecting Missing Values" on Real Python.
4. "Cleaning Data with Python" on Kaggle.
5. "Data Cleaning Techniques in Python" on Analytics Vidhya

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

31
Python for Data Science (3150713) 240183107018
Experiment No: 6
Develop a program that shows usage of following NumPy array operations: a)
any() b) all() c) isnan() d) isinf() e) isfinite() f) isinf() g) zeros() h) isreal()
i) iscomplex() j) isscalar() k) less() l) greater() m) less_equal() n)
greater_equal() and vector functions: a) arrange() b) reshape() c) linspace() d)
randint() e) dot()
Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces (GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Understanding of Data Structures

 Knowledge of Programming Languages
 Familiarity with Data Manipulation Libraries
 Proficiency in Using IDEs and Text Editors
 Problem Solving and Troubleshooting Skills

Relevant CO: CO2

Objectives:
a) To perform complex mathematical and logical operations on large arrays and matrices
efficiently.
b) To provide efficient and powerful tools for working with large arrays and matrices in
Python, along with a wide range of mathematical and scientific functions for manipulating
and analyzing these arrays.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
NumPy is a popular Python library for scientific computing that provides efficient and powerful
array operations. It enables users to work with multidimensional arrays and perform a variety of
mathematical and logical operations on them.

Here are the explanations of some of the NumPy array operations mentioned in the question:

a) any(): It returns True if any of the elements of an array evaluate to True, and False otherwise.

b) all(): It returns True if all the elements of an array evaluate to True, and False otherwise.

c) isnan(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is NaN (Not a Number), and False elsewhere.

d) isinf(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is +/-inf (positive or negative infinity), and False

Jhv v

32
Python for Data Science (3150713) 240183107018

e) isfinite(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is finite (i.e., not NaN, +/-inf), and False elsewhere.

f) isinf(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is +/-inf (positive or negative infinity), and False
elsewhere.

g) zeros(): It returns a new array of the specified shape and data type, filled with zeros.

h) isreal(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is real, and False where it is complex.

i) iscomplex(): It returns an array of the same shape as the input array, with True where the
corresponding element of the input array is complex, and False where it is real.

j) isscalar(): It returns True if the input is a scalar (i.e., a single value, not an array), and False
otherwise.

k) less(): It returns an array of the same shape as the input arrays, with True where the
corresponding element of the first input array is less than the corresponding element of the second
input array, and False otherwise.

l) greater(): It returns an array of the same shape as the input arrays, with True where the
corresponding element of the first input array is greater than the corresponding element of the
second input array, and False otherwise.

m) less_equal(): It returns an array of the same shape as the input arrays, with True where the
corresponding element of the first input array is less than or equal to the corresponding element of
the second input array, and False otherwise.

n) greater_equal(): It returns an array of the same shape as the input arrays, with True where the
corresponding element of the first input array is greater than or equal to the corresponding
element of the second input array, and False otherwise.

Here is a brief theory for each of the NumPy vector functions:

a) arrange(): This function is used to create a one-dimensional array with evenly spaced values
within a specified range. The function takes in three arguments: start (optional), stop, and step
(optional). The start argument is the starting value of the sequence (inclusive), the stop argument
is the ending value of the sequence (exclusive), and the step argument is the step size between
values. For example, [Link](0, 10, 2) creates an array with values [0, 2, 4, 6, 8].

b) reshape(): This function is used to reshape an array into a new shape without changing its data.
The function takes in one argument: the new shape of the array, specified as a tuple of integers.
For example, [Link](my_array, (3, 4)) reshapes the array my_array into a 3x4 matrix.

c) linspace(): This function is used to create a one-dimensional array with evenly spaced values
between a specified range. The function takes in three arguments: start, stop, and num (optional).
The start argument is the starting value of the sequence, the stop argument is the ending value of
the sequence, and the num argument is the number of values to generate. For example,
[Link](0, 1, 5) creates an array with values [0., 0.25, 0.5, 0.75, 1.].

d) randint(): This function is used to generate an array of random integers within a specified range.
Jhv v

33
Python for Data Science (3150713) 240183107018
The function takes in three arguments: low (optional), high, and size (optional). The low argument
is the lower bound of the range (inclusive), the high argument is the upper bound of the range
(exclusive), and the size argument is the shape of the output array. For example,
[Link](0, 10, size=(2, 3)) generates a 2x3 array of random integers between 0 and 10.

e) dot(): This function is used to perform matrix multiplication between two arrays. The function
takes in two arguments: the two arrays to be multiplied. The arrays must have compatible shapes
for matrix multiplication. For example, if A is a 2x3 array and B is a 3x2 array, [Link](A, B)
performs matrix multiplication between A and B and returns a 2x2 array.

Overall, these NumPy vector functions are commonly used for manipulating and analyzing arrays
in scientific computing and data analysis. By using these functions in a program, you can
efficiently perform operations on large arrays and matrices in Python.

Safety and necessary Precautions:

1. Make sure to import NumPy correctly.
2. Use appropriate data types.
3. Watch out for NaN and Inf.
4. Be careful with memory usage.
5. Test the program thoroughly.

Procedure:
1. Import the NumPy library: To use NumPy array operations, you need to import the
NumPy library into your Python environment. You can do this using the import statement.

2. Create a NumPy array: You need to create a NumPy array to perform the various
operations. You can create an array using the [Link]() function.

3. Use the array operations: Once you have created the array, you can use various NumPy
array operations such as any(), all(), isnan(), isinf(), isfinite(), zeros(), isreal(), iscomplex(),
isscalar(), less(), greater(), less_equal(), and greater_equal().

4. Import the NumPy library: Begin your program by importing the NumPy library using the
import statement.

5. Create an array: Create an array using one of the NumPy functions such as arrange() or
linspace(). You can also create an array from an existing data source such as a CSV file.

6. Reshape the array: Use the reshape() function to reshape the array to the desired shape. For
example, you can reshape a one-dimensional array into a two-dimensional array.

7. Generate random numbers: Use the randint() function to generate an array of random
integers within a specified range.

8. Perform matrix multiplication: Use the dot() function to perform matrix multiplication
between two arrays.

9. Print the results: Print the resulting arrays to the console using the print() function.
Observation:
import numpy as np

# create a numpy array

array = [Link]([1, 2, 3, 4, 5], dtype=int)

Jhv v

34
Python for Data Science (3150713) 240183107018

# numpy array operations

print([Link]()) # True
print([Link]()) # True
print([Link](array).any()) # False
print([Link](array).any()) # False
print([Link](array).all()) # False
print([Link](array)) # [0 0 0 0 0 0]
print([Link](array).all()) # True
print([Link](array).all()) # False
print([Link](array)) # False
print([Link](array, 3)) # [ True True False False False]
print([Link](array, 3)) # [False False False True True]
print(np.less_equal(array, 3)) # [ True True True False False]
print(np.greater_equal(array, 3)) # [False False True True True]

# create an array using arrange() or linspace()

array1 = [Link](1, 5, 1) # Adjusted to have 4 elements

# reshape the array

array1 = [Link](2, 2)

# use randint() to create an array of random integers

array2 = [Link](1, 10, 4).reshape(2, 2) # Reshape to (2, 2)

# perform matrix multiplication

result = [Link](array1, array2)
print(result)

Conclusion:
NumPy’s array operations and vector functions are essential tools for working with numerical
data in Python. They enable efficient data manipulation, analysis, and computation, making
NumPy a cornerstone of scientific computing and data science.

Jhv v

35
Python for Data Science (3150713) 240183107018
Suggested Reference:
1. NumPy User Guide: [Link]
2. NumPy Tutorial: [Link]
3. NumPy Cheat Sheet:
[Link]
pdf
4. NumPy Array Operations: [Link]
python/
5. NumPy Array Operations and Functions:
[Link]

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical and
Practical Problem Task Documentation
Professional
Understanding Solving Execution and Reporting
Rubrics Conduct Total
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

36
Python for Data Science (3150713) 240183107018
Experiment No: 7

Write python script using matplotlib library for following:

a) To display line plot for given values of X: [1,2,3,...,49] and values of Y
(thrice of X): [3,6,9,12,...,144,147]
b) To display bar plot for given value Languages = ['Java', 'Python', 'PHP',
'JavaScript', 'C#', 'C++'] and popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7]
c) To display pie plot for given value Languages = ['Java', 'Python', 'PHP',
'JavaScript', 'C#', 'C++'] and popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7]
using colors = ["#1f77b4", "#ff7f0e", "#2ca02c", "#d62728", "#9467bd",
"#8c564b"]
d) To display scatter plot for 200 random points for both X and Y axis.

Date:
Competency and Practical Skills:

Competency skills:

 Understanding the basics of data visualization

 Familiarity with Python programming language
 Knowledge of the different types of plots and when to use them
 Knowledge of the syntax and parameters for different matplotlib functions
 Understanding of data structures like arrays and data frames

Practical skills:

 Ability to create different types of plots such as line plots, scatter plots, bar plots, etc.
 Ability to customize the appearance of plots including labels, colors, legends, and titles
 Ability to add text, annotations, and shapes to the plots
 Ability to work with multiple plots and subplots
 Ability to export plots in different file formats like png, pdf, svg, etc.
 Ability to integrate matplotlib with other Python libraries like NumPy and Pandas.

Relevant CO: CO4

Objectives:
a) To create informative and visually appealing data visualizations that enable users to
explore, understand, and communicate complex data.
b) To learn how to interpret and analyze data visualizations, and to use them to draw insights
and make informed decisions.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
Matplotlib is a Python library that provides a variety of tools for creating high-quality data
visualizations. It is one of the most popular data visualization libraries due to its ease of use and
versatility. The library is built on NumPy and provides a range of options for creating different
types of plots and graphs, including line plots, scatter plots, bar charts, histograms, and many
more.

The main components of the Matplotlib library are:

pyplot module: This is the main module of Matplotlib, which provides a simple interface for
Jhv v

37
Python for Data Science (3150713) 240183107018
creating plots and charts. It is a collection of functions that allow users to create plots with
minimal coding.

Figure and Axes objects: The Figure object is the top-level container for all the plot elements. It
represents the entire plot and contains one or more Axes objects. The Axes object is the individual
plot area where data is plotted.

Plotting functions: Matplotlib provides a range of plotting functions that can be used to create
different types of plots and charts. These functions include plot(), scatter(), bar(), hist(), and many
more.

Customization options: Matplotlib allows users to customize the appearance of plots in various
ways, including changing the plot color, adding labels, titles, and legends, adjusting the axis limits,
and more.

To use Matplotlib, you first need to import the library and its pyplot module. Then, you can create
a figure object and one or more axes objects using the subplots() function. After that, you can use
the various plotting functions to create different types of plots and customize them as needed.

Use the bar() function to create the bar plot by passing the languages and popularity lists as
arguments. The bar() function automatically generates the rectangular bars for each category and
sets their lengths proportional to the values in the popularity list.

A scatter plot is useful for exploring the relationship between two continuous variables. It can be
used to identify patterns or trends in the data and to detect the presence of outliers or unusual
observations. Scatter plots can also be used to assess the correlation between the two variables.

Matplotlib provides the scatter() function for creating scatter plots. The function takes two arrays,
one for the X-axis data and one for the Y-axis data, as its input arguments. Additional parameters
can be used to customize the appearance of the scatter plot, such as the color, size, and
transparency of the points.

Overall, Matplotlib provides a powerful and flexible tool for creating data visualizations in Python.
With its wide range of options and customization features, it can be used for a variety of data
analysis and communication tasks.

Safety and necessary Precautions:

1. Keep Matplotlib libraries up-to-date.
2. Use Comments
3. Test your code.

Procedure:
(for sub practical A or B):

1. Import the required libraries - Matplotlib and NumPy.

2. Create two NumPy arrays for X and Y values using [Link]() and multiplication.
3. Create a figure and an axis object using [Link]().
4. Use a [Link]() function to plot X and Y values as a line plot.
5. Use the bar() function to create the plot, passing the data as arguments
6. Customize the plot with axis labels and a title.
7. Display the plot using [Link]() function.
Jhv v

38
Python for Data Science (3150713) 240183107018
(for sub practical C):
1. Import the necessary libraries ([Link])
2. Define the data to be used (Languages, Popularity, Colors)
3. Create a figure object and set the figure size
4. Define the title of the plot and add the data to be displayed (Popularity) and their
corresponding labels (Languages)
5. Set the colors of the pie chart using the Colors list
6. Add a legend to the chart with the labels and colors used
7. Display the plot.

(for sub practical D):

1. Import necessary libraries: We will need the Matplotlib and NumPy libraries for this task.
2. Generate random data for the X and Y axes: We can use the NumPy library to generate
random data for both the X and Y axes
3. Create a scatter plot: We can use the scatter method of the Matplotlib library to create a
scatter plot. We need to pass the X and Y data as arguments and specify the marker style
and color using the marker and c parameters, respectively
4. Add title and labels: We can add a title and labels for the X and Y axes using the title,
xlabel, and ylabel methods of the Matplotlib library.
5. Set axes limits: We can set the limits for the X and Y axes using the xlim and ylim
methods of the Matplotlib library.
6. Display the plot: We can display the plot using the show method of the Matplotlib library.

Observation:
import [Link] as plt
import numpy as np
import random

# Part A
X = [Link](1, 51, 1)
Y = [Link](X, 3)
[Link](X, Y, "b+")
[Link]("X values")
[Link]("Y values")
[Link]("Plot of X-values vs Y-values")
[Link]()

# Part B
languages = ["Java", "Python", "PHP", "JavaScript", "C#", "C++"]
popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7]
[Link](languages, popularity)
[Link]("Languages")
[Link]("Popularity")
[Link]("Popularity of Programming Languages")
[Link]()

# Part C
languages = ["Java", "Python", "PHP", "JavaScript", "C#", "C++"]
popularity = [22.2, 17.6, 8.8, 8, 7.7, 6.7]
colors = ["red", "blue", "green", "yellow", "purple", "orange"]
[Link](popularity, labels=languages, colors=colors, autopct="%1.1f%%")
[Link]("Popularity of Programming Languages")

Jhv v

39
Python for Data Science (3150713) 240183107018
[Link](languages, loc="best")
[Link]()

# Part D
X = [[Link](1, 201) for x in range(200)]
Y = [[Link](1, 201) for x in range(200)]
[Link](X, Y)
[Link]("Random X-values")
[Link]("Random Y-values")
[Link](0, 200)
[Link](0, 200)
[Link]("Random X-values vs Random Y-values")
[Link]()

Plots:

Jhv v

40
Python for Data Science (3150713) 240183107018

Conclusion:
Matplotlib is a powerful library for data visualization in Python. It offers a wide range of plotting
functions and customization options to effectively communicate insights from data

Sugested Reference:
1. [Link]
2. [Link]
3. Matplotlib Tutorial by Corey Schafer: [Link]
osiE80TeTvipOqomVEeZ1HRrcEvtZB_
4. Python Data Science Handbook by Jake VanderPlas:
[Link]
5. Mastering Matplotlib by Duncan M. McGreggor and Paul Ivanov:
[Link]

Jhv v

41
 Python for Data Science (3150713) 240183107018

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical
Practical Proble Task Documentation
an
Understanding m Execution and Reporting
Rubrics d Professional Total
Solving
Conduct
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)
Marks

Jhv v

42
 Python for Data Science (3150713) 240183107018

Experiment No : 8
Develop a program that reads .csv file from
the url:
([Link]
[Link]?raw=true) and plot the data of the dataset stored in
the .csv file.

Date:
Competency and Practical Skills:

Competency skills:
 Data analysis, data visualization, file handling and programming.

Practical skills:

 Reading data from a CSV file using Python's pandas library.

 Pre-processing and cleaning data as required.
 Plotting data using the matplotlib library.
 Dealing with missing values or null values in the data (if any).
 Analyzing and interpreting the data visualizations to draw insights
and make conclusions.

Relevant CO: CO3,

CO4 Objectives:
(a) To analyze and visualize the data in an efficient and effective way.
(b) To identify patterns, trends, and outliers in the data.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
Reading a .csv file from a URL and plotting the data is a common data analysis and
visualization task in many fields. Here are the main steps involved in this process:

Importing the necessary libraries: To read and plot the .csv file, we typically use
the pandas and matplotlib libraries. We need to import them at the beginning of
our program.

Loading the data from the URL: We can use the pandas library's read_csv
function to read the data from the URL. We need to provide the URL of the .csv file
as an argument to this function.

Data cleaning and preparation: Once we have loaded the data, we may need to clean
and prepare it for visualization. This may include dropping unnecessary columns,
filling missing values, and transforming the data.

Data visualization: Once the data is cleaned and prepared, we can use matplotlib's

Jhv v

43
 Python for Data Science (3150713) 240183107018

various plotting functions to create visualizations such as line plots, scatter plots, bar
plots, and more. We can customize the plot with various parameters such as colors,
labels, titles, and more.

Displaying the plot: After creating the plot, we need to display it using the show
function provided by the matplotlib library.

Safety and necessary Precautions:

1. Validate inputs.
2. Handle errors.
3. Secure the program
4. Optimize performance
5. Test and review.

Procedure:
1. Import the necessary libraries: You will need the pandas library to read the
.csv file, and matplotlib library to create the plot.
2. Read the .csv file from the URL: Use the pandas library to read the .csv file
from the URL and store it as a DataFrame object.
3. Preprocess the data: Preprocess the data as required. This may involve
cleaning the data, removing duplicates, handling missing values, and
converting data types.
4. Visualize the data: Use the matplotlib library to create a visualization of the
data. You can create scatter plots, line graphs, histograms, and other types of
visualizations based on the data.
5. Save or display the visualization: Save the visualization to a file or display it
on the screen, depending on the user requirements.
6. Test and validate the program: Test the program thoroughly to ensure that it
works as expected for various input datasets. Validate the results against the
expected output and fix any issues or errors.
7. Document the program: Document the program by providing clear and
concise comments in the code and a user manual that explains how to use the
program.

Observation:
import numpy as np
import pandas as pd

# Read the data

sales_data = pd.read_excel( "[Link]
[Link]?raw=True",
)
# Preprocess the data
sales_data["date"] = pd.to_datetime(sales_data["date"])

# visualize the data [Link](figsize=(10, 6))

[Link](sales_data["date"], sales_data["ext price"]) [Link]("Sales data")
[Link]("Date") [Link]("Sales") [Link]()

# save the plot [Link]("sales_data.png")

Jhv v

44
 Python for Data Science (3150713) 240183107018

Conclusion:
Combining Pandas and Matplotlib allows for efficient data retrieval and visualization.
By reading data directly from a URL and plotting it, we can quickly gain insights into
trends and patterns in the dataset.

Suggested Reference:
1. Pandas documentation on reading a CSV file from a URL:
[Link]
docs/stable/user_guide/[Link]#reading-csv-files
2. Matplotlib documentation on creating plots:
[Link]
/[Link]
3. Real Python tutorial on reading and writing CSV
files in Python: [Link]
4. DataCamp tutorial on data visualization with Matplotlib:
[Link]
tutorial-python
5. Towards Data Science tutorial on creating visualizations with Pandas
and Matplotlib: [Link]
pandas-and-matplotlib- 8dadc69f2f79

Jhv v

45
 Python for Data Science (3150713) 240183107018

References used by the students:

[Link]
Rubric wise marks obtained:

Ethical and
Practical Proble Task Documentation
Professional
Understanding m Execution and Reporting
Rubrics Conduct Total
Solving
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)
Marks

Jhv v

46
 Python for Data Science (3150713) 240183107018

Experiment No: 9
Write a text classification pipeline using a custom preprocessor and
CharNGram Analyzer using data from Wikipedia articles as a training set.
a) Evaluate the performance on some held out test sets
Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces
(GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Proficiency in Python programming language.

 Familiarity with the scikit-learn library for machine learning.
 Knowledge of text preprocessing techniques, such as tokenization, stop
word removal, stemming, and lemmatization.
 Understanding of feature extraction techniques, such as bag-of-words and
character n- grams.
 Ability to evaluate the performance of a text classification model using
metrics such as accuracy, precision, recall, and F1 score.
 Knowledge of cross-validation techniques for evaluating model performance on
held-out
test sets.
 Familiarity with data collection and preprocessing techniques for building a
training set from Wikipedia articles.

Relevant CO: CO3, CO4, CO5

Objectives: (a) To develop a machine learning model that can accurately

classify text documents into predefined categories that can be used for various
applications such as sentiment analysis, spam detection, and topic modeling.

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
Text classification is the task of assigning predefined categories or labels to text
documents based on their content. A text classification pipeline typically consists of
several stages, including data preprocessing, feature extraction, model training, and
evaluation.

In the context of Wikipedia articles, the first step in building a text classification
pipeline is to collect a dataset of articles with their corresponding labels. These
labels can be either manually assigned or obtained from existing metadata such as

Jhv v

47
 Python for Data Science (3150713) 240183107018

categories or tags.

Once a dataset is obtained, the next step is data preprocessing. This typically
involves text normalization, tokenization, stop word removal, and
stemming/lemmatization. The goal of data preprocessing is to clean the text and
reduce its dimensionality while retaining the relevant information for classification.

After preprocessing, the text is converted into numerical features that can be used as
input to a machine learning model. A popular technique for feature extraction is the
bag-of-words model, which represents each document as a vector of word
frequencies. However, this approach may not capture the semantic meaning of words
and their relationships in the text.

An alternative approach is to use character n-grams, such as CharNGramAnalyzer,

which captures the sequence of characters in the text. This method is particularly
useful for capturing the morphology and syntax of the text and can improve the
performance of the classifier.

The final stage in the text classification pipeline is model training and evaluation. A
common approach is to use supervised learning algorithms such as Naive Bayes,
Logistic Regression, or Support Vector Machines. The performance of the model is
evaluated using metrics such as accuracy, precision, recall, and F1 score on held-out
test sets.

In summary, building a text classification pipeline using a custom preprocessor and

CharNGramAnalyzer involves data preprocessing, feature extraction, model training,
and evaluation. This approach can be particularly useful for text classification tasks
where the meaning and relationships of words are important.

Safety and necessary Precautions:

1. Data privacy.
2. Bias and fairness.
3. Model accuracy and reliability
4. Ethical considerations
5. Test and review.

Procedure:
Collect and preprocess the data: Download a set of Wikipedia articles that represent
the different categories you want to classify (e.g., sports, politics, entertainment,
etc.). Preprocess the data by removing any unnecessary characters, converting all
text to lowercase, and removing any stop words.

Split the data: Split the preprocessed data into two sets: training and test sets. The
training set will be used to train the model, while the test set will be used to evaluate
the model's performance.

Feature extraction: Extract the features from the preprocessed text using

Jhv v

48
 Python for Data Science (3150713) 240183107018

CharNGramAnalyzer. This will convert each text document into a vector of features
that can be used as input to the classification model.

Train the model: Train a text classification model using the extracted features and
the training set. You can use any machine learning algorithm, such as Naive Bayes,
SVM, or Neural Networks.

Evaluate the model: Use the trained model to classify the test set and evaluate its
performance using metrics such as accuracy, precision, recall, and F1-score.

Tune the model: If the model's performance is not satisfactory, you can tune the
hyperparameters of the algorithm or try different algorithms to improve its
performance.

Deploy the model: Once you are satisfied with the model's performance, you can
deploy it in production to classify new text documents.

Observation:
# CharNGramAnalyzer was removed from python in 2012. Instead use CountVectorizer.

import numpy as np
from sklearn.feature_extraction.text import CharNGramAnalyzer
from [Link] import Pipeline
from sklearn.naive_bayes import MultinomialNB
from [Link] import fetch_20newsgroups
from sklearn.model_selection import train_test_split
from [Link] import accuracy_score

def custom_preprocessor(text):
text = [Link]() # Convert to lowercase
text = [Link]("'s", "") # Remove possessives
text = [Link]("[^a-zA-Z]", " ") # Remove non-alphabetical characters return text

# Load data
categories = ["[Link]", "[Link]"]
dataset = fetch_20newsgroups(subset="train", categories=categories) # Create the
pipeline
analyzer = CharNGramAnalyzer(min_n=2, max_n=4) classifier = MultinomialNB()
pipeline = Pipeline( [("preprocessor", custom_preprocessor), ("analyzer",
analyzer),("classifier", classifier),])

# Split data into training and test sets X = [Link]

y = [Link]
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42)

# Train the pipeline on the training data [Link](X_train, y_train)

# Evaluate performance on held-out test sets predictions = [Link](X_test)
accuracy = accuracy_score(y_test, predictions) print(f"Accuracy on test set: {accuracy}")

Jhv v

49
 Python for Data Science (3150713) 240183107018

Conclusion:
Text classification pipelines with custom preprocessors and CharNGramAnalyzer can
effectively classify text data based on character-level patterns. Evaluating performance
on held out test sets ensures the model'sgeneralizability to unseen data.

Suggested Reference:
1. "Building a Text Classification Pipeline with Python" by Dipanjan Sarkar:
This article provides a step-by-step guide on how to build a text classification
pipeline using Python and scikit-learn library. It covers preprocessing
techniques, feature extraction, model selection, and evaluation.

2. "Text Classification with NLTK and Scikit-Learn" by Ahmed Besbes: This

tutorial provides a detailed guide on how to perform text classification using
Python and two popular libraries, NLTK and scikit-learn. It covers data
preprocessing, feature extraction, and model training and evaluation.

3. "Using Wikipedia Articles for Text Classification" by Nikolay Krylov: This

article demonstrates how to use Wikipedia articles as a training set for text
classification. It covers data collection, preprocessing, feature extraction
using TF-IDF and CharNGramAnalyzer, model training, and evaluation.

4. "Text Classification with Python and Scikit-Learn" by Sebastian Raschka:

This book chapter provides a comprehensive guide on how to perform text
classification using Python and scikit-learn. It covers data preprocessing,
feature extraction, model training, and evaluation, as well as advanced topics
such as model selection and parameter tuning.

5. "A Complete Tutorial on Text Classification using Naive Bayes Algorithm"

Jhv v

50
 Python for Data Science (3150713) 240183107018

by Divya Gupta: This tutorial provides a detailed guide on how to perform

text classification using Naive Bayes algorithm in Python. It covers data
preprocessing, feature extraction, model training and evaluation, as well as
parameter tuning

References used by the students:

[Link]

Rubric wise marks obtained:

Ethical
Practical Proble Task Documentation
an
Understanding m Execution and Reporting
Rubrics d Professional Total
Solving
Conduct
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)
Marks

Jhv v

51
 Python for Data Science (3150713) 240183107018

Experiment No: 10

Write a text classification pipeline to classify movie reviews as

either positive or negative.
a) Find a good set of parameters using grid search.
b) Evaluate the performance on a held out test set.

Date:
Competency and Practical Skills:

Competency skills:

 Basic knowledge of computer systems, operating systems, and file systems.

 Familiarity with command-line interfaces (CLI) and graphical user interfaces
(GUI).
 Understanding of programming languages, syntax, and logic.

Practical skills:

 Strong understanding of Natural Language Processing (NLP)

concepts, such as tokenization, stemming, lemmatization, and feature
extraction.
 Familiarity with popular NLP libraries such as NLTK, SpaCy, and Scikit-learn.
 Knowledge of machine learning algorithms, such as Naive Bayes,
Support Vector Machines, and Neural Networks.
 Ability to preprocess text data, including removing stop words,
cleaning text, and performing feature engineering.
 Experience with data exploration and visualization tools, such as Pandas and
Matplotlib.
 Familiarity with Python programming language and its data science
ecosystem, including NumPy, SciPy, and Pandas.
 Ability to evaluate the performance of a classification model using
appropriate metrics such as accuracy, precision, recall, and F1-score.
 Knowledge of different hyperparameter tuning techniques and cross-
validation methods to optimize the model's performance.

Relevant CO: CO3, CO4, CO5

Objectives: (a) To create an accurate and reliable model that can automatically
classify movie reviews as positive or negative, which can be useful for analyzing
large volumes of reviews quickly and efficiently, as well as for providing
recommendations to users based on their preferences..

Equipment/Instruments: Personal Computer, Internet, Python

Theory:
The theory behind writing a text classification pipeline to classify movie reviews as
either positive or negative involves several key steps:

Jhv v

52
 Python for Data Science (3150713) 240183107018

Data preprocessing: This step involves cleaning and preparing the raw text data by
removing stop words, converting text to lowercase, and performing stemming or
lemmatization.

Feature extraction: This step involves converting the preprocessed text data into a
numerical representation that can be used as input to a machine learning algorithm.
Common techniques include Bag-of-Words, TF-IDF, and Word Embeddings.

Model selection and training: This step involves selecting an appropriate machine
learning algorithm and training it on the preprocessed and transformed data. Popular
algorithms include Naive Bayes, Support Vector Machines, and Neural Networks.

Hyperparameter tuning: This step involves selecting the optimal hyperparameters for
the chosen machine learning algorithm. This can be done using techniques such as
grid search or random search.

Evaluation: This step involves evaluating the performance of the trained model on a
held-out test set. This can be done using metrics such as accuracy, precision, recall,
and F1-score.

Deployment: This step involves deploying the trained model in a production

environment, where it can be used to classify new movie reviews.

Grid search is a hyperparameter tuning technique that involves searching for the
optimal set of hyperparameters for a given machine learning algorithm by
exhaustively trying all possible combinations of hyperparameter values. This can be
done by training and evaluating the model with different combinations of
hyperparameters on a validation set, and selecting the combination that yields the
best performance.

Evaluating the performance of the trained model on a held-out test set is important to
ensure that the model generalizes well to new, unseen data. This helps to avoid
overfitting, where the model performs well on the training data but poorly on new
data.

Overall, the theory behind writing a text classification pipeline to classify movie
reviews as either positive or negative involves a combination of data preprocessing,
feature extraction, model selection and training, hyperparameter tuning, evaluation,
and deployment.

Safety and necessary Precautions:

1. Data preprocessing
2. Feature extraction
3. Model selection
4. Hyper parameter tuning
5. evaluation

Jhv v

53
 Python for Data Science (3150713) 240183107018

Procedure:
1. Preprocess the data: Preprocess the movie review data by cleaning the text,
removing stop words, and performing stemming or lemmatization to reduce
the dimensionality of the feature space.

2. Split the data: Split the preprocessed data into training, validation, and test
sets. The training set will be used to train the model, the validation set will be
used to tune the hyperparameters, and the test set will be used to evaluate the
final performance of the model.

3. Extract features: Extract features from the preprocessed text using techniques
such as Bag- of-Words, TF-IDF, or Word Embeddings. This will convert the
text data into a numerical representation that can be used as input to a
machine learning algorithm.
Select a model: Choose a suitable machine learning algorithm, such
as Naive Bayes, Support Vector Machines, or Neural Networks, and
train it on the preprocessed and transformed data.

4. Hyperparameter tuning: Use grid search to find the best set of

hyperparameters for the chosen machine learning algorithm. This involves
training and evaluating the model with different combinations of
hyperparameters on the validation set, and selecting the combination that
yields the best performance.

5. Evaluate the model: Evaluate the performance of the trained model on the
held-out test set using metrics such as accuracy, precision, recall, and F1-
score.

6. Deploy the model: Deploy the trained model in a production environment,

where it can be used to classify new movie reviews.
Observation:
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from [Link] import SVC
from [Link] import Pipeline
from sklearn.model_selection import GridSearchCV
from [Link] import fetch_20newsgroups
from sklearn.model_selection import train_test_split
from [Link] import classification_report

# Load data
categories = ["[Link]", "[Link]"]
dataset = fetch_20newsgroups(subset="train", categories=categories)

# Split data into training and test sets

X = [Link] y = [Link]
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42)

# Define the pipeline

pipeline = Pipeline([("tfidf", TfidfVectorizer()), ("clf", SVC())])

# Define parameters for grid search

parameters = {"tfidf ngram_range": [(1, 1), (1, 2)],
"tfidf min_df": [2, 3],

Jhv v

54
 Python for Data Science (3150713) 240183107018

"clf kernel": ["linear", "rbf"], "clf C": [1, 10]}

# Perform grid search

grid_search = GridSearchCV(pipeline, parameters, cv=5)
grid_search.fit(X_train, y_train)

# Evaluate performance on test set

predictions = grid_search.predict(X_test)
print(classification_report(y_test, predictions))

Conclusion:
Text classification pipelines with grid search optimization can effectively classify movie
reviews based on sentiment. By systematically tuning parameters, we can improve the
model's performance and ensure its generalizability to unseen data.

Suggested Reference:
1. "Introduction to Machine Learning with Python" by Andreas C. Müller and
Sarah Guido - This book provides a comprehensive introduction to machine
learning and includes a section on text classification. It covers topics such as
preprocessing text data, feature extraction, and model evaluation.

2. "Natural Language Processing with Python" by Steven Bird, Ewan Klein,

and Edward Loper - This book provides an introduction to natural language
processing and includes a section on text classification. It covers topics such
as feature selection, training classifiers, and evaluation metrics.

Jhv v

55
 Python for Data Science (3150713) 240183107018

3. "Scikit-learn documentation" - Scikit-learn is a popular machine learning

library in Python. The documentation includes a section on text classification,
which covers preprocessing text data, feature extraction, and model selection.
It also provides examples of how to use grid search to find the best set of
hyperparameters for a model.

4. "Text Classification in Python using spaCy" by Dipanjan Sarkar - This

tutorial provides an introduction to text classification using spaCy, a popular
NLP library in Python. It covers topics such as preprocessing text data,
feature extraction, model selection, and hyperparameter tuning.

"Sentiment Analysis on Movie Reviews" Kaggle competition - This Kaggle

competition provides a dataset of movie reviews labeled as positive or negative. It
includes notebooks from participants that demonstrate how to preprocess the data,
extract features, and train models. It also provides examples of how to use
grid search to find the best set of hyperparameters for a model.

References used by the students:

[Link]
[Link]

Rubric wise marks obtained:

Ethical
Practical Proble Task Documentation an
Understanding m Execution and Reporting d
Rubrics Total
Solvin Professional
g Conduct
Good Avg. Good Avg. Good Avg. Good Avg. Good Avg.
(2) (1) (2) (1) (2) (1) (2) (1) (2) (1)

Marks

Jhv v

56