PYTHON FOR ARTIFICIAL

INTELLIGENCE
MODULE IV
 Introduction to Pandas

 Pandas is a Python library used for data manipulation and

analysis. Pandas provides a convenient way to analyze and clean
data.

 The Pandas library introduces two new data structures to Python -

Series and DataFrame, both of which are built on top of NumPy.

 Pandas is a powerful library generally used for:

 Data Cleaning
 Data Transformation
 Data Analysis
 Machine Learning
 Data Visualization
Import Pandas in Python

 We can import Pandas in Python using the import statement.

import pandas as pd
 The code above imports the pandas library into our program with
the alias pd.

 After this import statement, we can use Pandas functions and

objects by calling them with pd.
Pandas Series

 A Pandas Series is a one-dimensional labeled array-like object that

can hold data of any type.

 A Pandas Series can be thought of as a column in a spreadsheet or a

single column of a DataFrame. It consists of two main components:
the labels and the data.

 For example,

• 0 'John'
• 1 30
• 2 6.2
• 3 False
• dtype: object
 Here, the series has two columns, labels (0, 1, 2 and 3) and data
('John', 30, 6.2, False).

 The labels are the index values assigned to each data point, while
the data represents the actual values stored in the Series.
  Pandas Series can store elements of different data types. It uses a
concept called dtype (data type) to manage and represent the
underlying data in a Series.

 By default, Pandas internally represents the Series of different

datatypes using the object data type, a general-purpose data type
that can hold any text or numeric data type(strings, integers,
Createbooleans,
a Pandasetc).
Series

 There are multiple ways to create a Pandas Series, but the most
common way is by using a Python list. Let's see an example of
creating a Series using a list:
import pandas as pd

# create a list
data = [10, 20, 30, 40, 50]

# create a series from the list

my_series = [Link](data)

print(my_series)
 In this example, we created a Python list called data containing five
integer values. We then passed this list to the Series() function,
which converted it into a Pandas Series called my_series.

 Here, dtype: int64 denotes that the series stores the values of int64
Labels
types.

 The labels in the Pandas Series are index numbers by default. Like in
dataframe and array, the index number in series starts from 0.

 Such labels can be used to access a specified value. For example,

import pandas as pd

# create a list
data = [10, 20, 30, 40, 50]

# create a series from the list

my_series = [Link](data)

# display third value in the series

print(my_series[2])
 Here, we accessed the third element of my_series using a label.

 We can also specify labels while creating the series using the index
argument in the Series() method. For example,

import pandas as pd

# create a list
a = [1, 3, 5]

# create a series and specify labels

my_series = [Link](a, index =
["x", "y", "z"])

print(my_series)
 In this example,we passed index = ["x", "y", "z"] as an argument
to Series() to specify the labels explicitly.

 To access the series elements, we use the specified labels instead

of the default index number. For example,
eate Series From a Python Dictionary

import pandas as pd

# create a dictionary
grades = {"Semester1": 3.25, "Semester2": 3.28,
"Semester3": 3.75}

# create a series from the dictionary

my_series = [Link](grades)

# display the series

print(my_series)

Output

Semester1 3.25
Semester2 3.28
dtype: float64
Pandas DataFrame

A DataFrame is like a table where the data is organized in rows and

columns. It is a two-dimensional data structure like a two-dimensional
array. For example,
Country Capital Population
0 Canada Ottawa 37742154
1 Australia Canberra 25499884
2 UK London 67886011
3 Brazil Brasília 212559417
 Here, Country, Capital and Population are the column names.

 Each row represents a record, with the index value on the left. The
index values are auto-assigned starting from 0.

 Each column contains data of the same type. For instance, Country
and Capital contain strings, and Population contains integers.

 The DataFrame is similar to a table in a SQL database, or a

spreadsheet in Excel. It is designed to manage ordered and
unordered datasets in Python.
Create a Pandas DataFrame

 We can create a Pandas DataFrame in the following ways:

i. Using Python Dictionary

ii. Using Python List
iii. From a File
iv. Creating an Empty DataFrame

andas DataFrame Using Python Dictionary

import pandas as pd

# create a dictionary
data = {'Name': ['John', 'Alice', 'Bob'],
'Age': [25, 30, 35],
'City': ['New York', 'London', 'Paris']}

# create a dataframe from the dictionary

df = [Link](data)

print(df)
 Output

Name Age City

0 John 25 New York
1 Alice 30 London
2 Bob 35 Paris

 In this example, we created a dictionary called data that contains

the column names (Name, Age, City) as keys, and lists of values
as their respective values.

 We then used the [Link]() function to convert the

dictionary into a DataFrame called df.
i. Pandas DataFrame Using Python List

import pandas as pd

# create a two-dimensional list

data = [['John', 25, 'New York'],
['Alice', 30, 'London'],
['Bob', 35, 'Paris']]

# create a DataFrame from the list

df = [Link](data, columns=['Name', 'Age',
'City'])

print(df)
 Output

Name Age City

0 John 25 New York
1 Alice 30 London
2 Bob 35 Paris

 In this example, we created a two-dimensional list called data

containing nested lists.

 The DataFrame() function converts the 2-D list to a DataFrame. Each

nested list behaves like a row of data in the DataFrame.

 The columns argument provides a name to each column of the

DataFrame.
i. Pandas DataFrame From a File

 Another common way to create a DataFrame is by loading data

from a CSV (Comma-Separated Values) file. For example,

import pandas as pd

# load data from a CSV file

df = pd.read_csv('[Link]')

print(df)

 In this example, we used the read_csv() function which reads the

CSV file [Link], and automatically creates a DataFrame object df,
containing data from the CSV file.

 We can also create a DataFrame using other file types like JSON,
Excel spreadsheet, SQL database, etc. The methods to read
different file types are listed below:

 JSON - read_json()
 Excel spreadsheet - read_excel()
iv. Create an Empty DataFrame

Sometimes we may want to create an empty DataFrame and then add

data later. For example,
Output
import pandas as pd
Empty DataFrame
# create an empty DataFrame
Columns: []
df = [Link]()
Index: []
print(df)

 In this example, we have created an empty DataFrame by

calling [Link]() without any arguments.

 Here, both the Columns and Index lists are empty in the
[Link] DataFrame has no data, but it can be used as a
container to store and manipulate data later.
Pandas Index

 In Pandas, an index refers to the labeled array that identifies

rows or columns in a DataFrame or a Series. For example,

Name Age City

0 John 25 New York
1 Alice 28 London
2 Bob 32 Paris

 In the above DataFrame, the numbers 0, 1, and 2 represent the

index, providing unique labels to each row.

 We can use indexes to uniquely identify data and access data

with efficiency and precision.
Types of Pandas Indexes
Create Indexes in Pandas

 Pandas offers several ways to create indexes. Some common

methods are as follows:
 Default Index
 Setting Index
 Creating a Range Index

Default Index

 When we create a DataFrame or Series without specifying an

index explicitly, Pandas assigns a default integer index starting
from 0. For example,
 import pandas as pd Output

data = {'Name': ['John', 'Alice', 'Bob'], Name Age City

'Age': [25, 28, 32], 0 John 25 New York
'City': ['New York', 'London', 'Paris']}1 Alice 28 London
2 Bob 32 Paris
df = [Link](data)
print(df)

 In this example, the default index [0, 1, 2] is automatically assigned

to the rows.
Setting Index

 We can set an existing column as the index using the set_index()

method. For example,
import pandas as pd
Output
# create dataframe
data = {'Name': ['John', 'Alice', 'Bob'], Name Age City
'Age': [25, 28, 32], John 25 New York
'City': ['New York', 'London', 'Paris']} Alice 28 London
Bob 32 Paris
df = [Link](data)

# set the 'Name' column as index

df.set_index('Name', inplace=True)

print(df)
  In this example, the Name column is set as the index, replacing
the default integer index.

 Here, the inplace=True parameter performs the operation directly

on the object itself, without creating a new object. When we
specify inplace=True, the original object is modified, and the
Creatingchanges
a Rangeare directly applied.
Index

 We can create a range index with specific start and end values
using the RangeIndex() function. For example,
import pandas as pd

# create dataframe Output

data = {'Name': ['John', 'Alice', 'Bob'],
'Age': [25, 28, 32], Name Age City
'City': ['New York', 'London', Index
'Paris']} 5 John 25 New York
6 Alice 28 London
df = [Link](data) 7 Bob 32 Paris

# create a range index

df = [Link](data,
index=[Link](5, 8,
name='Index'))

print(df)
 Here, a range index from 5 to 8(excluded) is created with the
name Index.
Modifying Indexes in Pandas

 Pandas allows us to make changes to indexes easily. Some

common modification operations are:
 Renaming Index
 Resetting Index
 import pandas as pd

# create a dataframe
data = {'Name': ['John', 'Alice', 'Bob'], Output
'Age': [25, 28, 32],
'City': ['New York', 'London', 'Paris']} Original DataFrame:
df = [Link](data) Name Age City
0 John 25 New York
# display original dataframe 1 Alice 28 London
print('Original DataFrame:') 2 Bob 32 Paris
print(df)
print() Modified DataFrame
Name Age City
# rename index A John 25 New York
[Link](index={0: 'A', 1: 'B', 2: 'C'}, B Alice 28 London
inplace=True) C Bob 32 Paris

# display dataframe after index is

renamed
print('Modified DataFrame')
print(df)
In this example, we renamed the indexes 0, 1, and 2 to 'A', 'B',
and 'C' respectively.
Resetting Index
import pandas as pd

data = {'Name': ['John', 'Alice', 'Bob'],

'Age': [25, 28, 32],
'City': ['New York', 'London', 'Paris']} Output

# create a dataframe Original DataFrame:

df = [Link](data) Name Age City
A John 25 New York
# rename index B Alice 28 London
C Bob 32
[Link](index={0: 'A', 1: 'B', 2: 'C'}, inplace=True) Paris

# display dataframe Modified DataFrame:

print('Original DataFrame:') index Name Age
print(df) City
print('\n') 0 A John 25 New
York
# reset index 1 B Alice 28 London
df.reset_index(inplace=True) 2 C Bob 32 Paris

# display dataframe after index is reset

print('Modified DataFrame:')
print(df)
Pandas DataFrame Analysis

 Pandas DataFrame objects come with a variety of built-in

functions like head(), tail() and info() that allow us to view and
analyze DataFrames.
View Data in a Pandas DataFrame

 A Pandas Dataframe can be displayed as any other Python

variable using the print() function.

 However, when dealing with very large DataFrames with large

numbers of rows and columns, the print() function is unable to
display the whole DataFrame. Instead, it prints only a part of the
DataFrame.

 In the case of large DataFrames, we can use head(), tail() and

info() methods to get the overview of the DataFrame.
 Pandas head()

 The head() method provides a rapid summary of a DataFrame.

It returns the column headers and a specified number of rows
from the beginning. For example,
import pandas as pd

# create a dataframe
data = {'Name': ['John', 'Alice', 'Bob', 'Emma', 'Mike', 'Sarah', 'David',
'Linda', 'Tom', 'Emily'],
'Age': [25, 30, 35, 28, 32, 27, 40, 33, 29, 31],
'City': ['New York', 'Paris', 'London', 'Sydney', 'Tokyo', 'Berlin',
'Rome', 'Madrid', 'Toronto', 'Moscow']}
df = [Link](data)

# display the first three rows

print('First Three Rows:')
print([Link](3))
print()

# display the first five rows

print('First Five Rows:')
print([Link]())
 Output

First Three Rows:

Name Age City
0 John 25 New York
1 Alice 30 Paris
2 Bob 35 London

First Five Rows:

Name Age City
0 John 25 New York
1 Alice 30 Paris
2 Bob 35 London
3 Emma 28 Sydney
4 Mike 32 Tokyo

 In this example, we displayed selected rows of the df DataFrame

starting from the top using head().

 Notice that the first five rows are selected by default when no
argument is passed to the head() method.
 Pandas tail()

 The tail() method is similar to head() but it returns data starting

from the end of the DataFrame. For example
import pandas as pd

# create a dataframe
data = {'Name': ['John', 'Alice', 'Bob', 'Emma', 'Mike', 'Sarah',
'David', 'Linda', 'Tom', 'Emily'],
'Age': [25, 30, 35, 28, 32, 27, 40, 33, 29, 31],
'City': ['New York', 'Paris', 'London', 'Sydney', 'Tokyo', 'Berlin',
'Rome', 'Madrid', 'Toronto', 'Moscow']}

df = [Link](data)

# display the last three rows

print('Last Three Rows:')
print([Link](3))
print()

# display the last five rows

print('Last Five Rows:')
print([Link]())
Output

Last Three Rows:

Name Age City
7 Linda 33 Madrid
8 Tom 29 Toronto
9 Emily 31 Moscow

Last Five Rows:

Name Age City
5 Sarah 27 Berlin
6 David 40 Rome
7 Linda 33 Madrid
8 Tom 29 Toronto
9 Emily 31 Moscow

 In this example, we displayed selected rows of the df DataFrame

starting from the bottom using tail().

 Notice that the last five rows are selected by default when no
argument is passed to the tail() method.
Get DataFrame Information

 The info() method gives us the overall information about the

DataFrame such as its class, data type, size etc. For example,

import pandas as pd

# create dataframe
data = {'Name': ['John', 'Alice', 'Bob', 'Emma', 'Mike', 'Sarah', 'David',
'Linda', 'Tom', 'Emily'],
'Age': [25, 30, 35, 28, 32, 27, 40, 33, 29, 31],
'City': ['New York', 'Paris', 'London', 'Sydney', 'Tokyo', 'Berlin',
'Rome', 'Madrid', 'Toronto', 'Moscow']}
df = [Link](data)

# get info about dataframe

[Link]()
Output

<class
'[Link]'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Name 10 non-null object
1 Age 10 non-null int64
2 City 10 non-null object
dtypes: int64(1), object(2)
memory usage: 372.0+ bytes
 As you can see, the info() method provides the following
information about a Pandas DataFrame:

 Class: The class of the object, which indicates that it is a pandas

DataFrame
 RangeIndex: The index range of the DataFrame, showing the
starting and ending index values
 Data columns: The total number of columns in the DataFrame
 Column names: The names of the columns in the DataFrame
 Non-Null Count: The count of non-null values for each column
 Dtype: The data types of the columns
 Memory usage: The memory usage of the DataFrame in bytes
andas Aggregate Function

 Aggregate function in Pandas performs summary computations on

data, often on grouped data. But it can also be used on Series
objects.

 This can be really useful for tasks such as calculating mean, sum,
count, and other statistics for different groups within our data.
 Syntax

[Link](func, axis=0, *args, **kwargs)

 func - an aggregate function like sum, mean, etc.

 axis - specifies whether to apply the aggregation operation
along rows or columns.
 *args and **kwargs - additional arguments that can be
passed to the aggregation functions.
pply Single Aggregate Function

import pandas as pd

data = {
'Category': ['A', 'A', 'B', 'B', 'A', 'B'],
'Value': [10, 15, 20, 25, 30, 35]
Output
}
Total Sum: 135
df = [Link](data)
Average Value: 22.5
Maximum Value: 35
# calculate total sum of the Value
column
total_sum =
df['Value'].aggregate('sum')
print("Total Sum:", total_sum)

# calculate the mean of the Value

column
average_value =
df['Value'].aggregate('mean')
print("Average Value:", average_value)
 df['Value'].aggregate('sum') - calculates the total sum of the
Value column in the data DataFrame

 df['Value'].aggregate('mean') - calculates the mean (average) of

the Value column in the data DataFrame

 df['Value'].aggregate('max') - computes the maximum value in

the Value column.
pply Multiple Aggregate Functions in Pandas

import pandas as pd

data = {
'Category': ['A', 'A', 'B', 'B', 'A', 'B'],
'Value': [10, 15, 20, 25, 30, 35]
}

df = [Link](data)

# applying multiple aggregation functions to a single column

result = [Link]('Category')['Value'].agg(['sum', 'mean',
'max', 'min'])
print(result)

Output

sum mean max min

Category
A 55 18.333333 30 10
B 80 26.666667 35 20
 In the above example, we're using the aggregate() function to apply
multiple aggregation functions (sum, mean, max, and min) to the
Value column after grouping by the Category column.

 The resulting DataFrame shows the calculated values for each

category.
ply Different Aggregation Functions
Output
import pandas as pd
Value1 Value2
data = { sum mean max
'Category': ['A', 'A', 'B', 'B', 'A', 'B'], Category
'Value1': [10, 15, 20, 25, 30, 35], A 55 17.00 18
'Value2': [5, 8, 12, 15, 18, 21] B 80 16.00 21
}

df = [Link](data)

agg_funcs = {

# applying 'sum' to Value1 column

'Value1': 'sum',

# applying 'mean' and 'max' to Value2 column

'Value2': ['mean', 'max']
}

result = [Link]('Category').aggregate(agg_funcs)
print(result)
  Here, we're using the aggregate() function to apply different
aggregation functions to different columns after grouping by the
Category column.

 The resulting DataFrame shows the calculated values for each

category and each specified aggregation function.
Pandas groupby

 In Pandas, the groupby operation lets us group data based on

specific columns. This means we can divide a DataFrame into
smaller groups based on the values in these columns.

 Once grouped, we can then apply functions to each group

separately. These functions help summarize or aggregate the data
in each group.
roup by a Single Column in Pandas

 In Pandas, we use the groupby() function to group data by a

single column and then calculate the aggregates. For example,

import pandas as pd Output

# create a dictionary containing the data Category
data = {'Category': ['Electronics', 'Clothing', 'Electronics',
Clothing 800
'Clothing'], Electronics 1800
'Sales': [1000, 500, 800, 300]} Name: Sales, dtype:
# create a DataFrame using the data dictionary
df = [Link](data)

# group the DataFrame by the Category column and

# calculate the sum of Sales for each category
grouped = [Link]('Category')['Sales'].sum()

# print the grouped data

print(grouped)
 In the above example, [Link]('Category')['Sales'].sum() is used
to group by a single column and calculate sum.

 This line does the following:

 [Link]('Category') - groups the df DataFrame by the unique

values in the Category column.

 ['Sales'] - specifies that we are interested in the Sales column within

each group.

 sum() - calculates the sum of the Sales values for each group.
Pandas Data Cleaning

 Data cleaning means fixing and organizing messy data. Pandas

offers a wide range of tools and functions to help us clean and
preprocess our data effectively.
 Data cleaning often involves:

 Dropping irrelevant columns.

 Renaming column names to meaningful names.
 Making data values consistent.
 Replacing or filling in missing values.
Drop Rows With Missing Values

 In Pandas, we can drop rows with missing values using the

dropna() function
import pandas as pd

# define a dictionary with sample data which includes some missing

values
data = {
'A': [1, 2, 3, None, 5],
'B': [None, 2, 3, 4, 5],
'C': [1, 2, None, None, 5]
}

df = [Link](data)
print("Original Data:\n",df)
print()

# use dropna() to remove rows with any missing values

df_cleaned = [Link]()

print("Cleaned Data:\n",df_cleaned)
Output

Original Data:
A B C
0 1.0 NaN 1.0
1 2.0 2.0 2.0
2 3.0 3.0 NaN
3 NaN 4.0 NaN
4 5.0 5.0 5.0

Cleaned Data:
A B C
1 2.0 2.0 2.0
4 5.0 5.0 5.0

 Here, we have used the dropna() method to remove rows with any
missing values. The resulting DataFrame df_cleaned will only
contain rows without any missing values
Fill Missing Values

 To fill the missing values in Pandas, we use the fillna() function.

For example,
import pandas as pd

# define a dictionary with sample data which includes some

missing values
data = {
'A': [1, 2, 3, None, 5],
'B': [None, 2, 3, 4, 5],
'C': [1, 2, None, None, 5]
}

df = [Link](data)

print("Original Data:\n", df)

# filling NaN values with 0

[Link](0, inplace=True)

print("\nData after filling NaN with 0:\n", df)

Output

Original Data:
A B C
0 1.0 NaN 1.0
1 2.0 2.0 2.0
2 3.0 3.0 NaN
3 NaN 4.0 NaN
4 5.0 5.0 5.0
Data after filling NaN with 0:
A B C
0 1.0 0.0 1.0
1 2.0 2.0 2.0
2 3.0 3.0 0.0
3 0.0 4.0 0.0
4 5.0 5.0 5.0

 Here, we used [Link]() to fill the missing values(NaN) in each

column with 0.

 Note: The inplace=True argument here means that the operation will
modify the DataFrame directly, rather than returning a new
e Aggregate Functions to Fill Missing Values

 Instead of filling with 0, we can also use aggregate functions to fill

missing values.

 Let's look at an example to fill missing values with the mean of each
import pandas as pd
column.
# define a dictionary with sample data which includes some
missing values
data = {
'A': [1, 2, 3, None, 5],
'B': [None, 2, 3, 4, 5],
'C': [1, 2, None, None, 5]
}

df = [Link](data)

print("Original Data:\n", df)

# filling NaN values with the mean of each column

[Link]([Link](), inplace=True)

print("\nData after filling NaN with mean:\n", df)

Output

Original Data:
A B C
0 1.0 NaN 1.0
1 2.0 2.0 2.0
2 3.0 3.0 NaN
3 NaN 4.0 NaN
4 5.0 5.0 5.0
Data after filling NaN with mean:
A B C
0 1.00 3.5 1.000000
1 2.00 2.0 2.000000
2 3.00 3.0 2.666667
3 2.75 4.0 2.666667
4 5.00 5.0 5.000000

Here, the [Link]() calculates the mean for each column, and the
fillna() method then replaces NaN values in each column with the
respective mean.
Handle Duplicates Values

 In Pandas, to handle duplicate rows, we can use the duplicated()

and the drop_duplicates() function.

 duplicated() - to check for duplicates

 drop_duplicates() - remove duplicate rows

import pandas as pd

# sample data
data = {
'A': [1, 2, 2, 3, 3, 4],
'B': [5, 6, 6, 7, 8, 8]
}
df = [Link](data)

print("Original DataFrame:\n", df.to_string(index=False))

# detect duplicates
print("\nDuplicate Rows:\n", df[[Link]()].to_string(index=False))

# remove duplicates based on column 'A'

df.drop_duplicates(subset=['A'], keep='first', inplace=True)

print("\nDataFrame after removing duplicates based on column 'A':\n",

df.to_string(index=False))
Original DataFrame:
A B
1 5
2 6
2 6
3 7
3 8
4 8
Duplicate Rows:
A B
2 6
DataFrame after removing duplicates
based on column 'A':
A B
1 5
2 6
3 7
4 8
Here,

df[[Link]()] produces a boolean Series to identify duplicate rows.

df.drop_duplicates(subset=['A'], keep='first', inplace=True), removes
duplicates based on column A, retaining only the first occurrence of each
duplicate directly in the original DataFrame.
ame Column Names to Meaningful Names
 To rename column names to more meaningful names in Pandas, we
can use the rename() function. For example,

import pandas as pd
Output
# sample data
data = { Age Name Salary
'A': [25, 30, 35], 25 John 50000
'B': ['John', 'Doe', 'Smith'], 30 Doe 60000
'C': [50000, 60000, 70000] 35 Smith 70000
}

df = [Link](data)

# rename columns
[Link](columns={'A': 'Age', 'B': 'Name', 'C': 'Salary'},
inplace=True)

print(df.to_string(index=False))
Here, the columns of df are renamed from A, B, and C to more
meaningful names Age, Name, and Salary respectively.