Digital Skills and Python Programming

Chapter 9
Essential Libraries for Data Analysis:
Introduction to Pandas

Pr. Mehdia AJANA

Email: [Link]@[Link]
 What is Pandas?
PYTHON • Pandas is a Python library used for working
Data Analysis with large datasets efficiently.
• Pandas handles structured tabular data like
CSV, Excel, SQL tables… and provides easy
data cleaning, analysis, and transformation.
• The name "Pandas" refers to both "Panel
Pandas Data", and "Python Data Analysis“
Introduction • Pandas provides two main easy-to-use data
structures for data manipulation and
analysis: Series and DataFrames, that make it
easy to store and work with data.
• Pandas consists of the above data structures
and functions to perform efficient operations
on data.
Pr. Mehdia AJANA 2
 Installing and Importing pandas:
PYTHON Installing pandas:
Data Analysis You can install pandas using Python Package
Installer in Pycharm:

Pandas
Introduction Importing pandas:
Once Pandas is installed, import it in your
applications by adding the import keyword and
using the pd alias:
import pandas as pd
Checking Pandas Version:
The version string is stored under __version__
attribute:
print(pd.__version__) 2.2.3
Pr. Mehdia AJANA 3
 Introducing Series:
PYTHON • A Pandas Series is a one-dimensional data
Data Analysis structure, like a column in a table, holding
data of any type.
Example:
• data is a list of values labeled with their index
number.
Pandas • By default the first value has index 0, second
Series value has index 1 etc.
• This index/label can be used to access a
specified value:
0 10
import pandas as pd 1 20
data = [10, 20, 30, 40] 2 30
series = [Link](data) 3 40
dtype: int64
print(series)
10
print(series[0])
Pr. Mehdia AJANA 4
 Introducing Series:
PYTHON Creating Labels:
Data Analysis • With the index argument, you can name
your own labels.
• Each value is labeled with an index, allowing
us to access data by labels instead of
positions:
import pandas as pd
Pandas
data = [10, 20, 30, 40]
Series
series = [Link](data, index=['a', 'b', 'c',
'd'])
a 10
print(series) b 20
print(series['b']) c 30
d 40
dtype: int64
20
Pr. Mehdia AJANA 5
 Introducing Series:
PYTHON Dictionaries as Series:
Data Analysis • You can also use a key/value object, like a
dictionary, when creating a Series.
• The keys of the dictionary become the labels:
-To select only some of the items in the
dictionary, use the index argument and specify
only the items you want to include in the
Pandas Series.
Series import pandas as pd
calories = {"day1": 420, "day2": 380, "day3": 390}
series = [Link](calories) day1 420
day2 380
print(series)
day3 390
dtype: int64
series = [Link](calories, index = ["day1", "day2"])
print(series) day1 420
day2 380
Pr. Mehdia AJANA dtype: int64 6
 Introducing Series:
PYTHON
Simple NumPy Array as Series:
Data Analysis
• You can also use a 1 D NumPy array when
creating a Series:
import pandas as pd
import numpy as np

Pandas
# simple array
Series
data = [Link](['g', 'e', 'e', 'k', 's'])
0 g
series = [Link](data) 1 e
print(series) 2 e
3 k
4 s
dtype: object

Pr. Mehdia AJANA 7

 Introducing DataFrames:
PYTHON • Series is like a column, a DataFrame is the
Data Analysis whole table.
• A Pandas DataFrame is a 2 dimensional data
structure, like a table with rows and
columns.
Example:
create a DataFrame from a dictionary where
Pandas
each key becomes a column in the table:
DataFrame import pandas as pd
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
'City': ['Paris', 'London', 'Madrid']}
df = [Link](data) Name Age City
0 Ali 25 Paris
print(df) 1 Ahmed 30 London
2 Iyad 28 Madrid
Pr. Mehdia AJANA 8
 Introducing DataFrames:
PYTHON Locate Row:
Data Analysis Pandas use the loc attribute to return one or
more specified row(s).
Example:
import pandas as pd
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
Pandas 'City': ['Paris', 'London', 'Madrid']}
DataFrame df = [Link](data) Name Ali
#Return first row Age 25
print([Link][0]) City Paris
Name: 0, dtype: object
#Return 2 rows: use a list of indexes:
print([Link][[0, 1]]) Name Age City
0 Ali 25 Paris
1 Ahmed 30 London

Pr. Mehdia AJANA 9

 Introducing DataFrames:
PYTHON Named Indexes:
Data Analysis With the index argument, you can name your
own indexes.
Example:
import pandas as pd
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
Pandas
'City': ['Paris', 'London', 'Madrid']}
DataFrame
df = [Link](data, index = ["student1",
"student2", "student3"])
print(df)

Name Age City

student1 Ali 25 Paris
student2 Ahmed 30 London
student3 Iyad 28 Madrid
Pr. Mehdia AJANA 10
 Introducing DataFrames:
PYTHON Locate Named Indexes:
Data Analysis Use the named index in the loc attribute to
return the specified row(s).
Example:
import pandas as pd
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
Pandas
'City': ['Paris', 'London', 'Madrid']}
DataFrame
df = [Link](data, index = ["student1",
"student2", "student3"])
#refer to the named index:
print([Link]["student2"])
Name Ahmed
Age 30
City London
Name: student2, dtype: object
Pr. Mehdia AJANA 11
 Introducing DataFrames:
PYTHON Display Column Names, Data Types and
Data Analysis number of Rows:
Example:
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
'City': ['Paris', 'London', 'Madrid']}
df = [Link](data)
Pandas # Display column names
DataFrame print([Link])
Index(['Name', 'Age', 'City'], dtype='object')

# Display data types Name object

Age int64
print([Link]) City object
dtype: object
# Get the number of rows and columns
(3, 3)
print([Link]) 12
Pr. Mehdia AJANA
 Introducing DataFrames:
PYTHON Retrieve a column:
Data Analysis data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
'City': ['Paris', 'London', 'Madrid']}
df = [Link](data)
# Retrieve the ‘Name’ column
Pandas print(df['Name']) 0 Ali
1 Ahmed
DataFrame 2 Iyad
Name: Name,
dtype: object

# Retrieve rows where Age > 25

print(df[df['Age'] > 25])
Name Age City
1 Ahmed 30 London
2 Iyad 28 Madrid
13
 Introducing DataFrames:
PYTHON Add and Drop Columns:
Data Analysis
# Add a new column 'Old'
df['Old'] = df['Age'] > 25
print(df)
Name Age City Old
0 Ali 25 Paris False
1 Ahmed 30 London True
Pandas 2 Iyad 28 Madrid True
DataFrame
# Drop column 'Age'
from the dataframe
[Link](columns=['Age'], inplace=True)
print(df)
Name City Old
0 Ali Paris False
1 Ahmed London True
2 Iyad Madrid True
14
 Introducing DataFrames:
PYTHON Save the updated dataframe to CSV file:
Data Analysis Example:
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
'City': ['Paris', 'London', 'Madrid']}
df = [Link](data)
# Save the updated DataFrame to a CSV file
Pandas df.to_csv('updated_data.csv', index=False)
DataFrame print("Data saved to 'updated_data.csv'")
index=False: exclude the row index from being
written to the CSV file.
Data saved to 'updated_data.csv'

Name,City,Old
Ali,Paris,False
Ahmed,London,True
Iyad,Madrid,True
Pr. Mehdia AJANA 15
 Introducing DataFrames:
PYTHON Calculate Sum, Average, Max, and Min Age:
Data Analysis Example:
data = {'Name': ['Ali', 'Ahmed', 'Iyad'],
'Age': [25, 30, 28],
'City': ['Paris', 'London', 'Madrid']}
df = [Link](data)
# Calculate the sum of age
Pandas sum_age = df['Age'].sum() Sum of age: 83
DataFrame print("Sum of age:", sum_age)
# Calculate the average age
average_age = df['Age'].mean()
print("Average age:", average_age)
# Find the max and min age Average age: 27.66
max_age = df['Age'].max()
min_age = df['Age'].min() Max age: 30
print("Max age:", max_age) Min age: 25
print("Min age:", min_age) Pr. Mehdia AJANA 16
 Reading Data from CSV Files:
PYTHON • A simple way to store big datasets is to use CSV
Data Analysis files (comma separated values).
• In the following examples we will be using a CSV
file called ‘[Link]’: containing 14 rows of
data with columns: Name, Age, City, Grade.
• Tasks:
• Read the CSV into a Pandas DataFrame.
• Handle duplicate and missing values.
Pandas • Manage invalid data (e.g., invalid age and
Read CSV grades).
Name, Age, City, Grade
John, 21, New York, 16.0
Kevin, 22, Los Angeles, 12.0
Smith,, San Francisco, 14.0 #Missing Age value
Sarah, 25, Chicago, #Missing Grade value
John, 21, New York, 16.0 # Duplicate
Tom, abc, Los Angeles, 15.0 # Invalid Age
Ahmed,NaN,London,16.2 #Missing Age value
Ali,25,Paris, #Missing Grade value
…………………………….. Pr. Mehdia AJANA 17
 Reading Data using pandas:
PYTHON • pd.read_csv() reads a CSV file into a
Data Analysis DataFrame, making it easy to analyze and
manipulate the data.
• Once loaded, df contains the data from the
file in a structured format, ready for
exploration and manipulation.
• The to_string() method is used to print the
Pandas entire DataFrame.
Read CSV Example:
import pandas as pd
# Reading the CSV file into a DataFrame
df = pd.read_csv('[Link]')
# Display the entire rows
print(df.to_string())

Pr. Mehdia AJANA 18

 Reading Data using pandas:
PYTHON Example:
Data Analysis

Pandas
Read CSV

Pr. Mehdia AJANA 19

 Reading Data from CSV Files:
PYTHON • The to_string() method is used to print the entire
Data Analysis DataFrame.
• If you have a large DataFrame with many rows,
Pandas will only return the first 5 rows, and the
last 5 rows if you use print(df).
• [Link](): shows the first few rows of the
DataFrame (default is 5).
Pandas Example:
Basic Data Get a quick overview by printing the header and the
Exploration first 7 rows of the DataFrame:
print([Link]())
print([Link](7))

Pr. Mehdia AJANA 20

 Reading Data from CSV Files:
PYTHON • tail() method: returns the headers and a specified
Data Analysis number of rows, starting from the bottom
(default is 5).
Example:
import pandas as pd
df = pd.read_csv('[Link]')
Pandas # Display the header and the last 5 rows of the
Basic Data DataFrame
Exploration print([Link]())
# Display the header and the last 3 rows
print([Link](3))

Pr. Mehdia AJANA 21

 Pandas - Analyzing DataFrames:
PYTHON • [Link](): Gives a summary of the DataFrame,
Data Analysis showing the data types and missing values.
Example:
import pandas as pd
df = pd.read_csv('[Link]')
print([Link]())
Pandas
Analyzing
DataFrames

Pr. Mehdia AJANA 22

 Pandas - Analyzing DataFrames:
PYTHON • [Link](): Provides basic statistical
Data Analysis information about numerical columns.
Example:
import pandas as pd
df = pd.read_csv('[Link]')
print([Link]())
Pandas
Analyzing
DataFrames

Pr. Mehdia AJANA 23

 Inspecting the Data for Missing or
PYTHON Invalid Values:
Data Analysis • Data cleaning means fixing bad data in your
data set. Bad data could be:
 Empty cells
 Data in wrong
format (invalid data)
 Duplicates
Pandas Example: Identify
missing values
Data Cleaning (empty cells/null values) for each column
import pandas as pd
df = pd.read_csv(
'[Link]')
print([Link]())

Pr. Mehdia AJANA 24

 Inspecting the Data for Missing or
PYTHON Invalid Values:
Data Analysis • Data cleaning means fixing bad data in your
data set. Bad data could be:
 Empty cells
 Data in wrong format (invalid data)
 Duplicates
Example: Identify missing values (empty
Pandas cells/null values) for each column
Data Cleaning import pandas as pd
df = pd.read_csv('[Link]')
print([Link]().sum())

Explanation: isnull() checks for NaN values

(missing values) in the DataFrame, and sum()
calculates the total missing values per column.
Pr. Mehdia AJANA 25
 Cleaning Missing Values:
PYTHON To clean missing values, we can:
Data Analysis 1. Drop rows with missing values
2. Fill missing values
1. Drop / Remove Rows with Missing Values:
• dropna(): removes rows with any missing
values, which is a common data-cleaning step:
Pandas df = pd.read_csv('[Link]')
Data Cleaning #remove all the rows with NaN or empty
values
newdf = [Link]()
print(newdf)
# By default, the dropna() method returns
a new DataFrame, and will not change the
original. To change the original DataFrame:
[Link](inplcae=True)
Pr. Mehdia AJANA 26
 Cleaning Missing Values:
PYTHON 1. Drop / Remove Rows with Missing Values:
Data Analysis • dropna(): removes rows with any missing
values, which is a common data-cleaning step:

Pandas
Data Cleaning

Pr. Mehdia AJANA 27

 Cleaning Missing Values:
PYTHON 2. Fill Missing Values:
Data Analysis • fillna(): fills missing values with a specified
value.
Example: we fill missing ages with 0 and
missing grades with the average grade:
# Fill missing Age with 0
Pandas df['Age'].fillna(0, inplace=True)
Data Cleaning
# Fill missing Grade with the average
df['Grade'].fillna(df['Grade'].mean(),
inplace=True)
print(df)
We use the inplace = True argument to change
the original DataFrame.
Pr. Mehdia AJANA 28
 Removing Duplicate Data:
PYTHON 1. Identify Duplicates:
Data Analysis • duplicated(): returns a Boolean series,
showing True for duplicate rows:
import pandas as pd
df = pd.read_csv('[Link]')
print([Link]())
Pandas 0 False
Data Cleaning 1 False
2 False
3 False
4 True
5 False
.........
dtype: bool

In the students dataset the row for John is

duplicated.
Pr. Mehdia AJANA 29
 Removing Duplicate Data:
PYTHON 2. Remove Duplicates:
Data Analysis • drop_duplicates(): removes duplicate rows
from the DataFrame:
import pandas as pd
df = pd.read_csv('[Link]')
df=df.drop_duplicates()
Pandas print(df)
Data Cleaning
- In the students dataset the duplicate row for
John (with index=4) will be removed.
- We can reorder or reset the index after
dropping duplicate rows, by using the
reset_index() method with the drop=True
argument:
df = df.reset_index(drop=True)
Resets the index to start from 0 and drops the old index.
Pr. Mehdia AJANA 30
 Cleaning Invalid Values:
PYTHON Data of Wrong Format:
Data Analysis Cells with data of wrong format can make it difficult,
or even impossible, to analyze data.
In the students data set we have an invalid age in
the Age column for Tom:
Tom,abc,Seattle,15.0
To fix it, you have two options: convert it to
Pandas numeric value, or remove the row:
Data Cleaning Example: Convert Age column to numeric, invalid
values become NaN:
# Convert Age to numeric, coercing errors to NaN
df['Age'] = pd.to_numeric(df['Age'], errors='coerce')
# Fill missing Age values with 0 (after conversion)
df['Age'].fillna(0, inplace=True)
print(df)

Pr. Mehdia AJANA 31

 Cleaning Invalid Values:
PYTHON Data of Wrong Format:
Data Analysis To drop the row with invalid data in the Age
column after converting it to numeric values,
you can use the dropna() function to remove
rows with NaN values:
Tom,abc,Seattle,15.0

Pandas # Convert Age to numeric, coercing errors to NaN

Data Cleaning df['Age'] = pd.to_numeric(df['Age'], errors='coerce')

# Drop rows where Age is NaN (invalid data)

[Link](subset=['Age'], inplace=True)

This removes the rows where the Age column has

NaN values.
Pr. Mehdia AJANA 32
 Basic Data Analysis - Filtering:
PYTHON Filtering allows you to select rows based on
Data Analysis conditions.
Filtering with one condition:
Example: Filter students who are older than 25.
import pandas as pd
df = pd.read_csv('[Link]')
Pandas print(df)
Data Analysis # Filter students with Age greater than 25
filtered_data = df[df['Age'] > 25]
# Display the filtered data
print(filtered_data)
Name Age City Grade
8 Iyad 28 Rabat 12.5
12 Rachida 26 London 16.0
13 Karima 30 Rabat 18.0
Pr. Mehdia AJANA 33
 Basic Data Analysis - Filtering:
PYTHON Filtering allows you to select rows based on
Data Analysis conditions.
Filtering with multiple conditions:
Example: Filter students aged over 25 and with
a grade greater than 15:

filtered_data = df[(df['Age'] > 25) & (df['Grade']

Pandas > 15)]
Data Analysis # Display the filtered data
print(filtered_data)
(df['Age'] > 20) & (df['Grade'] > 15): Filters
students who meet both conditions.
Name Age City Grade
12 Rachida 26 London 16.0
13 Karima 30 Rabat 18.0

Pr. Mehdia AJANA 34

 Basic Data Analysis - Grouping Data by a
PYTHON Column:
Data Analysis Grouping allows you to aggregate data by a
specific column (e.g., calculating averages by a
specific category).
Example: Group students by City and calculate
average Age.
# Group students by City and calculate average
Pandas
Age
Data Analysis
grouped_data = [Link]('City')['Age'].mean()

# Display the grouped data

print(grouped_data)

Pr. Mehdia AJANA 35

 Basic Data Analysis - Grouping Data by a
PYTHON Column:
Data Analysis Example: Group students by City and calculate
average Age.
grouped_data = [Link]('City')['Age'].mean()

Pandas
Data Analysis

Pr. Mehdia AJANA 36

 Basic Data Analysis - Grouping and
PYTHON Aggregating Multiple Columns:
Data Analysis You can aggregate multiple columns after
grouping using a list of column names.
Example: Group students by City and calculate
both average Age and Grade.
grouped_data = [Link]('City')[['Age',
'Grade']].mean()
Pandas
# Display the grouped data
Data Analysis
print(grouped_data)

Pr. Mehdia AJANA 37

 Basic Data Analysis - Sorting Data:
PYTHON You can sort data based on column values.
Data Analysis Example: Sort students by Grade in descending
order.
# Sort students by Grade (descending order)
sorted_data = df.sort_values(by='Grade',
ascending=False)

print(sorted_data)
Pandas
Data Analysis

Pr. Mehdia AJANA 38

 Data Analysis - Combining Filters and
PYTHON Grouping:
Data Analysis You can filter and group data together to perform
more advanced analysis.
Example: Filter students aged over 25, group by City,
and calculate average Grade:

# Filter students over 25 years old

Pandas filtered_data = df[df['Age'] > 25]
Data Analysis
# Group by City and calculate average Grade
grouped_data =
filtered_data.groupby('City')['Grade'].mean()

# Display the result

print(grouped_data)

Pr. Mehdia AJANA 39

 Data Analysis – Aggregation Functions:
PYTHON
You can apply a variety of aggregation functions to
Data Analysis grouped data in Pandas, depending on the analysis
you want to perform. Here are some important
ones:
sum() - Calculates the total sum of values within
each group.
filtered_data.groupby('City')['Grade'].sum()
Pandas count() - Counts the number of non-null values in
Data Analysis each group.
filtered_data.groupby('City')['Grade'].count()
median() - Finds the median value in each group =
the value in the middle, after you have sorted all
values ascending.
filtered_data.groupby('City')['Grade'].median()

Pr. Mehdia AJANA 40

 Data Analysis – Aggregation Functions:
PYTHON
min() - Retrieves the smallest value in each group.
Data Analysis filtered_data.groupby('City')['Grade'].min()
max() - Retrieves the largest value in each group.
filtered_data.groupby('City')['Grade'].max()
std() - Calculates the standard deviation for each
group.
Pandas
filtered_data.groupby('City')['Grade'].std()
Data Analysis
first() - Retrieves the first value in each group.
filtered_data.groupby('City')['Grade'].first()
last() - Retrieves the last value in each group.
filtered_data.groupby('City')['Grade'].last()

Pr. Mehdia AJANA 41

 Data Analysis – Aggregation Functions:
PYTHON
agg() - Allows multiple functions to be applied at
Data Analysis once.
filtered_data.groupby('City')['Grade'].agg(['mean',
'sum', 'min', 'max'])
apply() - Lets you apply a custom function to each
group.
Pandas filtered_data.groupby('City')['Grade'].apply(lambd
Data Analysis a x: [Link]() + [Link]())
describe() - Provides descriptive statistics for each
group.
filtered_data.groupby('City')['Grade'].describe()

Pr. Mehdia AJANA 42