Chapter-5

Data Analysis with Pandas

Contents
•An overview of the Pandas package
•The Pandas data structure- Series
•The Data Frame
•The Essential Basic Functionality: Rendering and altering labels
•Head and tail
•Binary operations
•Functional statistics
• Function application
• Sorting, Indexing and selecting data
• Working with Missing Data
• Advanced Uses of Pandas for Data Analysis - Hierarchical indexing
• The Panel data
Pandas
• Pandas is a data manipulation package in Python for tabular data. That
is, data in the form of rows and columns, also known as DataFrames.
Intuitively, you can think of a DataFrame as an Excel sheet.
• Pandas functionality includes data transformations, like sorting rows
and taking subsets, to calculating summary statistics such as the
mean, reshaping DataFrames, and joining DataFrames together.
• Pandas works well with other popular Python data science packages,
often called the PyData ecosystem, including
• NumPy for numerical computing
• Matplotlib, Seaborn, Plotly, and other data visualization packages
• scikit-learn for machine learning
What is pandas used for?
Pandas is used throughout the data analysis workflow. With pandas, you
can:
• Import datasets from databases, spreadsheets, comma-separated values
(CSV) files, and more.
• Clean datasets, for example, by dealing with missing values.
• Tidy datasets by reshaping their structure into a suitable format for analysis.
• Aggregate data by calculating summary statistics such as the mean of
columns, correlation between them, and more.
• Visualize datasets and uncover insights.
Pandas also contains functionality for time series analysis and analyzing text
data.
Uses of Python
• Data set cleaning, merging, and joining.
• Easy handling of missing data (represented as NaN) in floating point
as well as non-floating point data.
• Columns can be inserted and deleted from DataFrame and
higher-dimensional objects.
• Powerful group by functionality for performing split-apply-combine
operations on data sets.
• Data Visualization.
Benefits of the pandas package
Pandas is a powerful data manipulation tool packaged with several benefits including:
• Made for Python: Python is the world's most popular language for machine learning
and data science.
• Less verbose per unit operations: Code written in pandas is less verbose, requiring
fewer lines of code to get the desired output.
• Intuitive view of data: pandas offers exceptionally intuitive data representation that
facilitates easier data understanding and analysis.
• Extensive feature set: It supports an extensive set of operations from exploratory
data analysis, dealing with missing values, calculating statistics, visualizing univariate
and bivariate data, and much more.
• Works with large data: pandas handles large data sets with ease. It offers speed and
efficiency while working with datasets of the order of millions of records and
hundreds of columns, depending on the machine.
Installing Pandas
The first step in working with Pandas is to ensure whether it is installed
in the system or not. If not, then we need to install it on our system
using the pip command.
Follow these steps to install Pandas:
Step 1: Type ‘cmd’ in the search box and open it.
Step 2: Locate the folder using the cd command where the python-pip
file has been installed.
Step 3: After locating it, type the command:
pip install pandas
Importing Pandas
• After the Pandas have been installed in the system, you need to
import the library. This module is generally imported as follows:
import pandas as pd
Note:
Here, pd is referred to as an alias for the Pandas. However, it is not
necessary to import the library using the alias, it just helps in writing
less code every time a method or property is called.
Data Structures in Pandas Library
Pandas generally provide two data structures for manipulating data. They
are:
• Series
• DataFrame
Panda Series:
A Pandas Series is a one-dimensional labeled array capable of holding data
of any type (integer, string, float, Python objects, etc.). The axis labels are
collectively called indexes.
The Pandas Series is nothing but a column in an Excel sheet. Labels need
not be unique but must be of a hashable type.
The object supports both integer and label-based indexing and provides a
host of methods for performing operations involving the index.
Creating Panda Series
Pandas DataFrame
Pandas DataFrame is a two-dimensional data structure with labeled
axes (rows and columns).
Creating DataFrame
• Pandas DataFrame is created by loading the datasets from existing
storage (which can be a SQL database, a CSV file, or an Excel file).
• Pandas DataFrame can be created from lists, dictionaries, a list of
dictionaries, etc
Creating a DataFrame Using the Pandas
Library example
Accessing DataFrame
The Essential Basic Functionality
Rendering and altering labels
Renaming Columns Renaming Index Labels
The Essential Basic Functionality
Setting a Column as Index Resetting Index Changing Labels to Lowercase or Uppercase
Importing datasets to DataFrame
To import a dataset into a pandas DataFrame, you can use various
functions depending on the data file format.
Importing datasets to DataFrame
Importing text files
Reading text files is similar to CSV files. The only nuance is that you need
to specify a separator with the sep argument, as shown below. The
separator argument refers to the symbol used to separate rows in a
DataFrame. Comma (sep = ","), whitespace(sep = "\s"), tab (sep = "\t"),
and colon(sep = ":") are the commonly used separators. Here \s
represents a single white space character.
Head & Tail
We can view the first few or last few rows of a DataFrame using the
.head() or .tail() methods, respectively. You can specify the number of
rows through the n argument (the default value is 5)
Head & Tail
.describe()
• The .describe() method prints the summary statistics of all numeric
columns, such as count, mean, standard deviation, range, and
quartiles of numeric columns.
Binary Operations
In pandas, binary operations allow you to perform element-wise
operations between two DataFrame or Series objects.
Binary Operations
Subtraction
Functional Statistics
Pandas provides a suite of functions to perform various statistical operations on DataFrames and
Series. These functions make it easy to calculate summary statistics, identify data distributions, and
perform other statistical analyses.
Functional Statistics
Function application in Pandas
• To apply your own or another library’s functions to Pandas objects,
you should be aware of the three important methods.
• The appropriate method to use depends on whether your function
expects to operate on an entire DataFrame, row- or column-wise, or
element wise.
• Table wise Function Application: pipe()
• Row or Column Wise Function Application: apply()
• Element wise Function Application: applymap()
Table wise Function Application: pipe()
• In pandas, "table-wise function application" refers to applying
functions to an entire DataFrame as a single unit, rather than on
individual rows, columns, or elements.
• This approach is often useful when working with functions that can
operate on the whole table structure at once.
• Custom operations can be performed by passing the function and the
appropriate number of parameters as pipe arguments. Thus,
operation is performed on the whole DataFrame.
Table wise Function Application: pipe()
Example

#mean
Row or Column Wise Function Application: apply()

• The apply() function in pandas is very versatile and allows you to

apply functions either row-wise or column-wise. This flexibility makes
it useful for performing custom operations on each row or column in
a DataFrame.
• By default, the operation performs column wise, taking each column
as an array-like.
Row or Column Wise Function Application: apply()
Example
Element wise Function Application: applymap()

• The applymap() function in pandas allows you to apply a function

element-wise to each value in a DataFrame. Unlike apply(), which is
used for row-wise or column-wise operations, applymap() operates
on individual elements within the DataFrame. This makes it ideal for
transformations or formatting applied to every cell in a DataFrame.
Element wise Function Application: applymap()
Example
Sort Pandas DataFrame
Sort Pandas DataFrame
Pandas DataFrame Sorting in Ascending Order Sorting the Pandas DataFrame in Descending order
Indexing and Selecting Data with Pandas
• Indexing in pandas means simply selecting particular rows and columns of data
from a DataFrame. Indexing could mean selecting all the rows and some of the
columns, some of the rows and all of the columns, or some of each of the rows
and columns. Indexing can also be known as Subset Selection.
• Indexing and selecting data are crucial for efficiently working with data in Series
and DataFrame objects. These operations help you to slice, dice, and access
subsets of your data easily.
• These operations involve retrieving specific parts of your data structure, whether
it's a Series or DataFrame. This process is crucial for data analysis as it allows you
to focus on relevant data, apply transformations, and perform calculations
Indexing and Selecting Data with Pandas
• Indexing in pandas is essential because it provides metadata that
helps with analysis, visualization, and interactive display. It
automatically aligns data for easier manipulation and simplifies the
process of getting and setting data subsets.
Types of Indexing in Pandas
• Similar to Python and NumPy indexing ([ ]) and attribute (.) operators,
Pandas provides straightforward methods for accessing data within its
data structures.
• Label-Based Indexing with .loc
• Integer Position-Based Indexing with .iloc
• Indexing with Brackets []
Label-Based Indexing with .loc
The .loc indexer is used for label-based indexing, which means you can access rows
and columns by their labels. It also supports boolean arrays for conditional
selection.
loc() has multiple access methods like −
• single scalar label: Selects a single row or column, e.g., [Link]['a'].
• list of labels: Select multiple rows or columns, e.g., [Link][['a', 'b']].
• Label Slicing: Use slices with labels, e.g., [Link]['a':'f'] (both start and end are
included).
• Boolean Arrays: Filter data based on conditions, e.g., [Link][boolean_array].
loc takes two single/list/range operator separated by ','. The first one indicates the
row and the second one indicates columns.
 Label-Based Indexing with .loc
Selects all rows for a specific column using the loc indexer.
Label-Based Indexing with .loc
This example selects the specific rows for the
This example selecting all rows for multiple columns.
specific columns
Label-Based Indexing with .loc
selecting a range of rows for all columns using the loc indexer.
Integer Position-Based Indexing with .iloc
• The .iloc indexer is used for integer-based indexing, which allows you
to select rows and columns by their numerical position.
.iloc() has multiple access methods:
• Single Integer: Selects data by its position, e.g., [Link][0].
• List of Integers: Select multiple rows or columns by their positions,
e.g., [Link][[0, 1, 2]].
• Integer Slicing: Use slices with integers, e.g., [Link][1:3].
• Boolean Arrays: Similar to .loc, but for positions.
Integer Position-Based Indexing with .iloc
selects 4 rows for the all column using
the iloc indexer.
Integer Position-Based Indexing with .iloc
selects the specific data using the integer slicing
Integer Position-Based Indexing with .iloc
• selects the data using the slicing through list of values
Direct Indexing with Brackets "[]"
• Direct indexing with [] is a quick and intuitive way to access data,
similar to indexing with Python dictionaries and NumPy arrays.
• Single Column: Access a single column by its name.
• Multiple Columns: Select multiple columns by passing a list of column
names.
• Row Slicing: Slice rows using integer-based indexing.
Direct Indexing with Brackets "[]"
Direct indexing with brackets for accessing a selects the multiple columns using the
single column. direct indexing.
Advanced Uses of Pandas for Data Analysis -
Hierarchical indexing
• The index is like an address, that’s how any data point across the data
frame or series can be accessed. Rows and columns both have
indexes, rows indices are called index and for columns, it’s general
column names.
• Hierarchical Indexes :Hierarchical Indexes are also known as
multi-indexing is setting more than one column name as the index.
Hierarchical indexing

• Now the dataframe is using Hierarchical Indexing or multi-indexing.

• Note that here we have made 3 columns as an index (‘region’, ‘state’, ‘individuals’ ). The
first index ‘region’ is called level(0) index, which is on top of the Hierarchy of indexes, next
index ‘state’ is level(1) index which is below the main or level(0) index, and so on. So, the
Hierarchy of indexes is formed that’s why this is called Hierarchical indexing.
• We may sometimes need to make a column as an index, or we want to convert an index
column into the normal column, so there is a pandas reset_index(inplace = True) function,
which makes the index column the normal column.
Selecting Data in a Hierarchical Index or using
the Hierarchical Indexing
• For selecting the data from the dataframe using the .loc() method we have to
pass the name of the indexes in a list.
We cannot use only level(1) index for getting data from the
dataframe, if we do so it will give an error. We can only use
level (1) index or the inner indexes with the level(0) or main
index with the help list of tuples.
Selecting Data in a Hierarchical Index or using
the Hierarchical Indexing
The Panel data
• A panel is a 3D container of data. The term Panel data is derived from
econometrics and is partially responsible for the name pandas −
pan(el)-da(ta)-s.
• The Panel class is deprecated and has been removed in recent
versions of pandas. The recommended way to represent 3-D data is
with a MultiIndex on a DataFrame via the to_frame() method or with
the xarray package. pandas provides a to_xarray() method to
automate this conversion.
• The names for the 3 axes are intended to give some semantic
meaning to describing operations involving panel data. They are −
• items: axis 0, each item corresponds to a DataFrame contained inside.
• major_axis: axis 1, it is the index (rows) of each of the DataFrames.
• minor_axis: axis 2, it is the columns of each of the DataFrames.
[Link]()
A Panel can be created using the following constructor −
[Link](data, items, major_axis, minor_axis, dtype, copy)
The parameters of the constructor are as follows −
Create Panel
A Panel can be created using multiple ways like −
• From ndarrays
• From dict of DataFrames
Create an Empty Panel
An empty panel can be created using the Panel constructor.
Selecting the Data from Panel
Select the data from the panel using −
• Items
• Major_axis
• Minor_axis
• We have two items, and we retrieved item1. The result is a DataFrame with 4 rows and 3
columns, which are the Major_axis and Minor_axis dimensions.
Selecting the Data from Panel
 Working with Missing Data
• Missing Data can occur when no information is provided for one or more items or
for a whole unit. Missing Data is a very big problem in a real-life scenarios. Missing
Data can also refer to as NA(Not Available) values in pandas. In DataFrame
sometimes many datasets simply arrive with missing data, either because it exists
and was not collected or it never existed. For Example, Suppose different users
being surveyed may choose not to share their income, some users may choose not
to share the address in this way many datasets went missing.
• In Pandas missing data is represented by two value:
None: None is a Python singleton object that is often used for missing data in Python
code.
NaN : NaN (an acronym for Not a Number), is a special floating-point value
recognized by all systems that use the standard IEEE floating-point representation.
Working with Missing Data

• Pandas treat None and NaN as essentially interchangeable for indicating

missing or null values.
• To facilitate this convention, there are several useful functions for
detecting, removing, and replacing null values in Pandas DataFrame :
isnull()
notnull()
dropna()
fillna()
replace()
interpolate()
Checking for missing values using isnull() and
notnull()
• In order to check missing values in Pandas DataFrame, we use a function isnull() and notnull().
Both function help in checking whether a value is NaN or not. These function can also be used in
Pandas Series in order to find null values in a series.
Checking for missing values using isnull()
• In order to check null values in Pandas DataFrame, we use isnull() function this function return
dataframe of Boolean values which are True for NaN values.
Checking for missing values using isnull() and
notnull()
Checking for missing values using notnull()
• In order to check null values in Pandas Dataframe, we use notnull()
function this function return dataframe of Boolean values which are
False for NaN values.
Filling missing values using fillna(), replace()
and interpolate()
• In order to fill null values in a datasets, we use fillna(), replace() and interpolate()
function these function replace NaN values with some value of their own. All
these function help in filling a null values in datasets of a DataFrame.
Interpolate() function is basically used to fill NA values in the dataframe but it
uses various interpolation technique to fill the missing values rather than
hard-coding the value.
Filling missing values using fillna()
Filling null values with the previous ones Filling null value with the next ones
Filling a null values using replace() method
Replace the all Nan value in the data frame with -99
value.
Using interpolate() function to fill the missing
values using linear method.
Dropping missing values using dropna()
• In order to drop a null values from a dataframe, we used dropna()
function this function drop Rows/Columns of datasets with Null
values in different ways.
Dropping rows with at least 1 null value
Dropping missing values using dropna()
we drop rows with at least one Nan value (Null value)
Dropping missing values using dropna()
Dropping columns with at least 1 null value.
Drop a columns which have at least 1
missing values
Dropping missing values using dropna()
Dropping rows if all values in that row are missing