Python Support Libraries
Introduction to Pandas
Pandas makes messy data clean and manageable for analysis.
Pandas is a powerful, open-source Python library that is
essential for data manipulation and analysis.
It provides fast, flexible, and expressive data structures
designed to make working with "relational" or "labeled"
data both easy and intuitive. It's a cornerstone tool in the
Python data science ecosystem, sitting on top of the NumPy
library.
That’s one of the main strengths of Pandas: it provides
powerful tools to clean, organize, and transform messy
Python Support Libraries
Introduction to Pandas
For example, with Pandas you can:
Handle missing values (.fillna(), .dropna())
Rename or reorder columns
Filter and slice data
Merge, join, and group data
Convert data types
Summarize with statistics (.mean(), .describe())
Python Support Libraries
Introduction to Pandas
Core Data Structures
Pandas introduces two primary data structures that form the
basis of nearly all its operations:
Series
A Series is a one-dimensional labeled array capable of holding
any data type (integers, strings, floats, Python objects, etc.).
It's like a single column in a spreadsheet or a single column in
a SQL table.
Each element has an associated label called the index.
Python Support Libraries
Introduction to Pandas
Core Data Structures
DataFrame.
A DataFrame is the most commonly used and important
Pandas structure. It's a two-dimensional labeled data structure
with columns of potentially different types.
It's like a spreadsheet or a SQL table.
It's a collection of Series objects, where each Series is a
column.
It has both a row index and distinct column labels.
Python Support Libraries
Introduction to Pandas
Key Features and Applications
Pandas is used extensively for tasks like:
Data Cleaning and Preparation: Handling missing data (e.g., NaN
values), removing duplicates, and standardizing formats.
Data Loading and Saving: Reading data from various formats like
CSV, Excel, SQL databases, and JSON, and writing data back out.
Data Wrangling/Transformation: Filtering, sorting, merging, and
reshaping data (pivoting, melting).
Python Support Libraries
Introduction to Pandas
Key Features and Applications
Data Alignment: Automatically aligning data by labels (indices)
when performing operations.
Statistical Analysis: Grouping data for aggregation (e.g.,
calculating sums, means, and counts).
Time Series Functionality: Providing specialized tools for working
with date and time data.
Python Support Libraries
Introduction to Pandas
Basic Usage Example
To start using Pandas, you typically import it under the conventional
alias pd.
Python Support Libraries
Introduction to Pandas
Series
A Series is a one-dimensional array of labeled data. It is
essentially a single column.
Structure: Has a sequence of values and an associated
sequence of labels called the index.
Analogy: A column in a spreadsheet
Python Support Libraries
Introduction to Pandas
DataFrame
A DataFrame is a two-dimensional, size-mutable, and
potentially heterogeneous tabular data structure.
Structure: A collection of Series objects, where each Series is
a column. It has both a row index and distinct column labels.
Analogy: A complete spreadsheet or a SQL table.
Python Support Libraries
Introduction to Pandas
Data Input (I/O)
Pandas provides fast tools for reading and writing data in
common formats:
Python Support Libraries
Introduction to Pandas
Selection and Indexing
Indexing is how you select specific subsets of your data (rows,
columns, or both). Pandas offers several ways to do this:
)
Python Support Libraries
Introduction to Pandas
Selection and Indexing
Row Selection and Selection by Label (.loc)
Python Support Libraries
Introduction to Pandas
Selection and Indexing
Row Selection by Position (.iloc)
The .iloc accessor is used for integer position-based indexing
(using 0-based index numbers.
Python Support Libraries
Introduction to Pandas
Conditional Selection (Boolean Indexing)
Conditional Selection (Boolean Indexing)
This is the most powerful method for filtering data. You pass a
boolean Series (True/False) to the DataFrame, and it returns
only the rows where the condition is True.
Filtering Rows based on a Column's Value:
Python Support Libraries
Introduction to Pandas
Index Setting
You can change the row index of a DataFrame using existing
columns.
Setting an Existing Column as the Index:
Python Support Libraries
Introduction to Pandas
Resetting the Index:
Moves the current index back into a column and sets a default
integer index (0, 1, 2, ...).
Python Support Libraries
DataFrame operations
DataFrame operations are the heart of data analysis in Pandas,
allowing you to inspect, clean, and transform your data.
Inspection and Summarization
These methods quickly tell you about the content and structure of your
DataFrame (df).
Python Support Libraries
Introduction to Pandas
Data Inspection and Cleaning
These methods focus on column values and handling missing
data.
Unique Values and Counts
These methods are typically applied to a single Series (a
column)
Python Support Libraries
Introduction to Pandas
Null Value Check and Replacement
Missing data is represented by NaN (Not a Number).
Python Support Libraries
Transformation and Restructuring
These methods modify the structure or values of the
DataFrame.
Dropping Data
Python Support Libraries
Sorting and Ordering
Python Support Libraries
Applying Custom Functions (The apply() Method)
The apply() method is extremely powerful for performing row-
wise or column-wise operations using a custom Python function
(often a lambda function).
Python Support Libraries
Missing data & its handling
Missing data is a common issue in real-world datasets and must be
handled before analysis. In Pandas, missing values are typically
represented by NaN (Not a Number), which stands for a float
data type in NumPy, but is used across all Pandas data types to
signify missing information.
Identifying Missing Data
The first step is always to locate where the missing data lies.
Check for Null Values:
[Link](): Returns a boolean DataFrame where True indicates
a missing value (NaN).
Python Support Libraries
Missing data & its handling
Count Null Values:
[Link]().sum(): This is the most common method. It returns a
Series showing the total count of missing values per column.
[Link]().sum().sum(): Returns the total number of missing
values in the entire DataFrame.
Python Support Libraries
Handling Missing Data (Imputation and Deletion)
There are two primary strategies for dealing with missing data:
Deletion and Imputation.
1. Deletion (Dropping)
If a column or row has too many missing values, or if the dataset is
large enough that losing a few rows won't significantly impact the
analysis, you can simply remove the data.
Python Support Libraries
Handling Missing Data (Imputation and Deletion)
Imputation (Filling)
Imputation involves replacing the missing values (NaNs) with substitute
values. The primary function for this is [Link]().
Python Support Libraries
Handling Missing Data (Imputation and Deletion)
Recommendation: Using the median is generally safer than the mean for
numerical data, as the median is less affected by outliers. The mode is used for
categorical data.
Note: Always use the inplace=True argument if you want to modify the
DataFrame directly, or assign the result back to a new (or the same)
variable: df = [Link](...).