BACSE101 – PROBLEM
SOLVING USING PYTHON
MODULE 5
DATA PROCESSING USING NUMPY AND PANDAS
TOPIC 3 –
PANDAS: HANDLING FILES
What Is Pandas?
• Pandas Is A Powerful Python Library Used For:
• Data Manipulation
• Data Analysis
• Reading And Writing Data (CSV, Excel, SQL, JSON, Etc.)
• Cleaning, Filtering, And Transforming Data
INSTALLING AND IMPORTING PANDAS
Example: Creating a DataFrame
import pandas as pd
data = {'Name': ['Alice', 'Bob', 'Charlie'],
'Age': [24, 27, 22],
'City': ['Delhi', 'Mumbai', 'Bangalore']}
df = [Link](data)
print(df)
Output:
Name Age City
0 Alice 24 Delhi
1 Bob 27 Mumbai
2 Charlie 22 Bangalore
Example: Reading Data from a CSV File
df = pd.read_csv('[Link]')
print([Link]())
•.head() → shows first 5 rows
•.tail() → shows last 5 rows
Data = {
'CHN': {'COUNTRY': 'china', 'POP': 1_398.72, 'area': 9_596.96,
'Gdp': 12_234.78, 'CONT': 'asia'},
'IND': {'COUNTRY': 'india', 'POP': 1_351.16, 'area': 3_287.26,
'Gdp': 2_575.67, 'CONT': 'asia', 'IND_DAY': '1947-08-15'},
‘USA': {'country': 'us', 'pop': 329.74, 'area': 9_833.52,
'Gdp': 19_485.39, 'cont': '[Link]',
'Ind_day': '1776-07-04'},
'IDN': {'COUNTRY': 'indonesia', 'POP': 268.07, 'area': 1_910.93,
'Gdp': 1_015.54, 'CONT': 'asia', 'IND_DAY': '1945-08-17'},
'BRA': {'COUNTRY': 'brazil', 'POP': 210.32, 'area': 8_515.77,
'Gdp': 2_055.51, 'cont': '[Link]', 'IND_DAY': '1822-09-07’}}
Columns = ('COUNTRY', 'POP', 'AREA', 'GDP', 'CONT', 'IND_DAY')
Versions of Python older than 3.6 did not guarantee the order of
keys in dictionaries. To ensure the order of columns is
maintained for older versions of Python and pandas, you can
specify index=columns:
Using the
pandas read_csv() and .to_csv() functions
• A comma-separated values (CSV) file is a plaintext file with
a .Csv extension that holds tabular data. This is one of the most
popular file formats for storing large amounts of data. Each row
of the CSV file represents a single table row. The values in the
same row are by default separated with commas, but you could
change the separator to a semicolon, tab, space, or some other
character.
Write a CSV file
• You can save your pandas dataframe as a CSV file with .to_csv():
Read a CSV File
Once your data is saved in a CSV file, you’ll likely want to load and
use it from time to time. You can do that with the
pandas read_csv() function:
USING PANDAS TO WRITE AND READ EXCEL
FILES
TOPIC 4, 5 – CREATING DATAFRAMES
AND SELECTION
• Data selection in Python—especially using Pandas—is a core skill for
any data scientist or inventor working with structured datasets.
• Select a Single Column
• Select Multiple Columns
• Select Rows by Label
• Select Rows by Position
• Filter Rows Conditionally
TOPIC 6 – DATA CLEANING
• Data cleaning means fixing and organizing messy data. Pandas
offers a wide range of tools and functions to help us clean and
preprocess our data effectively.
• Data cleaning often involves:
Dropping irrelevant columns.
Renaming column names to meaningful names.
Making data values consistent.
Replacing or filling in missing values.
Drop rows with missing values
• In pandas, we can drop rows with missing values using
the dropna() function.
Fill missing values
• To fill the missing values in pandas, we use the fillna() function.
Note: The inplace=True argument
here means that the operation will
modify the DataFrame directly,
rather than returning a new
DataFrame with the modifications.
Use aggregate functions to fill missing values
• Instead of filling with 0, we can also use aggregate functions to fill
missing values.
• Let's look at an example to fill missing values with the mean of
each column.
Handle duplicates values
• In pandas, to handle duplicate rows, we can use
the duplicated() and the drop_duplicates() function.
• duplicated() - to check for duplicates
• drop_duplicates() - remove duplicate rows
Rename column names to meaningful names
• To rename column names to more meaningful names in pandas, we can
use the rename() function.
TOPIC 7 – DATA FILTERING
• Filtering data is a common operation in data analysis. Pandas
allows us to filter data based on different conditions.
• We can filter the data in pandas in two main ways:
By column names (labels)
By the actual data inside (values)
Filter data by labels
• We can use the filter() function to select columns by their names or
labels.
• Filter data by values
• We can also filter data by values. Some of the common ways to
filter data by values are:
Using logical operator
The isin() method
The str accessor
The query() method
Logical operators
• You can filter rows based on column values using logical operators.
The isin() method
• The isin() method provides another way to filter data using
column values.
The str accessor
• We can effectively filter rows based on string values using
the str accessor.
The query() method
• This is the most flexible method for filtering a dataframe based
on column values.
A query containing the filtering conditions can be passed as a
string to the query() method.
Done in Module : 03
TOPIC 8,9 – GROUPING, AGGREGATION
Aggregation Functions
• Common aggregation functions include:
sum(): Total of values.
mean(): Average of values.
count(): Number of occurrences.
min() / max(): Minimum or maximum values.
1. Grouping with groupby() in Pandas
• The groupby() method in Pandas is used to group data based on one
or more columns and apply aggregation or transformation
functions.
• Basic Syntax
[Link]('column_name').agg({'another_column': 'function'})
import pandas as pd
# sample dataframe
data = {'category': ['a', 'b', 'a', 'b', 'c’], 'values': [10, 20, 30, 40, 50]}
df = [Link](data)
# group by 'category' and calculate the sum of 'values'
grouped = [Link]('category')['values'].sum()
print(grouped)
TOPIC 10 – PANDAS SORTING
• Sorting is a fundamental operation in data manipulation and
analysis that involves arranging data in a specific order.
• Sorting is crucial for tasks such as organizing data for better
readability, identifying patterns, making comparisons, and
facilitating further analysis.
Sort dataframe in pandas
• In pandas, we can use the sort_values() function to sort
a dataframe.
• In the above example, df.Sort_values(by='age') sorts
the df dataframe based on the values in the age column in
ascending order. And the result is stored in the sorted_df variable.
• To sort values in descending order, we use the ascending
parameter as:
Sort pandas dataframe by multiple columns
• We can also sort dataframe by multiple columns in pandas. When
we sort a pandas dataframe by multiple columns, the sorting is
done with a priority given to the order of the columns listed.
• To sort by multiple columns in pandas, you can pass the desired
columns as a list to the by parameter in
the sort_values() method.
Sort pandas series
• In pandas, we can use the sort_values() function to sort a series.
#Index sort pandas dataframe using sort_index()
• We can also sort by the index of a dataframe in pandas using
the sort_index() function.
• The sort_index() function is used to sort a dataframe or series by
its index. This is useful for organizing data in a logical order,
improving query performance, and ensuring consistent data
representation.
TOPIC 11 – PANDAS MERGE
• The merge operation in pandas merges two dataframes based on
their indexes or a specified column.
• The merge() in pandas works similar to joins in sql.
Types of join operations in merge()
• So far, we've not defined how to merge the dataframes, thus it defaults to
an inner join.
• However, we can specify the join type in the how argument. Here are the
5 join types we can use in the merge() method:
Left join
Right join
Outer join
Inner join (default)
Cross join
Left join
• A left join combines two dataframes based on a common key and
returns a new dataframe that contains all rows from the left
dataframe and the matched rows from the right dataframe.
• If values are not found in the right dataframe, it fills the space
with nan
Right join
• A right join is the opposite of a left join. It returns a new
dataframe that contains all rows from the right dataframe and the
matched rows from the left dataframe.
• If values are not found in the left dataframe, it fills the space
with nan.
Inner join
• An inner join combines two dataframes based on a common key
and returns a new dataframe that contains only rows that have
matching values in both of the original dataframes.
Outer join
• An outer join combines two dataframes based on a common key.
Unlike an inner join, an outer join returns a new dataframe that
contains all rows from both original dataframes.
• If values are not found in the dataframes, it fills the space
with nan.
Cross join
• A cross join in pandas creates the cartesian product of both
dataframes while preserving the order of the left dataframe.