Data Preprocessing with Python (Pandas)

Course Topic

Apply data preprocessing techniques and prepare data for mining

Learning Outcomes

By the end of the lesson, students will be able to:

1. Explain the importance of data preprocessing in data mining.

2. Perform data cleaning (handling missing values, duplicates, outliers).

3. Apply data integration (merging and concatenating datasets).

4. Execute data transformation (normalization, standardization, encoding).

5. Use Python Pandas in Google Colab/Jupyter for hands-on preprocessing tasks.

1. Data Preprocessing and its Role in Data Mining (5 mins)

Before we can apply data mining or machine learning algorithms, our data must be in the right
format. Real-world data is often incomplete, inconsistent, contains errors, or has irrelevant
information. This is where data preprocessing comes in.

 Definition:
Data preprocessing is the process of preparing raw data into a clean and usable format
so it can be effectively analyzed or mined.

 Role in Data Mining:

o Ensures accuracy (removes errors and inconsistencies).

o Improves efficiency (reduces complexity, handles large datasets).

o Increases performance (better results from machine learning models).

Examples:

 Raw student records may have missing grades, duplicated IDs, or inconsistent formats.
Without preprocessing, predictions (like student performance)

2. Data Processing Three Stages: Cleaning, Integration, Transformation (7 mins)

 Data Cleaning:

o Deals with missing values, duplicates, errors, outliers.

o Example: If “Age” is missing for some students, replace it with the average age.

 Data Integration:
 o Combines data from multiple sources into a single dataset.

o Example: Merging a student’s profile dataset with their grades dataset.

 Data Transformation:

o Converts data into a suitable format for analysis (scaling, normalization, encoding
categorical variables).

o Example: Convert “Male/Female” into numeric values (0 and 1).

What is Pandas?
Definition

Pandas is a popular open-source Python library used for data manipulation and analysis.

 The name comes from “Panel Data” (a term used in econometrics).

 It provides data structures and functions to handle structured data (tables,

Key Features

1. DataFrame and Series –

o Series → one-dimensional data (like a column).

o DataFrame → two-dimensional table (rows & columns, like Excel).

2. Data Cleaning –

o Handle missing values, duplicates, outliers easily.

3. Data Transformation –

o Convert, normalize, encode, and reshape data.

4. Data Integration –

o Merge, join, and concatenate datasets.

5. Fast and Efficient –

o Built on NumPy, optimized for performance.

Why Use Pandas in Data Preprocessing?

 Simplifies working with CSV, Excel, SQL, JSON files.

 Provides easy-to-read syntax compared to raw Python.

 Allows quick exploration with functions like:

 [Link]() # Shows first 5 rows

 [Link]() # Summary of dataset

Code Example. Download the CSV file with a file name ‘python_practice_dataset’ at:
[Link]

Step 1. Install Python extension in your VS code IDE and Create a Folder for your work
environment.

Step 2. Create your python file so that you can install pandas using terminal.

Step 3. Save your CSV File in your Folder to execute the dataset in your panda’s file using df =
pd.read_csv("filename"). Then print the dataset in
As you can see it is already printed in dataframe format of data processing were in you can
see the rows and columns.

Step 4. Import your code and CSV file in Jupiter Notebook for Interactive Coding.

What is Jupyter Notebook?

Definition:

Jupyter Notebook is an open-source interactive environment that allows you to write, run, and
document Python code in a web-based interface.

 It’s widely used in data science, machine learning, and education.

 The name Jupyter comes from Julia, Python, and R (the languages it was designed for).

Key Features

1. Interactive Coding – You can run code in small blocks called cells.

2. Mix Code & Documentation – Supports Markdown, so you can add explanations, math
formulas, and charts along with your code.

3. Data Visualization – Integrates with libraries like Matplotlib, Seaborn, Plotly for graphs
and charts.

4. Step-by-Step Execution – Run and test code line by line, making debugging and learning
easier.

5. Supports Multiple Languages – Not just Python (also R, Julia, etc.) through kernels.

Why Use Jupyter Notebook in Your Lesson?

 Perfect for teaching: students see code, output, and explanation in one place.
  Great for data preprocessing labs: they can load a dataset, clean it, and show results step
by step.

 Similar to a digital lab notebook where they can save work and share with others.

Click CTRL+Shift+P to open or create jupyter notebook file and save it to your folder.

Step 5. Click + Create code to Print the data frame of your CSV file.

1.

Common DataFrame (df) Exploration Codes in Jupyter Notebook

2. View the Dataset

[Link]() # Shows first 5 rows

[Link](10) # Shows first 10 rows

[Link]() # Shows last 5 rows

3. General Information

[Link]() # Column names, data types, missing values

[Link] # Number of rows and columns (rows, cols)

[Link] # List of column names

[Link] # Data type of each column

4. Descriptive Statistics

[Link]() # Stats summary for numeric columns

[Link](include='all') # Stats for all columns (numeric + categorical)

df['Age'].describe() # Stats summary for a single column

5. Check Missing Values & Duplicates

[Link]().sum() # Count missing values per column

[Link]().sum() # Count duplicated rows

6. Explore Unique Values

df['Gender'].unique() # Unique values in 'Gender' column

df['Gender'].value_counts() # Count how many Male/Female

7. Quick Data Summaries

[Link](3) # First 3 rows

[Link](3) # Last 3 rows

[Link](3) # Random 3 rows

8. Correlations (Numeric Data Only)

[Link](numeric_only=True) # Correlation between numeric columns

9. Sorting Data

df.sort_values(by='Age', ascending=True).head(5) # Youngest 5

df.sort_values(by='Grade', ascending=False).head() # Top grades

df.to_csv("cleaned_data.csv", index=False)

Activity 1. Paste the Outputs of Data Processing from number 2-10 in this file.