BAHRAIN POLYTECHNIC
IT7009 Artificial Intelligence
Week- 4: Cleaning Datasets
1. Uploading dataset on Colab:
For uploading datasets on your Colab notebook, follow the steps as below:
• Open Google Colab Notebook
• Locate the Files Panel on the left sidebar and click the folder icon.
• Click up-arrow icon “Upload to Session Storage” and browse your Dataset.
• Select your dataset file (e.g., [Link]) from your computer. And confirm Upload.
• The uploaded file will now appear under the Files panel.
2. Examples based on Data Cleaning
Examples 1: The following Python program loads a dataset from a CSV file using pandas. It
then checks for missing values in each column and displays the total count. Additionally, the
program provides detailed information about the dataset, including column names, data types,
and the number of non-null entries.
Examples 2: The following Python program demonstrates how rows and columns containing
missing values are removed. After cleaning, the updated dataset is displayed and saved as a
new CSV file named cleaned_dataset.csv. This ensures the dataset is free of null values and
ready for further analysis or machine learning tasks.
Examples 3: The following Python program demonstrates how to handle missing values in
numeric columns of a dataset using mean imputation with pandas. The dataset is first loaded
from a CSV file, and the total number of missing values in each column is displayed. Only
numeric columns are selected, and any missing values in these columns are replaced with the
mean of the respective column. Finally, the program prints the updated count of missing values
and shows the first few rows of the dataset after imputation. This approach helps maintain the
dataset size while filling missing numeric values for further analysis or machine learning.
Examples 4: The following Python program demonstrates how to handle missing values in a
dataset using mode imputation with pandas. The dataset is loaded from a CSV file, and the
program first displays the number of missing values in each column. Mode imputation replaces
missing values in both numeric and categorical columns with the most frequent value in each
column. After imputation, the program prints the updated count of missing values and displays
the first few rows of the dataset. This method ensures that missing values are filled with the
most common entries, maintaining the integrity of categorical and numeric data for further
analysis.
Examples 5: The following Python program demonstrates how to detect and remove duplicate
rows in a dataset using pandas. A sample dataset with duplicate entries is created and displayed.
The program first checks which rows are duplicates and then, duplicate rows are removed while
keeping the first occurrence. Finally, the cleaned dataset without duplicates is printed. This
process ensures data integrity and prevents repeated records from affecting analysis or machine
learning results.
Examples 6: The following Python program demonstrates how to encode categorical variables
in a dataset using pandas and scikit-learn. A sample dataset of employees with categorical
columns “Gender” and “Department” is created. The program first applies Label Encoding to
the “Gender” column, converting text labels into numeric form. Then, One-Hot Encoding is
applied to the “Department” column, creating separate binary columns for each department.
Finally, the transformed dataset is displayed. This process makes categorical data suitable for
machine learning models that require numerical input.
3. Practice Questions
Exercise 1: Download the dataset of employees named “employee_dataset” from the Moodle under
Week 4: Practical Resources. You are required to perform the following operations and then export
the cleaned dataset in your local repository.
• Remove duplicate rows.
• Impute missing Age and Salary values using mean.
• Encode Gender using Label Encoding.
• Detect outliers in Salary using the IQR method.
Exercise 2: Download the dataset of employees named “healthcare_dataset” from the Moodle
under Week 4: Practical Resources. You are required to perform the following operations and then
export the cleaned dataset in your local repository.
• Remove duplicate patient entries.
• Impute missing values in Blood_Pressure and BMI using mean.
• Encode Diagnosis and Gender appropriately.
• Detect outliers in age column of patients (check for unrealistic ages >100).