Practical 4
Title: Implement python code to clean the dataset for further analysis
Objective
Learn how to clean a CSV dataset for further analysis by handling missing values, removing
duplicates, trimming whitespace, and converting data types.
Requirements
Python 3.x installed
pandas library installed
Install pandas if you don’t have it:
pip install pandas
Text editor or IDE (e.g., VSCode, Jupyter Notebook, PyCharm)
Sample CSV dataset file
Dataset:
id,name,age,city,join_date,salary
1, Alice ,25,New York,2022-01-15,70000
2,Bob, ,Los Angeles,2021-12-01,80000
3,Charlie,30, New York,2022-02-20,
4,David,40,Chicago, ,90000
2,Bob, ,Los Angeles,2021-12-01,80000
5, Eve,29,Chicago,2022-03-10,85000
Step-by-Step Instructions
1. Load the CSV file using pandas
Python code
import pandas as pd
df = pd.read_csv('your_dataset.csv')
print("Initial Data:")
print(df)
2. Remove duplicate rows
Python code
df = df.drop_duplicates()
3. Handle missing values
For numeric columns (age, salary), fill missing values with the mean of the column.
For categorical/string columns (name, city), fill missing values with the mode (most
frequent value).
Python code
# Fill numeric columns
for col in df.select_dtypes(include=['number']).columns:
df[col].fillna(df[col].mean(), inplace=True)
# Fill categorical columns
for col in df.select_dtypes(include=['object']).columns:
df[col].fillna(df[col].mode()[0], inplace=True)
4. Trim whitespace from string columns
Python code
for col in df.select_dtypes(include=['object']).columns:
df[col] = df[col].[Link]()
5. Handle Missing Values in join_date
Replace empty strings in join_date with NaN:
Python code
df['join_date'].replace('', [Link], inplace=True)
Fill missing join_date values with the most frequent date (mode):
df['join_date'].fillna(df['join_date'].mode()[0], inplace=True)
Convert join_date column to datetime format:
df['join_date'] = pd.to_datetime(df['join_date'], errors='coerce')
6. Handle Missing Values in Numeric Columns (age and salary)
Convert columns to numeric type, coercing errors to NaN:
Python code
df['age'] = pd.to_numeric(df['age'], errors='coerce')
df['salary'] = pd.to_numeric(df['salary'], errors='coerce')
Fill missing values with the mean of each column:
Python code
df['age'].fillna(df['age'].mean(), inplace=True)
df['salary'].fillna(df['salary'].mean(), inplace=True)
7. Save the Cleaned Dataset to CSV
Python code
df.to_csv('your_dataset_cleaned.csv', index=False)
8. View the Cleaned Dataset
Python code
print("Cleaned Data:")
print(df)
Expected Ouput:
The missing age and salary values are replaced by column means; missing dates are converted
properly.
import pandas as pd
# Load CSV
df = pd.read_csv('your_dataset.csv')
# Remove duplicates
df = df.drop_duplicates()
# Strip whitespace from object columns
for col in df.select_dtypes(include=['object']).columns:
df[col] = df[col].[Link]()
# Fix empty strings in join_date to NaN
df['join_date'].replace('', [Link], inplace=True)
# Convert age and salary to numeric, coerce errors to NaN
df['age'] = pd.to_numeric(df['age'], errors='coerce')
df['salary'] = pd.to_numeric(df['salary'], errors='coerce')
# Fill missing numeric values with mean
df['age'].fillna(df['age'].mean(), inplace=True)
df['salary'].fillna(df['salary'].mean(), inplace=True)
# Fill missing join_date with mode BEFORE datetime conversion
df['join_date'].fillna(df['join_date'].mode()[0], inplace=True)
# Convert join_date to datetime
df['join_date'] = pd.to_datetime(df['join_date'], errors='coerce')
# Save cleaned dataset
df.to_csv('your_dataset_cleaned.csv', index=False)
# Show cleaned data
print("Cleaned Data:")
print(df)