Module 2:
Data Preprocessing and
Exploratory Data Analysis
By
Prof. Deepak V Ulape (Asst. Professor)
College of Management,
MIT ADT University, Loni Kalbhor, Pune
Data Preprocessing in Python
• Data preprocessing is the first step in any data analysis or machine
learning pipeline.
• It involves cleaning, transforming and organizing raw data into a
structured format to ensure accuracy, consistency and readiness for
modelling.
• This step improves data quality and directly impacts the performance
of analytical or predictive models.
Steps-by-Step implementation
Let's implement various preprocessing features,
Step 1: Import Libraries and Load Dataset
We prepare the environment with libraries liike pandas,
numpy, scikit learn, matplotlib and seaborn for data
manipulation, numerical operations, visualization and
scaling. Load the dataset for preprocessing.
import pandas as pd
import numpy as np
from [Link] import MinMaxScaler, StandardScaler
import seaborn as sns
import [Link] as plt
df = pd.read_csv('Geeksforgeeks/Data/[Link]')
[Link]()
• Step 2: Inspect Data Structure and Check Missing Values
• We understand dataset size, data types and identify any incomplete
(missing) data that needs handling.
• [Link](): Prints concise summary including count of non-null entries and
data type of each column.
• [Link]().sum(): Returns the number of missing values per column.
[Link]()
print([Link]().sum())
• Step 3: Statistical Summary and Visualizing Outliers
• Get numeric summaries like mean, median, min/max and detect unusual
points (outliers). Outliers can skew models if not handled.
• [Link](): Computes count, mean, std deviation, min/max and quartiles
for numerical columns.
• Boxplots: Visualize spread and detect outliers using matplotlib’s boxplot().
[Link]()
fig, axs = [Link](len([Link]), 1, figsize=(7, 18), dpi=95)
for i, col in enumerate([Link]):
axs[i].boxplot(df[col], vert=False)
axs[i].set_ylabel(col)
plt.tight_layout()
[Link]()
• Step 4: Remove Outliers Using the Interquartile Range (IQR) Method
• Remove extreme values beyond a reasonable range to improve model
robustness.
• IQR = Q3 (75th percentile) – Q1 (25th percentile).
• Values below Q1 - 1.5IQR or above Q3 + 1.5IQR are outliers.
• Calculate lower and upper bounds for each column separately.
• Filter data points to keep only those within bounds.
q1, q3 = [Link](df['Insulin'], [25, 75])
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
clean_df = df[(df['Insulin'] >= lower) & (df['Insulin'] <= upper)]
Step 5: Correlation Analysis
• Understand relationships between features and the target variable
(Outcome). Correlation helps gauge feature importance.
• [Link](): Computes pairwise correlation coefficients between columns.
• Heatmap via seaborn visualizes correlation matrix clearly.
• Sorting correlations with corr['Outcome'].sort_values() highlights features
most correlated with the target.
corr = [Link]()
[Link](dpi=130)
[Link](corr, annot=True, fmt='.2f', cmap='coolwarm')
[Link]()
print(corr['Outcome'].sort_values(ascending=False))