Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Experiment No. 3: Data Preprocessing and Visualization
Date: 01/09/2025
Aim: Write a program to demonstrate the Data Loading, Preprocessing and Visualizati
on by using an appropriate data set.
Description:
Data Loading:
It is the first step in any data-driven project where raw data is imported into the working envir
onment from various sources such as local files, databases, or online repositories. Common fo
rmats include CSV, Excel, JSON, or SQL tables. In Python, libraries like Pandas and NumPy
are widely used for loading data, for example using functions such as read_csv() or loadtxt().
This step is crucial because it ensures that data is made accessible for further analysis and mo
del building. Without properly loading data, no further processing or visualization can take pl
ace.
Data Preprocessing:
It is the process of cleaning and transforming raw data into a format that is suitable for analys
is or machine learning models. Real-world datasets often contain missing values, duplicates,
and inconsistencies that can reduce the quality of results if not handled properly. Preprocessin
g typically involves handling missing values through deletion or imputation, removing duplic
ates, correcting inconsistent data, and converting data types into usable forms. Transformatio
ns such as normalization, standardization, and encoding categorical variables are also part of t
his stage. In some cases, new features are engineered or irrelevant ones are dropped to impro
ve the performance of the model. Preprocessing is an essential step as it enhances data quality,
reduces bias, and increases the accuracy and efficiency of analytical models.
Data Visualization:
It involves representing data in graphical or pictorial formats to better understand patterns, dis
tributions, and relationships. By visualizing the data, one can easily detect trends, outliers, an
d anomalies that may not be visible in raw tabular form. Visualization plays a major role in e
xploratory data analysis (EDA) by helping researchers and analysts make decisions based on
clear insights. Different visualization techniques are used depending on the type of data, such
as histograms and bar charts for univariate analysis, scatter plots and box plots for bivariate a
nalysis, and heatmaps or pair plots for multivariate analysis. Python libraries like Matplotlib,
Seaborn, and Plotly provide powerful tools for creating both static and interactive visualizatio
ns. Visualization is not only helpful for understanding the dataset but also for presenting findi
ngs in a way that is easy to interpret for non-technical audiences.
Procedure:
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
1) Set up Anaconda & Jupyter Notebook: Open Anaconda Navigator
Figure 1: Open Anaconda Navigator
2) Launch Jupyter Notebook. Click New Python [conda env: base]
Figure 2: Launch Jupyter notebook
3) Loading and displaying the "[Link]" dataset. The process is broken down into t
hree key parts:
● Import Libraries: The code first imports the necessary libraries:
○ pandas (pd) for creating and manipulating DataFrames.
○ numpy (np) for numerical operations.
○ arff from the [Link] module for loading data from an ARFF (Attribute-Relati
on File Format) file.
● Load and Convert Data: The [Link]("[Link]") function reads the ARFF fil
e. This function returns a tuple containing the data and its metadata. The data is access
ed from the first element of the tuple (data[0]) and then converted into a pandas DataF
rame using [Link]().
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
● Display First 5 Rows: Finally, the [Link]() method is used to display the first five ro
ws of the DataFrame, providing a quick look at the dataset's structure, column names,
and initial values. This helps to verify that the data has been loaded correctly.
Figure 3: loading and inspecting dataset
4) Overview of the dataset's structure, including:
● The number of entries (rows), which is 768 in this case.
● The total number of columns, which is 9.
● A list of each column's name, its non-null count (showing no missing values), and its
data type (Dtype).
● The memory usage of the DataFrame.
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 4: dataset info
5) Class distribution of diabetes dataset
Figure 5: class distribution
6)
Decoding the 'class' Column: The first step addresses a data formatting issue. The 'class' colu
mn contains byte strings (e.g., b'tested_positive'). This is a common issue when loading speci
fic file types like ARFF. To fix this, the .[Link]("utf-8") method is applied to the 'class' co
lumn, converting the byte strings into standard UTF-8 strings.
Replacing Zeros with Missing Value: The next step handles a data quality issue common in t
his dataset, where a zero value (0) in certain columns actually represents a missing value.
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
● A list called cols_with_zero is created to specify the columns where zeros should be tr
eated as missing data ('plas', 'pres', 'skin', 'insu', 'mass').
● The .replace(0, [Link]) method is then applied to these columns. It finds every 0 in th
e specified columns and replaces it with [Link], which is NumPy's representation of a
"Not a Number" or missing value.
Counting Missing Values: Finally, the code checks for missing values after the replacement.
The [Link]().sum() method is used to count the number of NaN values in each column. The
output shows that the replacement process successfully identified and marked the missing val
ues, with some columns like insu and skin now having a significant number of missing entrie
s.
Figure 6: decoding and replacing missing values
7) Filling missing values with median
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 7: imputation by using median
8) Normalizing the dataset features
Figure 8: normalize dataset features
9) Import libraries for Visualization
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 9: importing libraries
10) Class Distribution Visualization
Figure 10: class distribution
11) Correlation Heatmap Visualization
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 11: Correlation Heatmap
12) Histogram Visualization
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 12: Histogram
13) Pairplot Visualization
Stanley College of Engineering and Technology for Women
Department of Computer Science and Engineering
Roll No : 160622733175
Name : Saba Anjum
Figure 13: pairplot
Conclusion:
This project effectively demonstrates a comprehensive data analysis and preprocessing pipeli
ne for the [Link] dataset. The initial steps involved loading the data and handling critica
l data quality issues by decoding the class column and replacing zero values with NaNs in col
umns like plas, pres, and insu. This crucial cleaning process was followed by a thorough expl
oratory data analysis (EDA). This included creating a bar chart to visualize class distribution,
histograms to understand the spread of each feature, a pair plot to examine relationships betw
een all variables, and a correlation heatmap to identify strong correlations. This focus on detai
led data preparation and visualization ensures that any subsequent machine learning models a
re built on a solid, clean, and well-understood foundation.
Stanley College of Engineering and Technology for Women