Data PreProcessing
Summary:
1- Acquire the dataset
2- Import all the crucial libraries
3- Import the dataset
4- Identifying and handling the missing values
5- Encoding the categorical data
6- Splitting the dataset
7- Feature scaling
Data preprocessing in Machine Learning is a crucial step that helps enhance the quality of data to promote the
extraction of meaningful insights from the data. Data preprocessing in Machine Learning refers to the technique of
preparing (cleaning and organizing) the raw data to make it suitable for a building and training Machine Learning
models. In simple words, data preprocessing in Machine Learning is a data mining technique that transforms raw data
into an understandable and readable format.
Why Data Preprocessing in Machine Learning?
When it comes to creating a Machine Learning model, data preprocessing is the first step marking the initiation of the
process. Typically, real-world data is incomplete, inconsistent, inaccurate (contains errors or outliers), and often lacks
specific attribute values/trends. This is where data preprocessing enters the scenario – it helps to clean, format, and
organize the raw data, thereby making it ready-to-go for Machine Learning models. Let’s explore various steps of data
preprocessing in machine learning.
1- Acquire the dataset
Acquiring the dataset is the first step in data preprocessing in machine learning. To build and develop Machine
Learning models, you must first acquire the relevant dataset. This dataset will be comprised of data gathered from
multiple and disparate sources which are then combined in a proper format to form a dataset. Dataset formats differ
according to use cases. For instance, a business dataset will be entirely different from a medical dataset. While a
business dataset will contain relevant industry and business data, a medical dataset will include healthcare-related
data.
2- Import all the crucial libraries
The predefined Python libraries can perform specific data preprocessing jobs. Importing all the crucial libraries is the
second step in data preprocessing in machine learning.
In [2]:
%matplotlib inline
import numpy as np
import pandas as pd
import [Link] as plt
import seaborn as sns
import sklearn
3- Import the dataset
In this step, you need to import the dataset/s that you have gathered for the ML project at hand. Importing the dataset
is one of the important steps in data preprocessing in machine learning.
In [4]:
dataset = pd.read_csv(r"data_preprocessing_tutorial.csv")
In [5]:
df = [Link](dataset)
In [8]: # df
# [Link]()
[Link]()
Out [8]:
Country Age Salary Purchased
5 France 35.0 58000.0 Yes
6 Spain NaN 52000.0 No
7 France 48.0 79000.0 Yes
8 Germany 50.0 83000.0 No
9 France 37.0 67000.0 Yes
In [30]: X = [Link][:, :-1].values
y = [Link][:, -1].values
In [6]:
print(X)
[['France' 44.0 72000.0]
['Spain' 27.0 48000.0]
['Germany' 30.0 54000.0]
['Spain' 38.0 61000.0]
['Germany' 40.0 nan]
['France' 35.0 58000.0]
['Spain' nan 52000.0]
['France' 48.0 79000.0]
['Germany' 50.0 83000.0]
['France' 37.0 67000.0]]
In [7]:
print(y)
['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']
4- Identifying and handling the missing values
In data preprocessing, it is pivotal to identify and correctly handle the missing values, failing to do this, you might draw
inaccurate and faulty conclusions and inferences from the data. Needless to say, this will hamper your ML project.
some typical reasons why data is missing:
A. User forgot to fill in a field.
B. Data was lost while transferring manually from a legacy database.
C. There was a programming error.
D. Users chose not to fill out a field tied to their beliefs about how the results would be used or interpreted.
Basically, there are two ways to handle missing data:
Deleting a particular row – In this method, you remove a specific row that has a null value for a feature or a particular
column where more than 75% of the values are missing. However, this method is not 100% efficient, and it is
recommended that you use it only when the dataset has adequate samples. You must ensure that after deleting the
data, there remains no addition of bias. Calculating the mean – This method is useful for features having numeric data
like age, salary, year, etc. Here, you can calculate the mean, median, or mode of a particular feature or column or row
that contains a missing value and replace the result for the missing value. This method can add variance to the dataset,
and any loss of data can be efficiently negated. Hence, it yields better results compared to the first method (omission
of rows/columns). Another way of approximation is through the deviation of neighbouring values. However, this works
best for linear data.
In [10]:
[Link]().sum()
Out [10]: Country 0
Age 1
Salary 1
Purchased 0
dtype: int64
Solution 1 : Dropna
In [12]:
df1 = [Link]()
In [13]: # summarize the shape of the raw data
print("Before:",[Link])
# drop rows with missing values
[Link](inplace=True)
# summarize the shape of the data with missing rows removed
print("After:",[Link])
Before: (10, 4)
After: (8, 4)
Solution 2 : Fillna
In [26]:
df2 = [Link]()
[Link]()
Out [26]:
Country Age Salary Purchased
0 France 44.0 72000.0 No
1 Spain 27.0 48000.0 Yes
2 Germany 30.0 54000.0 No
3 Spain 38.0 61000.0 No
4 Germany 40.0 NaN Yes
In [22]:
import warnings
[Link]('ignore')
In [28]:
# fill missing values with mean column values
# Fill numeric columns with their mean
df2['Age'].fillna(df2['Age'].mean(), inplace=True)
df2['Salary'].fillna(df2['Salary'].mean(), inplace=True)
print([Link]().sum())
df2
Country 0
Age 0
Salary 0
Purchased 0
dtype: int64
Out [28]:
Country Age Salary Purchased
0 France 44.000000 72000.000000 No
1 Spain 27.000000 48000.000000 Yes
2 Germany 30.000000 54000.000000 No
3 Spain 38.000000 61000.000000 No
4 Germany 40.000000 63777.777778 Yes
5 France 35.000000 58000.000000 Yes
6 Spain 38.777778 52000.000000 No
7 France 48.000000 79000.000000 Yes
8 Germany 50.000000 83000.000000 No
9 France 37.000000 67000.000000 Yes
Solution 3 : Scikit-Learn
In [31]:
X
Out [31]: array([['France', 44.0, 72000.0],
['Spain', 27.0, 48000.0],
['Germany', 30.0, 54000.0],
['Spain', 38.0, 61000.0],
['Germany', 40.0, nan],
['France', 35.0, 58000.0],
['Spain', nan, 52000.0],
['France', 48.0, 79000.0],
['Germany', 50.0, 83000.0],
['France', 37.0, 67000.0]], dtype=object)
In [32]:
from [Link] import SimpleImputer
imputer = SimpleImputer(missing_values=[Link], strategy='mean')
[Link](X[:, 1:3])
X[:, 1:3] = [Link](X[:, 1:3])
In [33]:
print(X)
[['France' 44.0 72000.0]
['Spain' 27.0 48000.0]
['Germany' 30.0 54000.0]
['Spain' 38.0 61000.0]
['Germany' 40.0 63777.77777777778]
['France' 35.0 58000.0]
['Spain' 38.77777777777778 52000.0]
['France' 48.0 79000.0]
['Germany' 50.0 83000.0]
['France' 37.0 67000.0]]
5- Encoding the categorical data
Categorical data refers to the information that has specific categories within the dataset. In the dataset cited above,
there are two categorical variables – country and purchased.
Machine Learning models are primarily based on mathematical equations. Thus, you can intuitively understand that
keeping the categorical data in the equation will cause certain issues since you would only need numbers in the
equations.
Solution 1 : ColumnTransformer
In [34]:
from [Link] import ColumnTransformer
from [Link] import OneHotEncoder
ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), [0])], remainder='pas
X = [Link](ct.fit_transform(X))
In [35]: df
Out [35]:
Country Age Salary Purchased
0 France 44.0 72000.0 No
Country Age Salary Purchased
1 Spain 27.0 48000.0 Yes
2 Germany 30.0 54000.0 No
3 Spain 38.0 61000.0 No
4 Germany 40.0 NaN Yes
5 France 35.0 58000.0 Yes
6 Spain NaN 52000.0 No
7 France 48.0 79000.0 Yes
8 Germany 50.0 83000.0 No
9 France 37.0 67000.0 Yes
In [36]: print(X)
[[1.0 0.0 0.0 44.0 72000.0]
[0.0 0.0 1.0 27.0 48000.0]
[0.0 1.0 0.0 30.0 54000.0]
[0.0 0.0 1.0 38.0 61000.0]
[0.0 1.0 0.0 40.0 63777.77777777778]
[1.0 0.0 0.0 35.0 58000.0]
[0.0 0.0 1.0 38.77777777777778 52000.0]
[1.0 0.0 0.0 48.0 79000.0]
[0.0 1.0 0.0 50.0 83000.0]
[1.0 0.0 0.0 37.0 67000.0]]
Soluton 2 : Pd.get_dummies()
In [20]:
df2
Out [20]:
Country Age Salary Purchased
0 France 44.000000 72000.000000 No
1 Spain 27.000000 48000.000000 Yes
2 Germany 30.000000 54000.000000 No
3 Spain 38.000000 61000.000000 No
4 Germany 40.000000 63777.777778 Yes
5 France 35.000000 58000.000000 Yes
6 Spain 38.777778 52000.000000 No
7 France 48.000000 79000.000000 Yes
8 Germany 50.000000 83000.000000 No
9 France 37.000000 67000.000000 Yes
In [21]:
pd.get_dummies(df2)
Out [21]:
Age Salary Country_France Country_Germany Country_Spain Purchased_No Purchased_Yes
0 44.000000 72000.000000 1 0 0 1 0
1 27.000000 48000.000000 0 0 1 0 1
2 30.000000 54000.000000 0 1 0 1 0
3 38.000000 61000.000000 0 0 1 1 0
4 40.000000 63777.777778 0 1 0 0 1
5 35.000000 58000.000000 1 0 0 0 1
6 38.777778 52000.000000 0 0 1 1 0
7 48.000000 79000.000000 1 0 0 0 1
Age Salary Country_France Country_Germany Country_Spain Purchased_No Purchased_Yes
8 50.000000 83000.000000 0 1 0 1 0
9 37.000000 67000.000000 1 0 0 0 1
Solution 3 : LabelEncoder
In [22]:
from [Link] import LabelEncoder
le = LabelEncoder()
y = le.fit_transform(y)
In [23]:
print(y)
[0 1 0 0 1 1 0 1 0 1]
6- Splitting the dataset
Splitting the dataset is the next step in data preprocessing in machine learning. Every dataset for Machine Learning
model must be split into two separate sets – training set and test set.
In [24]: from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state
In [25]:
print(X_train)
[[0.0 0.0 1.0 38.77777777777778 52000.0]
[0.0 1.0 0.0 40.0 63777.77777777778]
[1.0 0.0 0.0 44.0 72000.0]
[0.0 0.0 1.0 38.0 61000.0]
[0.0 0.0 1.0 27.0 48000.0]
[1.0 0.0 0.0 48.0 79000.0]
[0.0 1.0 0.0 50.0 83000.0]
[1.0 0.0 0.0 35.0 58000.0]]
In [26]:
print(X_test)
[[0.0 1.0 0.0 30.0 54000.0]
[1.0 0.0 0.0 37.0 67000.0]]
In [27]: print(y_train)
[0 1 0 0 1 1 0 1]
In [28]:
print(y_test)
[0 1]
7- Feature scaling
Feature scaling marks the end of the data preprocessing in Machine Learning. It is a method to standardize the
independent variables of a dataset within a specific range. In other words, feature scaling limits the range of variables
so that you can compare them on common grounds.
Another reason why feature scaling is applied is that few algorithms like gradient descent converge much faster with
feature scaling than without it.
Why feature scalling?
Most of the times, your dataset will contain features highly varying in magnitudes, units and range. But since, most of
the machine learning algorithms use Eucledian distance between two data points in their computations, this is a
problem.
If left alone, these algorithms only take in the magnitude of features neglecting the units. The results would vary greatly
between different units, 5kg and 5000gms. The features with high magnitudes will weigh in a lot more in the distance
calculations than features with low magnitudes.
MinMax Scaler
MinMax Scaler shrinks the data within the given range, usually of 0 to 1. It transforms data by scaling features to a
given range. It scales the values to a specific value range without changing the shape of the original distribution.
In [29]:
from [Link] import MinMaxScaler
mm = MinMaxScaler()
X_train[:, 3:] = mm.fit_transform(X_train[:, 3:])
X_test[:, 3:] = [Link](X_test[:, 3:])
In [30]: print(X_train[:, 3:])
[[0.5120772946859904 0.11428571428571432]
[0.5652173913043479 0.45079365079365075]
[0.7391304347826089 0.6857142857142855]
[0.4782608695652175 0.37142857142857144]
[0.0 0.0]
[0.9130434782608696 0.8857142857142857]
[1.0 1.0]
[0.34782608695652173 0.2857142857142856]]
In [31]:
print(X_test[:, 3:])
[[0.1304347826086958 0.17142857142857149]
[0.43478260869565233 0.5428571428571427]]
Standard Scaler
StandardScaler follows Standard Normal Distribution (SND). Therefore, it makes mean = 0 and scales the data to unit
variance.
In [32]:
from [Link] import StandardScaler
sta = StandardScaler()
X_train[:, 3:] = sta.fit_transform(X_train[:, 3:])
X_test[:, 3:] = [Link](X_test[:, 3:])
In [33]:
print(X_train[:, 3:])
[[-0.19159184384578537 -1.0781259408412425]
[-0.014117293757057581 -0.07013167641635436]
[0.5667085065333245 0.6335624327104541]
[-0.3045301939022482 -0.3078661727429788]
[-1.9018011447007983 -1.4204636155515822]
[1.1475343068237058 1.2326533634535486]
[1.4379472069688963 1.5749910381638883]
[-0.740149544120035 -0.5646194287757338]]
In [34]:
print(X_test[:, 3:])
[[-1.4661817944830116 -0.9069571034860727]
[-0.4497366439748436 0.20564033932252992]]
When to Use Feature Scalling?
k-nearest neighbors with an Euclidean distance measure is sensitive to magnitudes and hence should be scaled for all
features to weigh in equally.
Scaling is critical, while performing Principal Component Analysis(PCA). PCA tries to get the features with maximum
variance and the variance is high for high magnitude features. This skews the PCA towards high magnitude features.
We can speed up gradient descent by scaling. This is because θ will descend quickly on small ranges and slowly on
large ranges, and so will oscillate inefficiently down to the optimum when the variables are very uneven.
Tree based models are not distance based models and can handle varying ranges of features. Hence, Scaling is not
required while modelling trees.
Algorithms like Linear Discriminant Analysis(LDA), Naive Bayes are by design equipped to handle this and gives
weights to the features accordingly. Performing a features scaling in these algorithms may not have much effect.
Normalization Vs. Standardization
The two most discussed scaling methods are Normalization and Standardization. Normalization typically means
rescales the values into a range of [0,1]. Standardization typically means rescales data to have a mean of 0 and a
standard deviation of 1 (unit variance).