Module 2
Data Preprocessing & Feature
Engineering
Data Collection
• In real time, the original data
gathered are
• highly susceptible to noisy,
missing, and inconsistent due to
their typically huge size (often
several gigabytes or more)
• origin from multiple,
heterogenous sources
• Having irrelevant features in data can decrease the accuracy of the models and make the
model learn based on irrelevant features.
• Data preprocessing - process of preparing the raw data and making it suitable for a
machine learning model.
• Feature Engineering - Choosing informative, discriminating and independent features is a
crucial step for effective algorithms in pattern recognition, classification and regression.
• How can the data be preprocessed in order to improve the quality of the data and,
consequently, of the machine learning results?
• How can the data be preprocessed so as to improve the efficiency and ease of the
machine learning process?”
Know Your Data
• Knowledge about your data is useful for data preprocessing.
• Gaining insight from the data will help with the subsequent analysis. (EDA)
• types of attributes
• kind of values – discrete or continuous
• spot any outliers
• measure the similarity
• the values distributed
• Knowing the basic statistics regarding each attribute makes it easier to fill in missing
values, smooth noisy values, and spot outliers during data preprocessing.
• Knowledge of the attributes and attribute values can also help in fixing inconsistencies
incurred during data integration.
• Visualize the data to get a better sense that helps to identify relations, trends, and
biases “hidden” in unstructured data sets.
Zoo Animal Classification The 7 Class Types are: Mammal, Bird, Reptile, Fish, Amphibian,
Bug and Invertebrate
Machine Learning:
Attributes / Features
• An attribute is a data field, representing a characteristic or feature of a data
object.
• A set of attributes used to describe a given object is called an attribute vector (or
feature vector).
• The type of an attribute is determined by the set of possible values
• Nominal – categorical
• Binary - only two categories or states: 0 or 1
• Ordinal - meaningful order or ranking among them
• Numeric - measurable quantity
• The numerical attributes as being either discrete or continuous
• A feature is a measurable property of the object you’re trying to analyze.
• In datasets, features appear as columns:
Also features like--
• age of the house
• location of the house
• number of times it has been bought
and sold
One column of the data - one feature - "variable or attribute"
More number of features - attributes - increase dimensions
Basic statistical descriptions
• There are three areas of basic statistical descriptions
• Measures of central tendency
• measure the location of the middle or center of a data distribution.
• mean, median, mode, and midrange.
• Dispersion of the data
• how are the data spread out?
• range, quartiles, and interquartile range; the five-number summary and boxplots; and the
variance and standard deviation of the data
• Graphic displays of basic statistical descriptions
• bar charts, pie charts, and line graphs
• skewness and kurtosis
x x-x` (x-x`)^2 x x-x` (x-x`)^2
1 15 1 2
2 15 2 7
3 15 3 14
4 14 4 22
5 16 5 30
Mean Variance Mean Variance
SD
SD
values are grouped close values are spread out
x x-x` (x-x`)^2 x x-x` (x-x`)^2
1 15 1 2
2 15 2 7
3 15 3 14
4 14 4 22
5 16 5 30
Mean Variance Mean Variance
SD
SD
values are grouped close values are spread out
Note: Sort the data before use
Note: Sort the data before use
Box plot
[Link]
Inter quartile range (IQR) method
Normal distribution:
• Bell shaped
• Symmetrical
• Unimodal
Skewness speaks about the manner in which the data is spread Kurtosis indicates the way how the data is concentrated
across the mean across the mean or dispersed equally across it which
determines the peakedness of the curve
Data Quality
• The factors comprising data quality, including accuracy, completeness,
consistency, timeliness, believability, and interpretability.
• Inspect the company’s database, identifying and selecting the attributes or
dimensions to be included in the analysis
• Some needed information has not been recorded
• There may be errors, unusual values, and inconsistencies in the data
• The data are incomplete (lacking attribute values or certain attributes of interest, or
containing only aggregate data); inaccurate or noisy (containing errors, or values that deviate
from the expected); inconsistent (e.g., containing discrepancies)
• Three of the elements defining data quality: accuracy, completeness, and
consistency.
• Inaccurate, incomplete, and inconsistent data are common-place properties of
large real-world databases.
• Timeliness refers to the time expectation for accessibility and availability of
information.
• Timeliness can be measured as the time between when information is expected and when it
is readily available for use.
• Believability reflects how much the data are trusted by users
• Interpretability reflects how easy the data are understood.
Data Preprocessing Techniques
• Data cleaning
• can be applied to remove noise and correct inconsistencies in
data.
• Data integration
• merges data from multiple sources into a coherent data
store.
• Data reduction
• can reduce data size by, for instance, aggregating, eliminating
redundant features, or clustering.
• Data transformations (e.g., normalization)
• may be applied, where data are scaled to fall within a smaller
range like 0.0 to 1.0.
• These techniques are not mutually exclusive; they may
work together.
Filling in missing values,
Smoothing noisy data,
Identifying or removing outliers,
Resolving inconsistencies,
Redundant or duplicate data
Include data from multiple sources
Major Tasks in Data Preprocessing
reduced representation of the data set
data are transformed or consolidated into appropriate forms
1. Data Cleaning
• Data cleaning routines work to “clean” the data by filling in missing values,
smoothing noisy data, identifying or removing outliers, and resolving
inconsistencies.
• The dirty data can cause confusion for the machine learning process, resulting in
unreliable output.
• Most machine learning process have some procedures for dealing with
incomplete or noisy data.
• A useful preprocessing step is used to run your data through some data cleaning
routines.
Missing Data - inferences
Missing data in the training data
set can reduce the power / fit of
a model
or
can lead to a biased model
Reason : Not analyzed the
behavior and relationship with
other variables correctly.
Missing data can lead to
Inference - The chances of playing Inference - The chances of playing wrong prediction or
cricket by males is higher than cricket by females is higher than classification
females males
Handling missing values
• There are 2 primary ways of handling missing values: [Link]()
• Deleting the Missing values
• Imputing the Missing Values
• There are 2 ways one can delete the missing values:
• Deleting the entire row [Link]().sum()
• Deleting the entire column
[Link](axis=0) [Link](axis=1)
[Link](['col'],axis=1)
[Link](index=[Link][val],axis=0)
• Imputing the missing values:
• Fill in the missing value manually
• Use a global constant to fill in the missing value
• Use a measure of central tendency for the attribute (e.g., Symmetric - mean or Skewed –
median, Categorical – mode) to fill in the missing value
• Use the attribute mean or median for all samples belonging to the same class as the given
tuple
• Replacing with previous value, next value
• Use different interpolation methods like ‘polynomial’, ‘linear’, ‘quadratic’
• Use the most probable value to fill in the missing value (regression, Bayesian inference or
decision tree)
• Imputation of Missing Value Using sci-kit learn Library
• Univariate Approach – SimpleImputer
• Multivariate approach - KNNImputer or IterativeImputer
df['col'].fillna(val)
[Link](method='ffill') [Link](method='bfill') [Link](method='nearest')
mode
Linear Regression to predict missing continuous variable values.
Logistic Regression to predict missing categorical variable values.
KNN Imputer
X1 X2 Label D3=16
Test Input Bad
7 7 Bad
D1=9 D4=25
7 4 Bad D2=13 Bad
3 4 Good Good Good
1 4 Good
X1 X2 Distance, Rank Is it included in Label
D Minimum 3- nearest
Distance neighbors?
7 7 16 3 Yes Bad
7 4 25 4 No -
3 4 9 1 Yes Good
1 4 13 2 Yes Good
Interpolation
interpolation estimates the value of missing values based
on the surrounding trends and patterns.
use interpolation when there is an order or a sequence
and to estimate a missing value in the sequence
Pandas offers various interpolation methods ('linear',
'time', 'index', 'values', 'pad', 'nearest', 'zero', 'slinear',
'quadratic', 'cubic', 'barycentric', 'krogh', 'polynomial',
'spline', 'piecewise_polynomial', 'from_derivatives',
'pchip', 'akima', 'cubicspline’)
[Link](method='linear’)
Handling noisy data
• Noise is a random error or variance in a measured variable.
• Some basic statistical description techniques (e.g., boxplots and scatter plots),
and methods of data visualization can be used to identify outliers, which may
represent noise.
• Data smoothing techniques that “smooth” out the data to remove the noise
• Binning: Binning methods smooth a sorted data value by consulting its
“neighborhood,” that is, the values around it.
• The sorted values are distributed into a number of “buckets,” or bins.
• Because binning methods consult the neighborhood of values, they perform local smoothing.
Binning
• smoothing by bin means
• smoothing by bin medians
• smoothing by bin boundaries
In general, the larger the width, the greater the effect of the smoothing.
Binning
• smoothing by bin means
• smoothing by bin medians
• smoothing by bin boundaries
In general, the larger the width, the greater the effect of the smoothing.
• Outlier analysis:
• Intuitively, values that fall outside of the set of clusters may be considered
outliers
Data without outliers: [10, 12, 14, 15, 13] Data with an outlier: [10, 12, 14, 15, 100]
•Mean = 12.8, Median = 13 •Mean = 30.2 (significantly distorted), Median = 14 (unchanged)
Outlier analysis – process of identifying extreme values, or abnormal observations
• Mean > Median (right side) or Mean < Median (left side)
• outside of minimum and maximum points Q1–1.5*IQR and Q3+1.5*IQR
respectively
• Box plot & scatter plot visualization
• Z-score is >3
Q1- 3.5
Q2- 6.0
Q3- 8.0
IQR- 4.5
-3.25 to 14.75
Handling inconsistent and redundant data
• Inconsistencies in naming conventions or data codes, or inconsistent formats for
input fields.
• MM-DD-YYYY, YYYY/MM/DD
• Price(USD), Cost(INR)
• Duplicate tuples also require data cleaning.
• df.drop_duplicates()
Data cleaning as a process
• The first step in data cleaning as a process is discrepancy detection.
• Discrepancies can be caused by several factors, including poorly designed data
entry forms that have many optional fields, human error in data entry, deliberate
errors and data decay.
• Discrepancies may also arise from inconsistent data representations and
inconsistent use of codes.
• Other sources of discrepancies include errors in instrumentation devices that
record data and system errors.
• Errors can also occur when the data are (inadequately) used for purposes other
than originally intended.
• There may also be inconsistencies due to data integration.
• “So, how can we proceed with discrepancy detection?”
• As a starting point, use any knowledge you may already have regarding
properties of the data.
• Such knowledge or “data about data” is referred to as metadata.
• From this, you may find noise, outliers, and unusual values that need
investigation.
• The data should also be examined regarding unique rules, consecutive rules, and
null rules.
• A unique rule says that each value of the given attribute must be different from all other
values for that attribute.
• A consecutive rule says that there can be no missing values between the lowest and highest
values for the attribute, and that all values must also be unique (e.g., as in check numbers).
• A null rule specifies the use of blanks, question marks, special characters, or other strings
that may indicate the null condition (e.g., where a value for a given attribute is not available),
and how such values should be handled.
• There are a number of different commercial tools that can aid in the discrepancy
detection step.
• Data scrubbing tools use simple domain knowledge to detect errors and make
corrections in the data.
• Data auditing tools find discrepancies by analyzing the data to discover rules and
relationships, and detecting data that violate such conditions.
• Some data inconsistencies may be corrected manually using external references.
• Most errors, however, will require data transformations.
• Once we find discrepancies, we typically need to define and apply (a series of)
transformations to correct them.
• Commercial tools can assist in the data transformation step.
• Data migration tools allow simple transformations to be specified such as to
replace the string “gender” by “sex.”
• ETL (extraction/transformation/loading) tools allow users to specify transforms
through a graphical user interface (GUI).
• We may also choose to write custom scripts for this step of the data cleaning
process.
• The two-step process of discrepancy detection and data transformation iterates.
• This process, however, is error-prone and time consuming.
• Some transformations may introduce more discrepancies.
• Some nested discrepancies may only be detected after others have been fixed.
• The entire data cleaning process also suffers from a lack of interactivity.
• New approaches to data cleaning emphasize increased interactivity.
• Potter’s Wheel, for example, is a publicly available data cleaning tool that
integrates discrepancy detection and transformation.
• Users gradually build a series of transformations by composing and debugging individual
transformations, one step at a time, on a spreadsheet-like interface.
• The transformations can be specified graphically or by providing examples.
• Results are shown immediately on the records that are visible on the screen.
• The user can choose to undo the transformations, so that transformations that introduced
additional errors can be “erased.”
• The tool automatically performs discrepancy checking in the background on the latest
transformed view of the data.
• Users can gradually develop and refine transformations as discrepancies are found, leading to
more effective and efficient data cleaning.
Potter’s Wheel Architecture
[Link]
• Another approach to increased interactivity in data cleaning is the development
of declarative languages for the specification of data transformation operators.
• Such work focuses on defining powerful extensions to SQL and algorithms that
enable users to express data cleaning specifications efficiently.