DATA ANALYTICS
UE23CS342AA2
UNIT-1
Lecture 4: Data Preprocessing –Noisy
data and cleaning as a process
Gowri Srinivasa
Department of Computer Science and Engineering
Data Analytics
Unit 1
Lecture 4: Data Preprocessing - Noisy data and cleaning as a process
Slides collated by:
Nishanth M S, PESU-2023, Department of CSE, PESU
Slides excerpted from: Data Mining : Concepts and nishanthmsathish.23@[Link]
Techniques by Han, Kamber and Pei, 3rd Edition Harshitha Srikanth, PESU-2024, PES University
harshithasrikanth13@[Link]
Karthik Namboori, VII Sem, PESU, Department of CSE
namkarthik2003@[Link]
Gowri Srinivasa
With grateful thanks for contribution of slides to:
Department of Computer Science and Engineering Dr. Mamatha H R, Professor at the Department of CSE, PESU
DATA ANALYTICS
Noisy data
Noise is a random error or variance in a measured variable.
Data smoothening techniques to combat noise :
• Binning
▪ Sort the data and partition into bins(equal-width, equal-frequency, etc.)
▪ Smooth by bin means, bin medians, by bin boundaries etc.
▪ More on binning in further lectures.
• Regression - Data can be smoothened by fitting it to a regression model.
• Clustering - Outliers can be detected with the help of clustering and can be
removed to smoothen the data.
• Combined computer and human inspection – Computer detects suspicious
values and is validated by a human. Is useful when dealing with possible
outliers.
DATA ANALYTICS
Outliers
Outliers are data objects with characteristics
that are considerably different than most of the
other data objects in the data set.
Case 1 : Outliers are noise that interferes with
data analysis.
Case 2 : Outliers are the main goal of our
analysis. Examples :
• Credit card fraud
• Spam detection
• Intrusion detection
DATA ANALYTICS
Data Cleaning as a Process
• Data discrepancy detection
• Refer to the metadata of data to gain knowledge regarding its properties.
• Perform summary statistics for all attributes and discover the distributions,
dependencies , outliers and so on.
• Look for inconsistent representation of data. For example , make sure all dates are
following the same format , for instance , DD-MM-YYYY.
• Check for field overloading – practice of coupling two or more data elements to a
single field. It ensures efficient memory utilization.
• Check for uniqueness rule, consecutive rule and null rule.
• Uniqueness rule : Each value of the given attribute must be unique.
• Consecutive rule : There can’t be any missing values between lowest and highest
value for that attribute. All values must be unique. Example - cheque number.
• Null rule : Specifies how to record a null value. For example , use 0 for numeric
attribute and ‘?’ for nominal attribute.
DATA ANALYTICS
Data Cleaning as a Process
• Data discrepancy detection
• Use commercial tools that can aid in this step.
• Data scrubbing tools : Use simple domain knowledge (example, knowledge of
postal zip code and spell-check) to detect errors and make corrections.
• Data auditing tools : Find discrepancies by analyzing the data to discover rules
and relationships , and detect data that violates the discovered rules. For
example , it employs statistical analysis to find correlations or clustering to
detect outliers.
DATA ANALYTICS
Data Cleaning as a Process
• Data transformation
• Some data inconsistencies can be corrected manually but most errors require
data transformations.
• Data migration tools allow transformations to be specified.
• ETL (Extraction/Transformation/Loading) tools allow users to specify
transformations through a graphical user interface.
• Data transformations may introduce more discrepancies.
• The 2-step process of discrepancy detection and data transformation occurs
iteratively until no further anomalies are found.
• New approaches to data cleaning emphasize increased interactivity. Potter’s
wheel is a publicly available data cleaning tool that integrates both the steps.
DATA ANALYTICS
Test your understanding!
• Which of these is not a method to deal with noisy data?
a) Binning
b) Regression
c) Principal Component Analysis
d) Clustering
Solution
c) Principle Component Analysis
• Outliers need to be removed in every dataset , regardless of the problem
statement.
Solution
False
• Mean imputation can be done for which type of missing data?
Solution
MCAR
DATA ANALYTICS
Test your understanding!
• The statement “Most of the missing people from work are sickest people” denotes
what type of missingness?
MNAR
• Which type of missingness is called “non-ignorable”?
MNAR
Because the missing data mechanism itself has to be modelled as you deal with the
missing data. You have to include some model for why the data are missing and what the
likely values are.
DATA ANALYTICS
References
• Data Mining : Concepts and Techniques by Han, Kamber and Pei , The
Morgan Kaufmann Series in Data Management Systems ,3rd Edition
Chapter : 3.1-3.2
• [Link]
• [Link]
• [Link]
THANK YOU
Dr. Gowri Srinivasa
Professor, Department of Computer Science and
Engineering, PES University, Bengaluru
Email: gsrinivasa@[Link]