Data Representation, Data
Cleaning, Data Integration
Major Tasks in Data Preprocessing
• Data cleaning
• Fill in missing values, smooth noisy data, identify or remove outliers, and resolve
inconsistencies
• Data integration
• Integration of multiple databases, data cubes, or files
• Data transformation
• Normalization and aggregation
• Data reduction
• Obtains reduced representation in volume but produces the same or similar analytical results
• Data discretization
• Part of data reduction but with particular importance, especially for numerical data
Data Cleaning
• Data cleaning is the process of detecting and correcting errors,
inconsistencies, and inaccuracies in data so that it becomes reliable
and trustworthy for analysis.
• Real-world data is often:
• Incomplete
• Incorrect
• Inconsistent
• Noisy
• Data cleaning fixes these issues.
Missing Data
• Data is not always available
• E.g., many tuples have no recorded value for several attributes, such
as customer income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• not register history or changes of the data
• Missing data may need to be inferred.
How to handle missing data?
• Ignore the tuple: usually done when class label is missing (assuming the tasks in classification—
not effective when the percentage of missing values per attribute varies considerably.
• Fill in the missing value manually: tedious + infeasible?
• Use a global constant to fill in the missing value: e.g., “unknown”, a new class?!
• Use the attribute mean to fill in the missing value
• Use the attribute mean for all samples belonging to the same class to fill in the missing value:
smarter
• Use the most probable value to fill in the missing value: inference-based such as Bayesian formula
or decision tree
Noisy Data
• Noise: random error or variance in a measured variable
• Incorrect attribute values may due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
• Inconsistency in naming convention
• Other data problems which require data cleaning
• duplicate records
• incomplete data
• inconsistent data
How to handle Noisy data?
• Binning method:
• first sort data and partition into (equi-depth) bins
• then one can smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
• Clustering
• detect and remove outliers
• Combined computer and human inspection
• detect suspicious values and check by human
• Regression
• smooth by fitting the data into regression functions
Binning Methods for Data Smoothing
* Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
* Partition into (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Cluster Analysis
Regression
y
Y1
Y1’ y=x+1
X1 x
Common Data Cleaning Tasks
Handling Missing Values
• Remove rows/columns
• Fill with mean, median, mode
• Use domain logic
• Example: Missing age values replaced with average age.
Removing Duplicates
• Same record appears multiple times
• Example: Same customer listed twice.
Fixing Incorrect Values
• Impossible or wrong entries
• Example:
• Age = −5
• Temperature = 500°C
Continued…
Standardizing Data
• Different formats for the same thing
• Example: “Male”, “M”, “male” → “Male”
Correcting Data Types
• Numbers stored as strings
• Example:"85" → 85
Handling Outliers
• Values far from the normal range
• Example: Salary = ₹1 vs ₹1 crore
Data Integration
• Data integration is the process of combining data from multiple
sources into a single, unified view so it can be analyzed consistently.
• In real systems, data lives in silos:
• Databases
• CSV / Excel files
• APIs
• Logs
• Sensors
• Web platforms
• Each source captures part of the story. Integration gives the complete
picture
Continued..
• Example
• Student information comes from:
• Academic database → marks
• Attendance system → attendance
• Hostel system → residence
• LMS platform → login activity
Roll No Marks Attendance Hostel Logins
101 85 92% Yes 45