Scribd
Upload
4 views8 pages

Data Cleaning Techniques Explained

Chapter 7 discusses the use of transformation techniques to clean data, focusing on addressing missing (NULL) and invalid values. It outlines methods for handling missing values, including filtering out records or using data imputation strategies, and methods for correcting invalid values through calculation or marking them as invalid. The chapter emphasizes the importance of data standardization to improve dataset quality, particularly in cases of misspellings and ambiguous entries.

Uploaded by

Bala Murali
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
4 views8 pages

Data Cleaning Techniques Explained

Chapter 7 discusses the use of transformation techniques to clean data, focusing on addressing missing (NULL) and invalid values. It outlines methods for handling missing values, including filtering out records or using data imputation strategies, and methods for correcting invalid values through calculation or marking them as invalid. The chapter emphasizes the importance of data standardization to improve dataset quality, particularly in cases of misspellings and ambiguous entries.

Uploaded by

Bala Murali
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd

Chapter 7.

Using
Transformation
to Clean Data
Using Transformation to Clean Data
• Cleaning predominately involves manipulating individual field values
within records.
• The most common variants of cleaning involve addressing missing (or
NULL) values and addressing invalid values.
Addressing Missing/NULL Values
• There are two basic approaches to addressing missing/null values.

• Method 1 -> you can filter out records with missing or NULL fields.

• Method 2 -> replace missing or NULL values - data imputation.

• filling in missing or NULL values might utilize many different


strategies.

• Strategy1 -> inserting the average or median value.


Addressing Missing/NULL Values
• Strategy2 -> generate values from similar records; Eg: from similar
customers or similar transactions.

• Strategy3 -> if the data has strong ordering (because it is a time-series


dataset, for example), you might be able to fill in missing values by
using the last valid value.
Addressing Invalid Values
• because they are inconsistent with other fields (e.g., a customer age
compared with their data of birth).

• ambiguous (e.g., twodigit years or abbreviations like “CT”—is that


Connecticut or Court?)

• Two ways to deal with it…

• Method1 ->the correct or consistent value for the field can be


calculated and used to overwrite the original value in the dataset.
Addressing Invalid Values
• Method2 - > simply mark values as invalid.

• conduct two parallel analyses, one that includes the invalid values and
one that excludes them, providing insight into the impact that invalid
data is having on your insights

• complex variety of fixing invalid values involves data standardization.

• Eg: every customer represented in that dataset is known to reside in


the United States.
• A reasonable validity check - fall into one of the known US states.

• Suppose, however, that there are misspellings: “Californa,” Westvirginia,”


and “Dakota.”
• Standardizing these field values to a fixed library of valid values is a good
way to improve dataset quality.

• common method involves editing distance around misspelling; that is,


strings that are similar, like “Californa” and “California,” should be treated
as the same entity and converted to the same spelling.
• More specific standardization techniques - rely on domain knowledge.
• Eg: Is “Dakota” supposed to be “North Dakota” or “South Dakota”?

• If we have a ZIP code in another field of the record, perhaps we can


use a mapping of ZIP codes to states to make this determination.

You might also like