Why talk about data preprocessing?
Data characteristic are unknown. The meta-data you are given are unknown
to you. Data profiling 数据分析
Poor data quality Data wrangling
Too many data Data Transformation/Reduction
1. Data Profiling how to open it and know the format of the data
Data profiling: the process of diagnosing a new or otherwise unknown dataset for its
⽅式
access modalities and data space characteristics.
Determine access modalities 可以检索的频率/数量/详细数据
How often, how much, how detailed data can be retrieved.
Examples of access modalities: GDPR
- Different character encodings
- Different number formats/Date formats/Time formats
- Different time zones/standards
- Different coordinate formats
Determine space characteristics
Determine what you are you looking at and whether it is plausible
Examples of characteristics:
- Data formatting (for special data types like IP address, license plates, CPR#,
etc.)
- Value ranges
- Units (Celsius -273.15, Kelvin 0)
- Placeholders (When a data point is missing)
- Consistency
- Topology of data structure 数据结构拓扑
Make use of known value ranges – Sensible value ranges as simple guards
against “bad data”.
e.g.
’
Notice: The standard you may want to support may not be the standard that your
client has or in which the data actually comes.
Issues with “standards”
2. Data Wrangling
Data Wrangling: the process of making any raw dataset useful by identifying and
treating missing values, duplicate and possibly contradicting entries, formatting
issues, and other problems of data quality 重复且可能互相⽭盾的条⽬
Examples
Missing values: imputation vs. amputation
Duplicate entries: de-duplication by deleting or merging
Formatting issues: file format, version, interpretation, implementation,
validation
Data quality: uncertainty, accuracy
Handling missing value
1 and 2 are problematic, don't show something not in the data. 3 is very expressive,
but makes it hard to see the trend. 4 is the best.
Different ways of imputation:
Imputation: when a value of some data point is missing, find a plausible value to
fill it. imputation: imputing missing values means replacing
missing values with some meaningful data in a dataset
Different ways of amputation and Imputation vs. Amputation
Amputation: when a value of some data point is missing, delete data points or
attributes related to it.
For an attribute, see if the missing data <5% or >40%
逐⾏
逐列
Missingness map:
Missingness maps are is a helpful tool to look at missing values and help you to
decide imputation or amputation.
明确的，显式
Explicit encoding: Imputation or amputation, we mark the missing values
(e.g., use a different color) to tell viewers they are missing and we have dealt
with them.
De-duplication
How to diagnose duplication
Deal with it: deleting or merging
Data quality uncertainty: uncertainty data maybe introduced when generating the data
When you do some visualization, you can use most prominent channel to show the
measured data and use another channel to show the uncertainty.
3. Data Transformation/Reduction
Sampling: Data reduction - sampling 数据缩减
Random sample
If the sampling is truly random, the sampled set should still exhibit more or less
the same properties as the overall dataset.
分层
Stratified Sampling
Random sampling proportionally across strata/sub-groups
配额抽样 Quota Sampling (manually set the quotas)
non-random sampling proportionally across strata/sub-groups
Example: sampling 1000 AU students (500 male/500 female), proportionally
across faculties
Sampling spatial data: one way to do it is Z-order sampling.
Subdivide a part into 4 parts, and connect them as Z. If a subregion still has a lot
of points we subdivide it again. After subdivision, for every nth region we take 1/n
points. This can preserve spatial density distribution.
Improvement:
Not only simply “pick” an item for every nth region, but pick a presentative value
(e.g. median). This can preserve spatial and attribute distribution.
Hierarchical Clustering: 层次聚类
Types of hierarchical clustering:
聚类的⽅向性
分裂的
凝聚的