0% found this document useful (0 votes)
5 views60 pages

Data Processing

Chapter 2 discusses data analytics and AI, focusing on data measurement, preprocessing, and quality. It covers statistical measures of central tendency and dispersion, data cleaning techniques, and data transformation methods. Key topics include handling missing values, noise reduction, and data integration for effective analysis.

Uploaded by

dung.dinh2352206
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
5 views60 pages

Data Processing

Chapter 2 discusses data analytics and AI, focusing on data measurement, preprocessing, and quality. It covers statistical measures of central tendency and dispersion, data cleaning techniques, and data transformation methods. Key topics include handling missing values, noise reduction, and data integration for effective analysis.

Uploaded by

dung.dinh2352206
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Chapter 2

Data for Data analytics and AI

Ph.D. Nguyen Duc Duy

1
Section 1
Data, measurements, and data preprocessing

2
2.2 Statistics of data
2.2.1 Measuring the central tendency

▪ The most common and effective numeric measure of the “center” of a set of data
is the (arithmetic) mean.
• Mean
• Weighted arithmetic mean

Duc Duy Nguyen [Link]@[Link] 3


2.2 Statistics of data
2.2.1 Measuring the central tendency

▪ Trimmed mean
• which is the mean obtained after chopping off values at the high and low
extremes.
• For example, we can sort the values observed for salary and remove the top and
bottom 2% before computing the mean. We should avoid trimming too large a
portion (such as 20%) at both ends, as this can result in the loss of valuable
information.

Duc Duy Nguyen [Link]@[Link] 4


2.2 Statistics of data
2.2.1 Measuring the central tendency

▪ Skewed (asymmetric) data:


• Median
• Mode
➢ Mode for a set of data is the value that occurs most frequently compared to all
neighboring values in the set.
➢ Can be determined for qualitative and quantitative attributes.
➢ Data sets with one, two, or three modes are respectively called unimodal,
bimodal, and trimodal. In general, a data set with two or more modes is
multimodal.

Duc Duy Nguyen [Link]@[Link] 5


2.2 Data Preparation
2.2.1 Measuring the central tendency

▪ Midrange: It is the average of the largest and smallest values in the set.

Duc Duy Nguyen [Link]@[Link] 6


2.2 Statistics of data
2.2.2 Measuring the dispersion of data
Range, quartiles, and interquartile range
▪ The range of the set is the difference between the largest (max()) and smallest
(min()) values.
▪ Quantiles are points taken at regular intervals of a data distribution, dividing it into
essentially equal-size consecutive sets.
▪ Interquartile range (IQR) : The distance between the first and third quartiles is a
simple measure of spread that gives the range covered by the middle half of the data.
The quartiles are the three values that split the sorted data set into four equal parts

Duc Duy Nguyen [Link]@[Link] 7


2.2 Statistics of data
2.2.2 Measuring the dispersion of data
Five-number summary, boxplots, and outliers

▪ The five-number summary of a distribution consists of the median (Q2), the


quartiles Q1 and Q3, and the smallest and largest individual observations, written
in the order of Minimum, Q1, Median, Q3, Maximum.
▪ Variance and standard deviation
• The standard deviation, σ, of the observations is the square root of the variance,
σ2.

Duc Duy Nguyen [Link]@[Link] 8


2.2 Statistics of data
2.2.2 Measuring the dispersion of data
Five-number summary, boxplots, and outliers

▪ Boxplots and outliers

Duc Duy Nguyen [Link]@[Link] 9


2.2 Statistics of data
2.2.3 Covariance and correlation analysis

▪ Covariance of numeric data


▪ Correlation coefficient for numeric data -
Pearson’s product moment coefficient.
▪ χ2 correlation test for nominal data - Chi-
square test

Duc Duy Nguyen [Link]@[Link] 10


2.2 Statistics of data
2.2.3 Covariance and correlation analysis
Visually Evaluating Correlation

Duc Duy Nguyen [Link]@[Link] 11


2.2 Statistics of data
2.2.4 Graphic displays of basic statistics of data
Quantile plot

Duc Duy Nguyen [Link]@[Link] 12


2.2 Statistics of data
2.2.4 Graphic displays of basic statistics of data

▪ Histogram
▪ Scatter plots and data correlation

Duc Duy Nguyen [Link]@[Link] 13


2.3 Data quality, data cleaning, and data integration
2.3.1 Data quality measures

▪ Accuracy: Chính xác


▪ Completeness: Đầy đủ
▪ Consistency: Nhất quán
▪ Timeliness: Kịp thời
▪ Believability: Tin cậy
▪ Interpretability: Có thể diễn tả được
▪ (Relevance)

Duc Duy Nguyen [Link]@[Link] 14


2.3 Data quality, data cleaning, and data integration
2.3.1 Data quality measures

Big data primarily refers to data sets that


are too large or complex to be dealt
with by traditional data-processing
application software.
Data with many entries (rows) offer
greater statistical power, while data with
higher complexity (more attributes or
columns) may lead to a higher false
discovery rate.

Duc Duy Nguyen [Link]@[Link] 15


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning

▪ Data cleaning (or data cleansing) routines attempt to fill in missing values,
smooth out noise while identifying outliers, and correct inconsistencies in the
data.
• Missing values
• Noisy data
➢ e.g., Salary=“-10”

Duc Duy Nguyen [Link]@[Link] 16


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning - Why Is Data Dirty?

▪ Incomplete data comes from


• n/a data value when collected
• different consideration between the time when the data was collected and when it is analyzed.
• human/hardware/software problems
▪ Noisy data comes from the process of data
• collection
• entry
• transmission
▪ Inconsistent data comes from
• different data sources
• functional dependency violation

Duc Duy Nguyen [Link]@[Link] 17


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Missing values
1. Ignore the tuple: This is usually done when the class label is missing.
2. Fill in the missing value manually: In general, this approach is time
consuming and may not be feasible given a large data set with many missing
values.
3. Use a global constant to fill in the missing value: Replace all missing
attribute values by the same constant such as a label like “Unknown” or −∞.
4. Use a measure of central tendency for the attribute (e.g., the mean or
median) to fill in the missing value.
• For normal (symmetric) data distributions, the mean can be used, whereas
skewed data distribution should employ the median

Duc Duy Nguyen [Link]@[Link] 18


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Missing values
5. Use the attribute mean or median for all samples belonging to the same
class as the given tuple.
6. Use the most probable value to fill in the missing value: This may be
determined with regression, inference-based tools using a Bayesian formalism or
decision tree induction.
Notes:
• Methods 3 through 6 bias the data—the filled-in value may not be correct.
• Method 6, however, is a popular strategy. In comparison to the other methods, it
uses the most information from the present data to predict missing values.

Duc Duy Nguyen [Link]@[Link] 19


2.3 Data quality, data cleaning, and data integration
2.3.3 Data cleaning
Missing values
When applying for a credit card, candidates may be asked to supply their Passport
number. Candidates who do not have a passport may naturally leave this field
blank.
“Missing values imply an error in the data” is correct or not?

Duc Duy Nguyen [Link]@[Link] 20


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Noisy data
▪ “What is noise?” Noise is a random error or variance in a measured variable.
▪ Incorrect attribute values may be due to
• faulty data collection instruments
• data entry problems
• data transmission problems
• technology limitation
▪ Inconsistency in naming convention
• Other data problems which require data cleaning
• duplicate records
• incomplete data
• inconsistent data
Duc Duy Nguyen [Link]@[Link] 21
2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Noisy data
▪ Data smoothing techniques:
• Binning: Binning methods smooth a
sorted data value by consulting its
“neighborhood,” that is, the values
around it.
➢ Smoothing by bin means
➢ Smoothing by bin medians
➢ Smoothing by bin boundaries
• the minimum and maximum values in a
given bin are identified as the bin
boundaries

Duc Duy Nguyen [Link]@[Link] 22


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Noisy data
▪ Data smoothing techniques (Cont.)
• Regression
➢ Linear regression
➢ Multiple linear regression
• Outlier analysis
• Combined computer and human
inspection
➢ Detect suspicious values and check by
human (e.g., deal with possible outliers) A 2-D customer data plot with respect to customer
locations in a city, showing three data clusters. Outliers
may be detected as values that fall outside of the cluster
sets.
Duc Duy Nguyen [Link]@[Link] 23
2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Data Cleaning as a Process
▪ Data discrepancy detection
• Use metadata (e.g., domain, range, dependency, distribution)
• Check field overloading
• Check uniqueness rule, consecutive rule and null rule
• Use commercial tools
➢ Data scrubbing: use simple domain knowledge (e.g., postal code, spell-check) to
detect errors and make corrections
➢ Data auditing: by analyzing data to discover rules and relationship to detect
violators (e.g., correlation and clustering to find outliers)

Duc Duy Nguyen [Link]@[Link] 24


2.3 Data quality, data cleaning, and data integration
2.3.2 Data cleaning
Data Cleaning as a Process
▪ Data migration and integration
• Data migration tools: allow transformations to be specified
• ETL (Extraction/Transformation/Loading) tools: allow users to specify
transformations through a graphical user interface
• Integration of the two processes
• Iterative and interactive (e.g., Potter’s Wheels)

Duc Duy Nguyen [Link]@[Link] 25


2.3 Data quality, data cleaning, and data integration
2.3.3 Data Integration
▪ Data integration: Combines data from multiple sources into a coherent store;
Integrate metadata from different sources
▪ Problem:
• Entity identification problem
➢ Identify real world entities from multiple data sources
• Redundancy and correlation analysis
• Tuple duplication
➢ Duplication should also be detected at the tuple level (e.g., where there are two or more
identical tuples for a given unique data entry).
• Detecting and resolving data value conflicts
➢ For the same real world entity, attribute values from different sources are different
➢ Possible reasons: different representations, different scales, e.g., metric vs. British units

Duc Duy Nguyen [Link]@[Link] 26


Duc Duy Nguyen [Link]@[Link] 27
2.4 Data transformation
▪ In data transformation, the data are transformed or consolidated into forms appropriate for
mining.
▪ Methods
• Smoothing: Remove noise from data
• Attribute/feature construction
➢ New attributes constructed from the given ones
• Aggregation: Summarization, data cube construction
• Normalization: Scaled to fall within a smaller, specified range
➢ min-max normalization
➢ z-score normalization
➢ normalization by decimal scaling
• Discretization: Concept hierarchy climbing

Duc Duy Nguyen [Link]@[Link] 28


2.4 Data transformation
Normalization

▪ Min-max normalization: to [new_minA, new_maxA]


v − minA
v' = (new _ maxA − new _ minA) + new _ minA
maxA − minA
▪ Z-score normalization (μ: mean, σ: standard deviation):
v − A
v' =
 A

▪ Normalization by decimal scaling


v
v'= j Where j is the smallest integer such that Max(|ν’|) < 1
10
Duc Duy Nguyen [Link]@[Link] 29
2.4 Data transformation
Normalization

Duc Duy Nguyen [Link]@[Link] 30


2.4 Data transformation
Discretization
▪ Three types of attributes
• Nominal—values from an unordered set, e.g., color, profession
• Ordinal—values from an ordered set, e.g., military or academic rank
• Numeric—real numbers, e.g., integer or real numbers
▪ Discretization: Divide the range of a continuous attribute into intervals
• Interval labels can then be used to replace actual data values
• Reduce data size by discretization
• Supervised vs. unsupervised
• Split (top-down) vs. merge (bottom-up)
• Discretization can be performed recursively on an attribute
• Prepare for further analysis, e.g., classification

Duc Duy Nguyen [Link]@[Link] 31


2.4 Data transformation
Discretization

▪ A concept hierarchy for the attribute price, where an interval ($X . . .$Y] denotes the range from $X
(exclusive) to $Y (inclusive).

Duc Duy Nguyen [Link]@[Link] 32


2.4 Data transformation
Data Discretization Methods

▪ Typical methods:
• Binning
➢ Top-down split, unsupervised
• Histogram analysis
➢ Top-down split, unsupervised
• Clustering analysis (unsupervised, top-down split or bottom-up
merge)
• Decision-tree analysis (supervised, top-down split)
• Correlation (e.g., 2) analysis (unsupervised, bottom-up merge)

Duc Duy Nguyen [Link]@[Link] 33


2.4 Data transformation
Discretization by histogram analysis
▪ Equal-width (distance) partitioning
• Divides the range into N intervals of equal size: uniform grid
• if A and B are the lowest and highest values of the attribute, the width of intervals will be:
W = (B –A)/N.
• The most straightforward, but outliers may dominate presentation
• Skewed data is not handled well
▪ Equal-depth (frequency) partitioning
• Divides the range into N intervals, each containing approximately same number of
samples
• Good data scaling
• Managing categorical attributes can be tricky

Duc Duy Nguyen [Link]@[Link] 34


2.4 Data transformation
Discretization by histogram analysis

Example
The following data are a list of prices for
commonly sold items in the company
(rounded to the nearest dollar). The numbers
have been sorted: 1, 1, 5, 5, 5, 5, 5, 8, 8, 10,
10, 10, 10, 12, 14, 14, 14, 15, 15, 15, 15, 15,
15, 18, 18, 18, 18, 18, 18, 18, 18, 20, 20, 20,
20, 20, 20, 20, 21, 21, 21, 21, 25, 25, 25, 25,
25, 28, 28, 30, 30, 30

Duc Duy Nguyen [Link]@[Link] 35


2.4 Data transformation
Discretization by histogram analysis

Example
An equal-width histogram for price, where
values are aggregated so that each bucket
has a uniform width of $10.

Duc Duy Nguyen [Link]@[Link] 36


2.4 Data transformation
Binning Methods for Data Smoothing
❑ Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
Duc Duy Nguyen [Link]@[Link] 37
2.4 Data transformation
Discretization Without Using Class Labels (Binning vs. Clustering)

Data Equal interval width (binning)

Equal frequency (binning)


Duc Duy K-means clustering leads to better results
Nguyen [Link]@[Link] 38
2.4 Data transformation
Discretization by Classification & Correlation Analysis

▪ Classification (e.g., decision tree analysis)

• Supervised: Given class labels, e.g., cancerous vs. benign

• Using entropy to determine split point (discretization point)

• Top-down, recursive split

▪ Correlation analysis (e.g., Chi-merge: χ2-based discretization)

• Supervised: use class information

• Bottom-up merge: find the best neighboring intervals (those having similar distributions of
classes, i.e., low χ2 values) to merge

• Merge performed recursively, until a predefined stopping condition

Duc Duy Nguyen [Link]@[Link] 39


2.4 Data transformation
Concept Hierarchy Generation

▪ Concept hierarchy organizes concepts (i.e., attribute values) hierarchically and is usually
associated with each dimension in a data warehouse
▪ Concept hierarchies facilitate drilling and rolling in data warehouses to view data in
multiple granularity
▪ Concept hierarchy formation: Recursively reduce the data by collecting and replacing low
level concepts (such as numeric values for age) by higher level concepts (such as youth,
adult, or senior)
▪ Concept hierarchies can be explicitly specified by domain experts and/or data warehouse
designers
▪ Concept hierarchy can be automatically formed for both numeric and nominal data. For
numeric data, use discretization methods shown.

Duc Duy Nguyen [Link]@[Link] 40


2.4 Data transformation
Concept Hierarchy Generation for Nominal Data

▪ Specification of a partial/total ordering of attributes explicitly at the schema


level by users or experts
• street < city < state < country
▪ Specification of a hierarchy for a set of values by explicit data grouping
• {Urbana, Champaign, Chicago} < Illinois
▪ Specification of only a partial set of attributes
• E.g., only street < city, not others
▪ Automatic generation of hierarchies (or attribute levels) by the analysis of the
number of distinct values
• E.g., for a set of attributes: {street, city, state, country}

Duc Duy Nguyen [Link]@[Link] 41


2.4 Data transformation
Automatic Concept Hierarchy Generation

▪ Some hierarchies can be automatically


generated based on the analysis of the
number of distinct values per attribute in
the data set
• The attribute with the most distinct values
is placed at the lowest level of the
hierarchy.
• Exceptions, e.g., weekday, month, quarter,
year

Duc Duy Nguyen [Link]@[Link] 42


2.4 Data transformation
Sampling
▪ Sampling: obtaining a small sample s to represent the whole data set N
▪ Allow a mining algorithm to run in complexity that is potentially sub-linear to the
size of the data
▪ Key principle: Choose a
representative subset of the data
• Simple random sampling may have very
poor performance in the presence of skew
• Develop adaptive sampling methods, e.g.,
stratified sampling.
▪ Note: Sampling may not reduce
database I/Os (page at a time)

Duc Duy Nguyen [Link]@[Link] 43


2.4 Data transformation
Types of Sampling

▪ Simple random sampling


• There is an equal probability of selecting any particular item
▪ Sampling without replacement
• Once an object is selected, it is removed from the population
▪ Sampling with replacement
• A selected object is not removed from the population
▪ Stratified sampling:
• Partition the data set, and draw samples from each partition (proportionally, i.e.,
approximately the same percentage of the data)
• Used in conjunction with skewed data

Duc Duy Nguyen [Link]@[Link] 44


2.4 Data transformation
Sampling: With or without Replacement

Raw Data
Duc Duy Nguyen [Link]@[Link] 45
2.4 Data transformation
Sampling: Cluster or Stratified Sampling

Raw Data Cluster/Stratified Sample

Duc Duy Nguyen [Link]@[Link] 46


2.5 Dimensionality reduction
Data Reduction Strategies
▪ Data reduction: Obtain a reduced representation of the data set that is much smaller in
volume but yet produces the same (or almost the same) analytical results
▪ Why data reduction? — A database/data warehouse may store terabytes of data. Complex
data analysis may take a very long time to run on the complete data set.
▪ Data reduction strategies
• Dimensionality reduction, e.g., remove unimportant attributes
➢ Wavelet transforms
➢ Principal Components Analysis (PCA)
➢ Feature subset selection, feature creation
• Numerosity reduction (some simply call it: Data Reduction)
➢ Regression and Log-Linear Models
➢ Histograms, clustering, sampling
➢ Data cube aggregation
• Data compression
Duc Duy Nguyen [Link]@[Link] 47
2.5 Dimensionality reduction
Data Reduction Strategies
- Dimensionality reduction (1)
▪ Curse of dimensionality: “Lời nguyền của số chiều”
• When dimensionality increases, data becomes increasingly sparse
• Density and distance between points, which is critical to clustering, outlier analysis, becomes
less meaningful
• The possible combinations of subspaces will grow exponentially
▪ Dimensionality reduction: Giảm chiều dữ liệu
• Avoid the curse of dimensionality
• Help eliminate irrelevant features and reduce noise
• Reduce time and space required in data mining
• Allow easier visualization

Duc Duy Nguyen [Link]@[Link] 48


2.5 Dimensionality reduction
Data Reduction Strategies
Principal Component Analysis (PCA)
▪ Find a projection that captures the largest amount of variation in data
▪ The original data are projected onto a much smaller space, resulting in dimensionality reduction. We
find the eigenvectors of the covariance matrix, and these eigenvectors define the new space

Duc Duy Nguyen [Link]@[Link] 49


2.5 Dimensionality reduction
Data Reduction Strategies
Principal Component Analysis (PCA)
▪ Given N data vectors from n-dimensions, find k ≤ n orthogonal vectors (principal
components) that can be best used to represent data
• Normalize input data: Each attribute falls within the same range
• Compute k orthonormal (unit) vectors, i.e., principal components
• Each input data (vector) is a linear combination of the k principal component vectors
• The principal components are sorted in order of decreasing “significance” or strength
• Since the components are sorted, the size of the data can be reduced by eliminating
the weak components, i.e., those with low variance (i.e., using the strongest principal
components, it is possible to reconstruct a good approximation of the original data)
▪ Works for numeric data only

Duc Duy Nguyen [Link]@[Link] 50


Dimensionality Reduction: PCA

Duc Duy Nguyen [Link]@[Link]


2.5 Dimensionality reduction
Data Reduction Strategies
Heuristic Search in Attribute Selection
▪ Another way to reduce dimensionality of data
▪ Redundant attributes
• Duplicate much or all of the information contained in one or more other
attributes
• E.g., purchase price of a product and the amount of sales tax paid
▪ Irrelevant attributes
• Contain no information that is useful for the data mining task at hand
• E.g., students' ID is often irrelevant to the task of predicting students'
GPA
Duc Duy Nguyen [Link]@[Link] 52
2.5 Dimensionality reduction
Data Reduction Strategies
Attribute Subset Selection
▪ There are 2d possible attribute combinations of d attributes
▪ Typical heuristic attribute selection methods:
• Best single attribute under the attribute independence assumption: choose
by significance tests
• Best step-wise feature selection:
➢ The best single-attribute is picked first
➢ Then next best attribute condition to the first, ...
• Step-wise attribute elimination:
➢ Repeatedly eliminate the worst attribute
• Best combined attribute selection and elimination
• Optimal branch and bound:
➢ Use attribute elimination and backtracking

Duc Duy Nguyen [Link]@[Link] 53


2.5 Dimensionality reduction
Data Reduction Strategies
Attribute Subset Selection

Duc Duy Nguyen


Section [Link]@[Link]
7.1 Feature selection and engineering 54
2.5 Dimensionality reduction
Data Reduction Strategies
Attribute Creation (Feature Generation)
▪ Create new attributes (features) that can capture the important information
in a data set more effectively than the original ones
▪ Three general methodologies
• Attribute extraction
➢ Domain-specific
• Mapping data to new space (see: data reduction)
➢ E.g., Fourier transformation, wavelet transformation, manifold approaches (not
covered)
• Attribute construction
➢ Combining features (see: discriminative frequent patterns in Chapter 7)
➢ Data discretization

Duc Duy Nguyen [Link]@[Link] 55


2.5 Dimensionality reduction
Data Reduction Strategies
Data Reduction 2: Numerosity Reduction
▪ Reduce data volume by choosing alternative, smaller forms of data
representation
▪ Parametric methods (e.g., regression) – Phương pháp tham số
• Assume the data fits some model, estimate model parameters, store only the
parameters, and discard the data (except possible outliers)
• Ex.: Log-linear models—obtain value at a point in m-D space as the product on
appropriate marginal subspaces
▪ Non-parametric methods – Phương pháp phi tham số
• Do not assume models
• Major families: histograms, clustering, sampling, …

Duc Duy Nguyen [Link]@[Link] 56


2.5 Dimensionality reduction
Data Reduction Strategies
Parametric Data Reduction: Regression and Log-Linear Models
▪ Linear regression –Hồi quy tuyến tính
• Data modeled to fit a straight line
• Often uses the least-square method to fit the line
▪ Multiple regression- Hồi quy đa biến
• Allows a response variable Y to be modeled as a linear function of
multidimensional feature vector
▪ Log-linear model – Mô hình Log- tuyến tính
• Approximates discrete multidimensional probability distributions

Duc Duy Nguyen [Link]@[Link] 57


2.5 Dimensionality reduction
Data Reduction Strategies
Regression Analysis
▪ Regression analysis: A collective name for techniques for the
modeling and analysis of numerical data consisting of values of a Y1
dependent variable (also called response variable or
measurement) and of one or more independent variables (aka.
Y1’
explanatory variables or predictors) y=x+1
▪ The parameters are estimated so as to give a "best fit" of the
data
X1 x
▪ Most commonly the best fit is evaluated by using the least
squares method, but other criteria have also been used
▪ Used for prediction (including forecasting of time-series data), inference, hypothesis testing,
and modeling of causal relationships
Duc Duy Nguyen [Link]@[Link] 58
2.5 Dimensionality reduction
Data Reduction Strategies
Regress Analysis and Log-Linear Models
▪ Linear regression: Y = w X + b
• Two regression coefficients, w and b, specify the line and are to be estimated by using
the data at hand
• Using the least squares criterion to the known values of Y1, Y2, …, X1, X2, ….
▪ Multiple regression: Y = b0 + b1 X1 + b2 X2
• Many nonlinear functions can be transformed into the above
▪ Log-linear models:
• Approximate discrete multidimensional probability distributions
• Estimate the probability of each point (tuple) in a multi-dimensional space for a set of
discretized attributes, based on a smaller subset of dimensional combinations
• Useful for dimensionality reduction and data smoothing

Duc Duy Nguyen [Link]@[Link] 59


Reading books

Han, J., Kamber, M., & Pei, J. (2022). Data mining concepts and techniques
fouth edition. University of Illinois at Urbana-Champaign Micheline Kamber
Jian Pei Simon Fraser University.

Duc Duy Nguyen [Link]@[Link] 60

You might also like