Scribd
Upload
10 views3 pages

11 Data Preprocessing

Chapter 11 discusses the importance of data pre-processing in machine learning, emphasizing that raw data often requires preparation through techniques like input normalization, dimensionality reduction, and data cleaning. It illustrates the necessity of standardizing attributes with an example involving credit card approvals based on age and income. The chapter concludes with key takeaways about statistical learning, model complexity, and generalization capabilities.

Uploaded by

Emilia Lan Da Wang
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
10 views3 pages

11 Data Preprocessing

Chapter 11 discusses the importance of data pre-processing in machine learning, emphasizing that raw data often requires preparation through techniques like input normalization, dimensionality reduction, and data cleaning. It illustrates the necessity of standardizing attributes with an example involving credit card approvals based on age and income. The chapter concludes with key takeaways about statistical learning, model complexity, and generalization capabilities.

Uploaded by

Emilia Lan Da Wang
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Chapter 11 Data Pre-processing

 In many cases, data cannot be put “as is” into the learning machine
 Without “data-snooping”, there are a few procedures that usually prepare the data in the best way
for learning:
o Input pre-processing
 Input centering
 Input normalization
 Input whitening
o Dimensionality reduction/feature selection
o Data cleaning

1.1 Why do we need pre-processing?


Example:
The SLAS bank gave Mr. Good and Mr. Bad credit cards based on their age and income:
Mr. Good Mr. Bad

Age(YRS) 47 22

Income($) 35K 40K


Mr. Good paid off his credit card bill, where as Mr. Bad defaulted.
Now, Mr. Unknown, with coordinates
Mr. Unknown

Age(YRS) 21

Income($) 36K
Applies for credit.
Should the SLAS bank give him credit? Would the answer change if the income was measured
in k$?
Answer: The result depends on the algorithm, but… it is generally true that an attribute of

⟶DATA MUST BE STANDARDIZED


thousands of units would make the other attributes negligible!

1.2 Input Pre-processing


Goal: given input data X ∈ R N × d

[]
T
¿¿
T
X = ¿¿
⋮
T
¿¿

d
Each row is an input data vector x n ∈ R (not augmented with 1).
draw the line minimizing the sum of squared distances between predictions and points

Ideal objective:

y=f ( x )+ ε , E [ ε ] =0 5.1

Minimize w.r.t. h ( x ) :

E out ( h )=E [(h ( x )− y)2 ]=0 5.2

Where E is taken w.r.t. P ( x , y ) , x is unknown!

We resort to in-sample version of the error:


N
1
E¿ ( h ) = ∑ (h ( x n )− y n )2=0
N n=1
5.3

Main assumption: linearity


d
h ( x )=∑ wi xi=wT x 5.4
i=0

x 0=1 , x ∈ {1 } × R , w ∈ R
d d+ 1

Other assumptions:

homogeneity E [ e ] =0 5.5

homoscedasticity E [ e 2 ] =σ 2 (constant variance) 5.6

independence E [ e i e j ]=0 5.7

Define:

[ ]
x 01 x 11 … x d 1
x ⋱ ⋱ ⋮
X = 02
⋮ ⋱ ⋱ ⋮
x 0 N x 1 N … x dN
1.3 Take-home Lessons
 Learning is not feasible from a deterministic perspective
 It is indeed possible from a statistical point of view

STATISTICAL LEARNING
 Hoeffding’s inequality quantifies the capability of a model to perform well on unseen data
(GENERALIZATION)
 There are other ways to address this issue (same conditions)
 Complexity of the model is the key knob to tune the error

You might also like