Boolean classification
I embed the two classes as y = 1
I use RERM to fit, with various loss functions and regularizers
I validate using Neyman-Pearson metric on test data, Efn + Efp
I is our relative distaste for mistaking a positive example
I for = 1, reduces to error rate
2
Loss functions
3
Loss functions for Boolean classification
I y can only take values 1 or 1, so to specify `, we only need to give two functions of y^:
I `(y;
^ 1) is how much y
^ irritates us when y = 1
I `(y;
^ 1) is how much y
^ irritates us when y = 1
I we will define ` via a penalty function p : R ! R
I `(y;
^ 1) = p(y
^)
I `(y;
^ 1) = p( y ^) = `( y;
^ 1)
I p(y^) should be small for y^ negative
I p(y^) should be larger y^ positive
I gives our relative dislike of mistaking y = 1
4
Square loss
3.0 3.0
2.5 `(y;
^ 1) 2.5 `(y;
^ 1)
2.0 2.0
1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
I `(y;
^ 1) = (1 + y
^)
2
, `(y;
^ 1) = `( y;
^ 1) = (1 y^)2
I doesn’t satisfy desired properties, e.g., `( 3; 1) should be very small, not large
I ERM is least squares problem, and so, easy to solve
5
Neyman-Pearson loss
I Neyman-Pearson loss is
y^ 0
I `
NP ( ^ y; 1) =
1
0 y^ < 0
y^ < 0
I `NP (y;
^ 1) = lNP (y;
^ 1) =
0 y^0
3 3
`NP (y;
^ 1) `NP (y;
^ 1)
2 2
1 1
0 0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
6
Neyman-Pearson loss
I it’s the same as our performance metric, which would seem good
I but it’s very hard to minimize L(), since it’s discontinuous, has zero derivative almost everywhere
I surprisingly, we get better performance using different loss functions, that are also easier to minimize
I if they’re convex, and the regularizer is convex, we can solve the RERM problem efficiently
7
Sigmoid loss
3.0 3.0
2.5 `(y;
^ 1) 2.5 `(y;
^ 1)
2.0 2.0
1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
1
I `(y;
^ 1) =
1 + e y^
, `(y;
^ 1) = `( y;
^ 1) =
1 + ey^
I differentiable approximation of Neyman-Pearson loss
I but not convex
8
Logistic loss
3.0 3.0
2.5 `(y;
^ 1) 2.5 `(y;
^ 1)
2.0 2.0
1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
I `(y;
^
^
y
1) = log(1 + e ), `(y;
^ 1) = `( y;
^ 1) = log(1 + e
^
y
)
I differentiable and convex approximation of Neyman-Pearson loss
9
Hinge loss
3.0 3.0
2.5 `(y;
^ 1) 2.5 `(y;
^ 1)
2.0 2.0
1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
I `(y;
^ 1) = (1 + y
^)+ , `(y;
^ 1) = `( y;
^ 1) = (1 y^)+
I another convex approximation of Neyman-Pearson loss
10
Hubristic loss
3.0 3.0
2.5 `(y;
^ 1) 2.5 `(y;
^ 1)
2.0 2.0
1.5 1.5
1.0 1.0
0.5 0.5
0.0 0.0
3 2 1 0 1 2 3 3 2 1 0 1 2 3
y^ y^
I define the hubristic loss (huber + logistic) as
8
>
<0 y^ < 1
`(y;
^ 1) =
>
(y
^ + 1)
2
1 y^ 0
:1 + 2y^ y^ > 0
I `(y;
^ 1) = `( y;
^ 1)
11
Boolean classifiers
12
Boolean classifiers with names
I least squares classifier uses square loss, square regularizer
I logistic regression uses logistic loss, any regularizer, as in, logistic regression with `1 regularizer
I support vector machine (SVM) uses hinge loss, square regularizer
13
Example
8 8
6 6
4 4
2 2
0 0
2 2
4 4
4 2 0 2 4 6 8 10 4 2 0 2 4 6 8 10
89 5 89 4
logistic loss: C = squared loss: C =
11 95 11 96
14
Support vector machine
3.0 4
2.5 hinge loss, `(^
y ; 1)
2.0
3
1.5
2
1.0
0.5 1
0.0
3 2 1 0 1 2 3 0
3.0
2.5 hinge loss, `(^
y; 1) 1
2.0
1.5 2
1.0
3
0.5
0.0 4
3 2 1 0 1 2 3 4 3 2 1 0 1 2 3 4
I decision boundary is T x = 0
I black lines show points where T x = 1
I what is the training risk here?
15
Example: Australian weather
I we have measurements of multiple attributes of weather at multiple locations in Australia
I over 10 years from 2007 to 2017
I 142,193 records
I given measurements from today, predict if it will rain tomorrow
I removing records with missing data leaves 112,925 records
I data from Australian weather stations, downloaded from
[Link]
16
Example: Australian weather
I numeric fields
I MinTemp, MaxTemp, Rainfall, WindGustSpeed, WindSpeed9am, WindSpeed3pm, Humidity9am,
Humidity3pm, Pressure9am, Pressure3pm, Temp9am, Temp3pm
I categorical fields
I location (44 possible locations)
I WindGustDir, WindDir9am, WindDir3pm (16 compass points)
I RainToday (yes or no)
I additional field: date
17
Some data
45
40 1030
max temperature
35 1020
3pm pressure
30
1010
25
20 1000
15
990
10
5 0 5 10 15 20 25 30 10 20 30 40
min temperature 3pm temperature
I here we look at a random 2% of the data, for a few features
I blue points indicate next day rainfall
18
Embedding
I for x = (u)
I embed 12 numeric fields via identity map
I embed 3 wind directions as one-hot (16 compass points)
I embed RainToday as f g
1; 1
I do not use date or location fields (did not improve validation performance)
I standardize
I add constant feature
I results in x 2 R62
I embed y = (v ) as f 1; 1 g where v is RainTomorrow
19
ERM
I use logistic loss function
^
y ^
y
`(y;
^ 1) = log(1 + e ); `(y;
^ 1) = log(1 + e )
I linear predictor y^ = T x
I and square regularization r() = k2: k22
20
ROC
0.7
0.6
0.5
I randomly split 80/20 into train/test sets
0.4
I test and train results very similar (test in red, train in blue)
I minimum probability of error = 16% Cfp =n 0.3
I rain frequency = 22%, so a predictor that always
predicts no rain will achieve 22% error 0.2
0.1
0.0
0.0 0.1 0.2 0.3
Cfn =n 21
Important features
j j
i 1
3
0 10 20 30 40 50 60
i
I important feature: Pressure9am - Pressure3pm (i = 10; 11)
I rapidly falling pressure indicates a storm is coming
I note 16 one-hot embedded values for WindDir9am (i = 30; : : : ; 45) all sum to one
I retraining with 6 features: MinTemp, MaxTemp, WindGustSpeed, Humidity3pm, Pressure9am, Pres-
sure3pm achieves 16.5% probability of error
22