Below is a simple synthetic dataset for disease prediction.
In this dataset, we have features related to
patient’s health, and the target variable is whether the patient has a particular disease or not (binary
classification: ‘no’ for no disease, ‘yes’ for disease).
Blood High Heart sample
Age smoker Disease
Pressure Rate weight
1 52 normal no no no 0.1
2 62 high yes no yes 0.1
3 35 normal no yes no 0.1
4 50 normal no yes yes 0.1
5 68 high yes no yes 0.1
6 42 normal no no no 0.1
7 55 normal yes no yes 0.1
8 60 high no no yes 0.1
9 38 normal no yes no 0.1
10 70 high yes no yes 0.1
Assume that we would like to build an AdaBoost model using stumps (trees with one root and
two leaves) as the weak learners. Please answer the following questions.
First, we assign sample weight by 1 / number of samples = 1/10
Blood High Heart sample
Age smoker Disease
Pressure Rate weight
1 52 normal no no no 0.1
2 62 high yes no yes 0.1 Age > 53.2
3 35 normal no yes no 0.1
4 50 normal no yes yes 0.1
yes no
5 68 high yes no yes 0.1
6 42 normal no no no 0.1
Disease No Disease
7 55 normal yes no yes 0.1
correct (5) incorrect (0) correct (4) incorrect (1)
8 60 high no no yes 0.1
9 38 normal no yes no 0.1
10 70 high yes no yes 0.1
Q1: Build the first stump using the Gini index.
There are many ways to split continuous features. Here, let's
split 'Age' using the simple method of taking the median
among all ages.
53.2 )
Blood Pressure
High Normal
Disease No Disease
correct (4) incorrect (0) correct (4) incorrect (2)
Smoker
yes no
Disease No Disease
correct (4) incorrect (0) correct (4) incorrect (2)
High Heart Rate
yes no
Disease No Disease
correct (1) incorrect (2) correct (2) incorrect (5)
We select 'Age > 53.2' as the first stump because it has the smallest Gini Index.
Q2: Calculate the ‘Amount of Say’ for the first stump and update the ‘sample weights’.
Age > 52.5 Total Error = 0.1
yes no
Disease No Disease
correct (5) incorrect (0) correct (4) incorrect (1)
Norm.
Blood High Heart sample New sample
Age smoker Disease sample
Pressure Rate weight weight
weight
1 52 normal no no no 0.1 0.03 0.053
2 62 high yes no yes 0.1 0.03 0.053
3 35 normal no yes no 0.1 0.03 0.053
4 50 normal no yes yes 0.1 0.3 0.523
5 68 high yes no yes 0.1 0.03 0.053
6 42 normal no no no 0.1 0.03 0.053
7 55 normal yes no yes 0.1 0.03 0.053
8 60 high no no yes 0.1 0.03 0.053
9 38 normal no yes no 0.1 0.03 0.053
10 70 high yes no yes 0.1 0.03 0.053
Increase the sample weight for misclassify sample
= 0.3
Decrease the sample weight for misclassify sample
Age Blood Pressure smoker High Heart Rate Disease sample weight
1 52 normal no no no 0.053 [0, 0.053)
2 62 high yes no yes 0.053 [0.053, 0.106)
3 35 normal no yes no 0.053 [0.106, 0.159)
4 50 normal no yes yes 0.523 [0.159, 0.682)
5 68 high yes no yes 0.053 [0.682, 0.735)
6 42 normal no no no 0.053 [0.735, 0.788)
7 55 normal yes no yes 0.053 [0.788, 0.841)
8 60 high no no yes 0.053 [0.841, 0.894)
9 38 normal no yes no 0.053 [0.894, 0.947)
10 70 high yes no yes 0.053 [0.947, 1]
Q3: Create a new collection of samples that takes into account the updated ‘sample
weights’ from Q2.
Use the new sample weight distribution. Pick 10 random numbers between 0 and 1. For
each number, determine its range. For instance, if the first random number is 0.5 and it
falls within [0.159, 0.682), select sample 4 in the new collection.
For each randomly picked number, determine its range and select the corresponding sample. This
process results in a new collection below.
Then assign the sample weight by 0.1.
Age Blood Pressure smoker High Heart Rate Disease sample weight
4 50 normal no yes yes 0.1
4 50 normal no yes yes 0.1
3 35 normal no yes no 0.1
5 68 high yes no yes 0.1
6 42 normal no no no 0.1
4 50 normal no yes yes 0.1
8 60 high no no yes 0.1
9 38 normal no yes no 0.1
4 50 normal no yes yes 0.1
10 70 high yes no yes 0.1