Scribd
Upload
2 views4 pages

Complete Example

The document outlines the process of calculating entropy and information gain for a dataset related to playing a game based on weather conditions. It details the steps taken to compute the entropy of the dataset, the information gain for attributes like Outlook and Humidity, and concludes with the construction of a decision tree using the ID3 algorithm. The final decision tree is based on Humidity as the root node, leading to classifications for different weather conditions.

Uploaded by

prof.severussnape.hp
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views4 pages

Complete Example

The document outlines the process of calculating entropy and information gain for a dataset related to playing a game based on weather conditions. It details the steps taken to compute the entropy of the dataset, the information gain for attributes like Outlook and Humidity, and concludes with the construction of a decision tree using the ID3 algorithm. The final decision tree is based on Humidity as the root node, leading to classifications for different weather conditions.

Uploaded by

prof.severussnape.hp
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

1.

Sunny, High, No

2. Sunny, Low, Yes

3. Overcast, High, Yes

4. Overcast, Low, Yes

5. Rainy, High, No

6. Rainy, Low, Yes

Step 2: Calculate Entropy for the Entire Dataset


The formula for entropy is:

Entropy(S) = −p+ log2 (p+ ) − p− log2 (p− )


​ ​ ​ ​ ​ ​

Total samples: 6

Play = Yes: 4

Play = No: 2

4 4 2 2
Entropy(S) = − log2 ( ) − log2 ( )
6 6 6 6
​ ​ ​ ​ ​ ​

Let's compute this.

I'll calculate all the values and build the decision tree now.

The entropy of the entire dataset is approximately 0.918.

Step 3: Calculate Information Gain for Each Attribute

A) Information Gain for Outlook

The possible values for Outlook are: Sunny, Overcast, Rainy


Sunny (2 samples): 1 Yes, 1 No

1 1 1 1
Entropy(Sunny) = − log2 ( ) − log2 ( ) = 1
2 2 2 2
​ ​ ​ ​ ​ ​

Overcast (2 samples): 2 Yes, 0 No

Entropy(Overcast) = 0 (pure subset)


Rainy (2 samples): 1 Yes, 1 No

Entropy(Rainy) = 1

Weighted Entropy for Outlook:

2 2 2
Entropy(Outlook) = (1) + (0) + (1)
6 6 6
​ ​ ​

Information Gain:

Gain(Outlook) = Entropy(Total) − Entropy(Outlook)

B) Information Gain for Humidity

The possible values for Humidity are: High, Low

High (3 samples): 1 Yes, 2 No

1 1 2 2
Entropy(High) = − log2 ( ) − log2 ( )
3 3 3 3
​ ​ ​ ​ ​

Low (3 samples): 3 Yes, 0 No

Entropy(Low) = 0 (pure subset)

Weighted Entropy for Humidity:

3 3
Entropy(Humidity) = Entropy(High) + Entropy(Low)
6 6
​ ​

Information Gain:

Gain(Humidity) = Entropy(Total) − Entropy(Humidity)


Let's compute these information gains now. ​ ​

Information Gain for Outlook: 0.252

Information Gain for Humidity: 0.459

Since Humidity has the highest information gain, it becomes the root node.

Step 4: Split on Humidity


For High Humidity: Play = Yes (1), No (2)

For Low Humidity: Play = Yes (3), No (0) (pure leaf)

Subtree for High Humidity:

We must decide whether to split further using Outlook.

Sunny (1 sample): Play = No

Overcast (1 sample): Play = Yes

Rainy (1 sample): Play = No

Since each outlook type has a pure class, they become leaves directly.

Final Decision Tree:


pgsql Copy Edit

Humidity
/ \
High Low
/ | \
Sunny Overcast Rainy
| | |
No Yes No
Play: Yes

This is the decision tree using the ID3 algorithm

You might also like