0% found this document useful (0 votes)

28 views13 pages

California Housing Decision Tree Model

Q: Why might 'max_depth' vary significantly in optimal parameter settings across different datasets?

'Max_depth' dictates the complexity level of a decision tree. Variations in optimal settings across datasets occur because each dataset has unique patterns and noise levels. A richer dataset with intricate patterns might benefit from deeper trees to capture complexities, whereas simpler datasets could perform better with shallower trees to prevent overfitting. Hence, 'max_depth' directly ties the model's capacity to the dataset's inherent characteristics .

Q: What are the benefits of using grid search with cross-validation when tuning hyperparameters for decision tree models?

Using grid search with cross-validation allows for systematic exploration of hyperparameter combinations, ensuring the optimal parameters are chosen for the model. Cross-validation helps provide a more accurate assessment of model performance by using different subsets of the training data for validation. This reduces the risk of overfitting and ensures the model generalizes well to unseen data. Moreover, it provides a balanced evaluation metric across parameter choices, such as accuracy or negative mean squared error .

Q: What implications do scoring metrics have for decision tree models in regression analysis?

Scoring metrics such as R2, mean squared error (MSE), and others, serve as benchmarks for evaluating the performance of decision tree models in regression analysis. High R2 scores indicate that the model accounts for the variance in the dependent variable. It implies that the decision tree is effectively capturing relationships in the data. Conversely, a high MSE indicates large errors in predictions. These metrics guide model improvements and validate if the chosen model and parameters appropriately fit the dataset characteristics .

Q: How does the decision tree visualize its model, and what can be interpreted from it?

A decision tree visualization provides a graphical representation of the model's decision-making process, showing the feature splits and corresponding entropy or impurity values at each node. From this visualization, one can interpret the hierarchy of feature importance, the number of samples following each branch, and the expected class probabilities at the leaf nodes. This aids in understanding the model's logic and identifying potential issues such as overfitting or unnecessary complexity in the tree structure .

Q: What role does criterion selection play in the decision tree's performance, as illustrated by GridSearchCV evaluations?

Criterion selection, such as 'entropy' vs. 'gini', affects how the decision tree evaluates splits at each node. Different criteria can influence which features are prioritized during the tree-building process, impacting the structure and, ultimately, the overall accuracy of the model. As demonstrated by GridSearchCV evaluations, optimal criteria results in a decision tree that better captures the underlying data patterns, enhancing predictive performance .

Q: Why might a decision tree regressor perform unpredictably on the California Housing dataset?

A decision tree regressor might perform unpredictably on the California Housing dataset due to the complexity and variability inherent in the dataset. The dataset contains features with potentially large deviations, such as median income or average rooms, which can lead to complex splits that overfit the data. Furthermore, geographical and sociological factors impacting housing prices might not be effectively captured leading to generalization errors .

Q: In what ways does pre-pruning enhance the efficiency of decision tree classifiers?

Pre-pruning enhances efficiency by stopping the growth of the decision tree early, before it becomes too complex. This is achieved by setting conditions like maximum depth or minimum number of samples required to split a node. These conditions help prevent the tree from overfitting the training data and reduce computational costs by limiting tree growth, thereby making the model simpler and more generalizable .

Q: How does entropy affect the decision-making process in a Decision Tree Classifier?

Entropy is used as a criterion in decision tree classifiers to measure the impurity or disorder of a node. In this context, lower entropy signifies higher purity, which implies that the samples at a node predominantly belong to a single class. As the decision tree splits the data, it aims to reduce entropy by making splits that cluster similar class labels together, eventually resulting in more accurate and reliable classification decisions .

Q: What insights can be drawn from comparing the classification report metrics for different classifiers?

Comparing classification report metrics such as precision, recall, and f1-score provides insights into the strengths and weaknesses of different classifiers. Precision indicates the classifier's accuracy when predicting positive labels, while recall represents its ability to identify all relevant positive samples. F1-score, as a harmonic mean of precision and recall, offers a balanced view. Metrics variations highlight which model experiences overfitting, underfitting, or balance issues. A classifier with consistently high scores across these measures is generally more reliable and effective .

Q: How does the concept of overfitting relate to post-pruning techniques in decision trees?

Overfitting occurs when a decision tree model becomes too complex, capturing noise rather than the true data pattern. Post-pruning combats overfitting by trimming the tree after it has been built, eliminating branches that provide little power in predicting the target variable. This simplification of the tree helps improve the model's ability to generalize to new data, enhancing performance on the test set while reducing variance .

The document discusses implementing a decision tree regressor to predict California housing prices using various housing features. It loads and explores the California housing dataset, then performs hyperparameter tuning on the decision tree regressor using grid search cross validation to find the optimal parameters.

Uploaded by

Vasudevan

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

28 views13 pages

California Housing Decision Tree Model

Uploaded by

Vasudevan

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

Decision Tree Classifier Implementation

With Post Prunning And

Preprunning
In [26]: import pandas as pd
import numpy as np
import [Link] as plt
%matplotlib inline

In [27]: from [Link] import load_iris

In [28]: dataset=load_iris()

In [29]: import seaborn as sns

df=sns.load_dataset('iris')

In [30]: #Independent and dependent features

X=[Link][:,:-1]
y=[Link]

In [31]: ### train test split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.33, random_state=42)

In [32]: from [Link] import DecisionTreeClassifier

In [33]: classifier=DecisionTreeClassifier(criterion='entropy')

In [34]: [Link](X_train,y_train)

Out[34]: DecisionTreeClassifier(criterion='entropy')
In [35]: X_train.head()

Out[35]:
sepal_length sepal_width petal_length petal_width

96 5.7 2.9 4.2 1.3

105 7.6 3.0 6.6 2.1

66 5.6 3.0 4.5 1.5

0 5.1 3.5 1.4 0.2

122 7.7 2.8 6.7 2.0

In [36]: from sklearn import tree
[Link](figsize=(5,6))
tree.plot_tree(classifier,filled=True)

Out[36]: [Text(0.4444444444444444, 0.9285714285714286, 'X[3] <=

0.8\nentropy = 1.583\nsamples = 100\nvalue = [31, 35, 3
4]'),
Text(0.3333333333333333, 0.7857142857142857, 'entropy
= 0.0\nsamples = 31\nvalue = [31, 0, 0]'),
Text(0.5555555555555556, 0.7857142857142857, 'X[3] <=
1.75\nentropy = 1.0\nsamples = 69\nvalue = [0, 35, 3
4]'),
Text(0.3333333333333333, 0.6428571428571429, 'X[2] <=
5.35\nentropy = 0.485\nsamples = 38\nvalue = [0, 34,
4]'),
Text(0.2222222222222222, 0.5, 'X[3] <= 1.45\nentropy =
0.31\nsamples = 36\nvalue = [0, 34, 2]'),
Text(0.1111111111111111, 0.35714285714285715, 'entropy
= 0.0\nsamples = 26\nvalue = [0, 26, 0]'),
Text(0.3333333333333333, 0.35714285714285715, 'X[1] <=
2.6\nentropy = 0.722\nsamples = 10\nvalue = [0, 8,
2]'),
Text(0.2222222222222222, 0.21428571428571427, 'X[0] <=
6.15\nentropy = 0.918\nsamples = 3\nvalue = [0, 1,
2]'),
Text(0.1111111111111111, 0.07142857142857142, 'entropy
= 0.0\nsamples = 2\nvalue = [0, 0, 2]'),
Text(0.3333333333333333, 0.07142857142857142, 'entropy
= 0.0\nsamples = 1\nvalue = [0, 1, 0]'),
Text(0.4444444444444444, 0.21428571428571427, 'entropy
= 0.0\nsamples = 7\nvalue = [0, 7, 0]'),
Text(0.4444444444444444, 0.5, 'entropy = 0.0\nsamples
= 2\nvalue = [0, 0, 2]'),
Text(0.7777777777777778, 0.6428571428571429, 'X[2] <=
4.85\nentropy = 0.206\nsamples = 31\nvalue = [0, 1, 3
0]'),
Text(0.6666666666666666, 0.5, 'X[1] <= 3.1\nentropy =
0.918\nsamples = 3\nvalue = [0, 1, 2]'),
Text(0.5555555555555556, 0.35714285714285715, 'entropy
= 0.0\nsamples = 2\nvalue = [0, 0, 2]'),
Text(0.7777777777777778, 0.35714285714285715, 'entropy
= 0.0\nsamples = 1\nvalue = [0, 1, 0]'),
Text(0.8888888888888888, 0.5, 'entropy = 0.0\nsamples
= 28\nvalue = [0, 0, 28]')]

In [37]: ## Post Prunning

classifier=DecisionTreeClassifier(criterion='entropy',max_depth
[Link](X_train,y_train)

Out[37]: DecisionTreeClassifier(criterion='entropy', max_depth=

2)
In [38]: from sklearn import tree
[Link](figsize=(5,6))
tree.plot_tree(classifier,filled=True)

Out[38]: [Text(0.4, 0.8333333333333334, 'X[3] <= 0.8\nentropy =

1.583\nsamples = 100\nvalue = [31, 35, 34]'),
Text(0.2, 0.5, 'entropy = 0.0\nsamples = 31\nvalue =
[31, 0, 0]'),
Text(0.6, 0.5, 'X[3] <= 1.75\nentropy = 1.0\nsamples =
69\nvalue = [0, 35, 34]'),
Text(0.4, 0.16666666666666666, 'entropy = 0.485\nsampl
es = 38\nvalue = [0, 34, 4]'),
Text(0.8, 0.16666666666666666, 'entropy = 0.206\nsampl
es = 31\nvalue = [0, 1, 30]')]
In [39]: ##prediction
y_pred=[Link](X_test)

In [40]: y_pred

Out[40]: array([1, 0, 2, 1, 1, 0, 1, 2, 1, 1, 2, 0, 0, 0, 0, 1,
2, 1, 1, 2, 0, 2,
0, 2, 2, 2, 2, 2, 0, 0, 0, 0, 1, 0, 0, 2, 1, 0,
0, 0, 2, 1, 1, 0,
0, 1, 1, 2, 1, 2])

In [41]: from [Link] import accuracy_score,classification_repo

score=accuracy_score(y_pred,y_test)
print(score)
print(classification_report(y_pred,y_test))

0.98
precision recall f1-score support

0 1.00 1.00 1.00 19

1 1.00 0.94 0.97 16
2 0.94 1.00 0.97 15

accuracy 0.98 50
macro avg 0.98 0.98 0.98 50
weighted avg 0.98 0.98 0.98 50

DecisionTree Prepruning And

Hyperparameter Tuning For Huge
Data
In [42]: import warnings
[Link]('ignore')
In [43]: parameter={
'criterion':['gini','entropy','log_loss'],
'splitter':['best','random'],
'max_depth':[1,2,3,4,5],
'max_features':['auto', 'sqrt', 'log2']

In [44]: from sklearn.model_selection import GridSearchCV

In [45]: classifier=DecisionTreeClassifier()
clf=GridSearchCV(classifier,param_grid=parameter,cv=5,scoring

In [46]: [Link](X_train,y_train)

Out[46]: GridSearchCV(cv=5, estimator=DecisionTreeClassifier(),

param_grid={'criterion': ['gini', 'entropy
', 'log_loss'],
'max_depth': [1, 2, 3, 4, 5],
'max_features': ['auto', 'sqrt
', 'log2'],
'splitter': ['best', 'random
']},
scoring='accuracy')

In [50]: clf.best_params_

Out[50]: {'criterion': 'gini',

'max_depth': 4,
'max_features': 'sqrt',
'splitter': 'best'}

In [51]: y_pred=[Link](X_test)
In [52]: from [Link] import accuracy_score,classification_repo
score=accuracy_score(y_pred,y_test)
print(score)
print(classification_report(y_pred,y_test))

0.98
precision recall f1-score support

0 1.00 1.00 1.00 19

1 1.00 0.94 0.97 16
2 0.94 1.00 0.97 15

accuracy 0.98 50
macro avg 0.98 0.98 0.98 50
weighted avg 0.98 0.98 0.98 50

In [ ]:
Decision Tree Regressor Implementation
In [120]: import pandas as pd
import [Link] as plt
%matplotlib inline

In [121]: ##California House Pricing Dataset

from [Link] import fetch_california_housing
california_df=fetch_california_housing()
In [122]: california_df

Out[122]: {'data': array([[ 8.3252 , 41. , 6.98412698, ..., 2.55555

556,
37.88 , -122.23 ],
[ 8.3014 , 21. , 6.23813708, ..., 2.10984183,
37.86 , -122.22 ],
[ 7.2574 , 52. , 8.28813559, ..., 2.80225989,
37.85 , -122.24 ],
...,
[ 1.7 , 17. , 5.20554273, ..., 2.3256351 ,
39.43 , -121.22 ],
[ 1.8672 , 18. , 5.32951289, ..., 2.12320917,
39.43 , -121.32 ],
[ 2.3886 , 16. , 5.25471698, ..., 2.61698113,
39.37 , -121.24 ]]),
'target': array([4.526, 3.585, 3.521, ..., 0.923, 0.847, 0.894]),
'frame': None,
'target_names': ['MedHouseVal'],
'feature_names': ['MedInc',
'HouseAge',
'AveRooms',
'AveBedrms',
'Population',
'AveOccup',
'Latitude',
'Longitude'],
'DESCR': '.. _california_housing_dataset:\n\nCalifornia Housing dataset\n
--------------------------\n\n**Data Set Characteristics:**\n\n :Number of
Instances: 20640\n\n :Number of Attributes: 8 numeric, predictive attribut
es and the target\n\n :Attribute Information:\n - MedInc med
ian income in block group\n - HouseAge median house age in block
group\n - AveRooms average number of rooms per household\n
- AveBedrms average number of bedrooms per household\n - Populatio
n block group population\n - AveOccup average number of househ
old members\n - Latitude block group latitude\n - Longitud
e block group longitude\n\n :Missing Attribute Values: None\n\nThis da
taset was obtained from the StatLib repository.\n[Link]
orgo/Regression/cal_housing.html\n\nThe target variable is the median house v
alue for California districts,\nexpressed in hundreds of thousands of dollars
($100,000).\n\nThis dataset was derived from the 1990 U.S. census, using one
row per census\nblock group. A block group is the smallest geographical unit
for which the U.S.\nCensus Bureau publishes sample data (a block group typica
lly has a population\nof 600 to 3,000 people).\n\nAn household is a group of
people residing within a home. Since the average\nnumber of rooms and bedroom
s in this dataset are provided per household, these\ncolumns may take surpins
ingly large values for block groups with few households\nand many empty house
s, such as vacation resorts.\n\nIt can be downloaded/loaded using the\n:func:
`[Link].fetch_california_housing` function.\n\n.. topic:: Reference
s\n\n - Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions,\
n Statistics and Probability Letters, 33 (1997) 291-297\n'}

In [123]: df=[Link](california_df.data,columns=california_df.feature_names)
df['Target']=california_df.target
In [124]: [Link]

Out[124]: (20640, 9)

In [125]: # Taking Sample Data

df=[Link](frac=0.25)

In [126]: [Link]

Out[126]: (5160, 9)

In [127]: #independent features

X=[Link][:,:-1]
#dependent features
y=[Link][:,-1]

In [128]: [Link]()

Out[128]: MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude Longitude

12416 2.1217 38.0 5.557377 1.154098 1279.0 4.193443 33.73 -116.22

15514 4.4477 4.0 5.752632 1.025263 2844.0 2.993684 33.17 -117.06

14661 3.7237 15.0 4.695906 1.096491 784.0 2.292398 32.80 -117.13

13069 2.6471 16.0 4.098667 0.970667 1125.0 3.000000 38.58 -121.30

17491 2.1417 31.0 2.651163 1.149502 699.0 2.322259 34.43 -119.83

In [129]: ### train test split

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.33, random_state=42)

In [130]: from [Link] import DecisionTreeRegressor

regressor=DecisionTreeRegressor()

In [131]: [Link](X_train,y_train)

Out[131]: DecisionTreeRegressor()

In [132]: y_pred=[Link](X_test)

In [133]: y_pred

Out[133]: array([2.875 , 3.964 , 1.193 , ..., 0.803 , 3.133 , 5.00001])

In [134]: from [Link] import r2_score

score=r2_score(y_pred,y_test)
In [135]: score

Out[135]: 0.523157666245229

In [136]: ## Hyperparameter Tunning

parameter={
'criterion':['squared_error','friedman_mse','absolute_error','poisson'],
'splitter':['best','random'],
'max_depth':[1,2,3,4,5,6,7,8,10,11,12],
'max_features':['auto', 'sqrt', 'log2']

}
regressor=DecisionTreeRegressor()

In [153]: #[Link]
import warnings
[Link]('ignore')
from sklearn.model_selection import GridSearchCV
regressorcv=GridSearchCV(regressor,param_grid=parameter,cv=2,scoring='neg_mean_squared

In [154]: [Link](X_train,y_train)

Out[154]: GridSearchCV(cv=2,
estimator=DecisionTreeRegressor(criterion='absolute_error',
max_depth=7, max_features='auto
'),
param_grid={'criterion': ['squared_error', 'friedman_mse',
'absolute_error', 'poisson'],
'max_depth': [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12],
'max_features': ['auto', 'sqrt', 'log2'],
'splitter': ['best', 'random']},
scoring='neg_mean_squared_error')

In [155]: regressorcv.best_params_

Out[155]: {'criterion': 'absolute_error',

'max_depth': 8,
'max_features': 'auto',
'splitter': 'best'}

In [156]: regressor=DecisionTreeRegressor(criterion='absolute_error',max_depth=7,max_features

In [157]: [Link](X_train,y_train)

Out[157]: DecisionTreeRegressor(criterion='absolute_error', max_depth=7,

max_features='auto')
In [158]: from sklearn import tree
[Link](figsize=(12,10))
tree.plot_tree(regressor,filled=True)
[Link]()

In [159]: y_pred=[Link](X_test)

In [160]: r2_score(y_pred,y_test)

Out[160]: 0.5660230963433509

In [ ]:

Common questions