30/12/2023, 15:06 RandomsearchCV.
ipynb - Colaboratory
Hyper Parameter Optimization Techniques
1. GridSearchCV
2. RandomizedSearchCV
3. Bayesian Optimization -Automate Hyperparameter Tuning (Hyperopt)
4. Optuna- Automate Hyperparameter Tuning
import warnings
[Link]('ignore')
import pandas as pd
df=pd.read_csv('/content/randomforest_diabetes.csv')
[Link]()
account_circle Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
0 6 148 72 35 0 33.6 0.627 50 1
1 1 85 66 29 0 26.6 0.351 31 0
2 8 183 64 0 0 23.3 0.672 32 1
3 1 89 66 23 94 28.1 0.167 21 0
4 0 137 40 35 168 43.1 2.288 33 1
import numpy as np
df['Glucose']=[Link](df['Glucose']==0,df['Glucose'].median(),df['Glucose'])
[Link]()
Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigree
0 6 148.0 72 35 0 33.6
1 1 85.0 66 29 0 26.6
2 8 183.0 64 0 0 23.3
3 1 89.0 66 23 94 28.1
4 0 137.0 40 35 168 43.1
#### Independent And Dependent features
X=[Link]('Outcome',axis=1)
y=df['Outcome']
[Link](X,columns=[Link][:-1])
Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigr
0 6 148.0 72 35 0 33.6
1 1 85.0 66 29 0 26.6
2 8 183.0 64 0 0 23.3
3 1 89.0 66 23 94 28.1
4 0 137.0 40 35 168 43.1
... ... ... ... ... ... ...
763 10 101.0 76 48 180 32.9
764 2 122.0 70 27 0 36.8
765 5 121.0 72 23 112 26.2
766 1 126.0 60 0 0 30.1
767 1 93.0 70 31 0 30.4
768 rows × 8 columns
[Link] 1/7
30/12/2023, 15:06 [Link] - Colaboratory
#### Train Test Split
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.20,random_state=0)
from [Link] import RandomForestClassifier
rf_classifier=RandomForestClassifier(n_estimators=10).fit(X_train,y_train)
prediction=rf_classifier.predict(X_test)
y.value_counts()
0 500
1 268
Name: Outcome, dtype: int64
from [Link] import confusion_matrix,classification_report,accuracy_score
print(confusion_matrix(y_test,prediction))
print(accuracy_score(y_test,prediction))
print(classification_report(y_test,prediction))
[[93 14]
[17 30]]
0.7987012987012987
precision recall f1-score support
0 0.85 0.87 0.86 107
1 0.68 0.64 0.66 47
accuracy 0.80 154
macro avg 0.76 0.75 0.76 154
weighted avg 0.80 0.80 0.80 154
The main parameters used by a Random Forest Classifier are:
criterion = the function used to evaluate the quality of a split.
max_depth = maximum number of levels allowed in each tree.
max_features = maximum number of features considered when splitting a node.
min_samples_leaf = minimum number of samples which can be stored in a tree leaf.
min_samples_split = minimum number of samples necessary in a node to cause node splitting.
n_estimators = number of trees in the ensamble.
### Manual Hyperparameter Tuning
model=RandomForestClassifier(n_estimators=300,criterion='entropy',
max_features='sqrt',min_samples_leaf=10,random_state=100).fit(X_tr
predictions=[Link](X_test)
print(confusion_matrix(y_test,predictions))
print(accuracy_score(y_test,predictions))
print(classification_report(y_test,predictions))
[[98 9]
[18 29]]
0.8246753246753247
precision recall f1-score support
0 0.84 0.92 0.88 107
1 0.76 0.62 0.68 47
accuracy 0.82 154
macro avg 0.80 0.77 0.78 154
weighted avg 0.82 0.82 0.82 154
keyboard_arrow_down Randomized Search Cv
[Link] 2/7
30/12/2023, 15:06 [Link] - Colaboratory
import numpy as np
from sklearn.model_selection import RandomizedSearchCV
# Number of trees in random forest
n_estimators = [int(x) for x in [Link](start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt','log2']
# Maximum number of levels in tree
max_depth = [int(x) for x in [Link](10, 1000,10)]
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10,14]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4,6,8]
# Create the random grid
random_grid = {'n_estimators': n_estimators,
'max_features': max_features,
'max_depth': max_depth,
'min_samples_split': min_samples_split,
'min_samples_leaf': min_samples_leaf,
'criterion':['entropy','gini']}
print(random_grid)
{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt', 'log2'], 'max_depth': [1
rf=RandomForestClassifier()
rf_randomcv=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,n_iter=100,cv=3,ver
random_state=100,n_jobs=-1)
### fit the randomized model
rf_randomcv.fit(X_train,y_train)
Fitting 3 folds for each of 100 candidates, totalling 300 fits
▸ RandomizedSearchCV
▸ estimator: RandomForestClassifier
▸ RandomForestClassifier
ut
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-17-c0786cf6f4d3> in <cell line: 1>()
----> 1 ut
NameError: name 'ut' is not defined
SEARCH STACK OVERFLOW
rf_randomcv.best_params_
rf_randomcv
best_random_grid=rf_randomcv.best_estimator_
from [Link] import accuracy_score
y_pred=best_random_grid.predict(X_test)
print(confusion_matrix(y_test,y_pred))
print("Accuracy Score {}".format(accuracy_score(y_test,y_pred)))
print("Classification report: {}".format(classification_report(y_test,y_pred)))
Gridsearch CV
rf_randomcv.best_params_
[Link] 3/7
30/12/2023, 15:06 [Link] - Colaboratory
from sklearn.model_selection import GridSearchCV
param_grid = {
'criterion': [rf_randomcv.best_params_['criterion']],
'max_depth': [rf_randomcv.best_params_['max_depth']],
'max_features': [rf_randomcv.best_params_['max_features']],
'min_samples_leaf': [rf_randomcv.best_params_['min_samples_leaf'],
rf_randomcv.best_params_['min_samples_leaf']+2,
rf_randomcv.best_params_['min_samples_leaf'] + 4],
'min_samples_split': [rf_randomcv.best_params_['min_samples_split'] - 2,
rf_randomcv.best_params_['min_samples_split'] - 1,
rf_randomcv.best_params_['min_samples_split'],
rf_randomcv.best_params_['min_samples_split'] +1,
rf_randomcv.best_params_['min_samples_split'] + 2],
'n_estimators': [rf_randomcv.best_params_['n_estimators'] - 200, rf_randomcv.best_params_['
rf_randomcv.best_params_['n_estimators'],
rf_randomcv.best_params_['n_estimators'] + 100, rf_randomcv.best_params_['
}
print(param_grid)
#### Fit the grid_search to the data
rf=RandomForestClassifier()
grid_search=GridSearchCV(estimator=rf,param_grid=param_grid,cv=10,n_jobs=-1,verbose=2)
grid_search.fit(X_train,y_train)
grid_search.best_estimator_
best_grid=grid_search.best_estimator_
best_grid
y_pred=best_grid.predict(X_test)
print(confusion_matrix(y_test,y_pred))
print("Accuracy Score {}".format(accuracy_score(y_test,y_pred)))
print("Classification report: {}".format(classification_report(y_test,y_pred)))
keyboard_arrow_down Automated Hyperparameter Tuning
Automated Hyperparameter Tuning can be done by using techniques such as
Bayesian Optimization
Gradient Descent
Evolutionary Algorithms
keyboard_arrow_down Bayesian Optimization
Bayesian optimization uses probability to find the minimum of a function. The final aim is to find the input value to a function which can gives
us the lowest possible output value.
It is used to reduce the no. of trails. It builds the probabilitics model of objective function called Surrogate.
The three Surrogate methods are
1. Gaussian processes (Spearmint and MOE )
2. Random Forest Regression ( SMAC )
3. Tree Parzon Estimator ( Hyperopt )
In Hyperopt, Bayesian Optimization can be implemented giving 3 three main parameters to the function fmin.
Objective Function = defines the loss function to minimize.
Domain Space = defines the range of input values to test (in Bayesian Optimization this space creates a probability distribution for each of
the used Hyperparameters).
[Link] 4/7
30/12/2023, 15:06 [Link] - Colaboratory
Optimization Algorithm = defines the search algorithm to use to select the best input values to use in each new iteration.
from hyperopt import hp,fmin,tpe,STATUS_OK,Trials
(a) Search Space
The hyperopt have different functions to specify ranges for input parameters, these are stochastic search spaces. The most common options
for a search space to choose are :
[Link](label, options) — This can be used for categorical parameters, it returns one of the options, which should be a list or [Link]:
[Link](“criterion”, [“gini”,”entropy”,])
space = {'criterion': [Link]('criterion', ['entropy', 'gini']),
'max_depth': [Link]('max_depth', 10, 1200, 10),
'max_features': [Link]('max_features', ['auto', 'sqrt','log2', None]),
'min_samples_leaf': [Link]('min_samples_leaf', 0, 0.5),
'min_samples_split' : [Link] ('min_samples_split', 0, 1),
'n_estimators' : [Link]('n_estimators', [10, 50, 300, 750, 1200,1300,1500])
}
space
(b) Objective Function
This is a function to minimize that receives hyperparameters values as input from the search space and returns the loss. This means during the
optimization process, we train the model with selected hyperparameters values and predict the target feature and then evaluate the prediction
error and give it back to the optimizer. The optimizer will decide which values to check and iterate again.
Our function to minimize is called objective and the classification algorithm to optimize its hyperparameter is Random Forest. Use cross-
validation to avoid overfitting and then the function will return a loss values and its status.
minimizes the function, so add a negative sign in the accuracy
def objective(space):
model = RandomForestClassifier(criterion = space['criterion'],
max_depth = space['max_depth'],
max_features = space['max_features'],
min_samples_leaf = space['min_samples_leaf'],
min_samples_split = space['min_samples_split'],
n_estimators = space['n_estimators'],
)
accuracy = cross_val_score(model, X_train, y_train, cv = 5).mean()
# We aim to maximize accuracy, therefore we return it as a negative value
return {'loss': -accuracy, 'status': STATUS_OK }
The Trials object is used to keep All hyperparameters, loss, and other information, so that we can access them after running optimization. Also,
trials can help you to save important information and later load and then resume the optimization process.
(c) fmin The fmin function is the optimization function that iterates on different sets of algorithms and their hyperparameters and then
minimizes the objective function. the fmin takes 5 inputs which are:-
1. The objective function to minimize
2. The defined search space
3. The search algorithm to use such as Random search, TPE (Tree Parzen Estimators), and Adaptive TPE. Note: [Link]
provides logic for a sequential search of the hyperparameter space.
4. The maximum number of evaluations.
5. The trials object (optional).
[Link] 5/7
30/12/2023, 15:06 [Link] - Colaboratory
from sklearn.model_selection import cross_val_score
trials = Trials()
best = fmin(fn= objective,
space= space,
algo= [Link],
max_evals = 80,
trials= trials)
best
crit = {0: 'entropy', 1: 'gini'}
feat = {0: 'auto', 1: 'sqrt', 2: 'log2', 3: None}
est = {0: 10, 1: 50, 2: 300, 3: 750, 4: 1200,5:1300,6:1500}
print(crit[best['criterion']])
print(feat[best['max_features']])
print(est[best['n_estimators']])
best['min_samples_leaf']
trainedforest = RandomForestClassifier(criterion = crit[best['criterion']], max_depth = best['m
max_features = feat[best['max_features']],
min_samples_leaf = best['min_samples_leaf'],
min_samples_split = best['min_samples_split'],
n_estimators = est[best['n_estimators']]).fit(X_train,y_
predictionforest = [Link](X_test)
print(confusion_matrix(y_test,predictionforest))
print(accuracy_score(y_test,predictionforest))
print(classification_report(y_test,predictionforest))
acc5 = accuracy_score(y_test,predictionforest)
keyboard_arrow_down Optimize hyperparameters of the model using Optuna
Optuna
Optuna is “an open-source hyperparameter optimization framework to automate hyperparameter search.”
The key features of Optuna include
“automated search for optimal hyperparameters,”
“efficiently search large spaces and prune unpromising trials for faster results,” and
“parallelize hyperparameter searches over multiple threads or processes.”
The first step is to define the objective function for Optuna to maximize. The objective function takes a “Trial” object as the input and return the
score, a float value or a list of float values.
The next step is to use the objective function to create a “Study” object and then optimize it.
Optuna is highly efficient, as the tuning process is much faster than scikit-learn’s grid search, and the result is better than a random search.
!pip install optuna
[Link] 6/7
30/12/2023, 15:06 [Link] - Colaboratory
import optuna
import [Link]
from [Link] import RandomForestClassifier
from sklearn.model_selection import cross_val_score
def objective(trial):
iris = [Link].load_iris()
x, y = [Link], [Link]
criterion = trial.suggest_categorical("criterion", ["gini", "entropy"])
max_depth = trial.suggest_int("max_depth", 2, 32, log=True)
n_estimators = trial.suggest_int("n_estimators", 100,500)
rf = [Link](criterion =criterion,
max_depth=max_depth,
n_estimators=n_estimators
)
score = cross_val_score(rf, x, y, n_jobs=-1, cv=3)
accuracy = [Link]()
return accuracy
study = optuna.create_study(direction="maximize")
[Link](objective, n_trials=15)
study.best_params
Multi-Fidelity Optimization (MFO)
[Link] 7/7