Multi Class classification using K- fold cross validation

In the multi-class classification problem, there are more than two classes to be predicted.

iris flowers dataset consists of four input variables as numeric and have the same scale in
centimeters. Each instance describes the properties of an observed flower’s measurements, and
the output variable is a specific iris species.

Import Classes and Functions

This includes both the functionality you require from Keras and the data loading from pandas, as
well as data preparation and model evaluation from scikit-learn.
1
import pandas
2
from [Link] import Sequential
3
from [Link] import Dense
4
from [Link].scikit_learn import KerasClassifier
5
from [Link] import np_utils
6
from sklearn.model_selection import cross_val_score
7
from sklearn.model_selection import KFold
8
from [Link] import LabelEncoder
9
from [Link] import Pipeline
10
Load the Dataset

The dataset can be loaded directly. Because the output variable contains strings, it is easiest to
load the data using pandas. You can then split the attributes (columns) into input variables (X)
and output variables (Y).

1 ...
2 # load dataset
3 dataframe = pandas.read_csv("[Link]", header=None)
4 dataset = [Link]
5 X = dataset[:,0:4].astype(float)
6 Y = dataset[:,4]
Encode the Output Variable

The output variable contains three different string values.

When modeling multi-class classification problems using neural networks, it is good practice to
reshape the output attribute from a vector that contains values for each class value to a matrix
with a Boolean for each class value and whether a given instance has that class value or not.

This is called one-hot encoding or creating dummy variables from a categorical variable.
For example, in this problem, three class values are Iris-setosa, Iris-versicolor, and Iris-virginica.
If you had the observations:

1 Iris-setosa
2 Iris-versicolor
3 Iris-virginica
You can turn this into a one-hot encoded binary matrix for each data instance that would look
like this:

1 Iris-setosa, Iris-versicolor,Iris-virginica
2 1, 0, 0
3 0, 1, 0
4 0, 0, 1
You can first encode the strings consistently to integers using the scikit-learn class
LabelEncoder. Then convert the vector of integers to a one-hot encoding using the Keras
function to_categorical().

1 ...
2 # encode class values as integers
3 encoder = LabelEncoder()
4 [Link](Y)
5 encoded_Y = [Link](Y)
6 # convert integers to dummy variables (i.e. one hot encoded)
7 dummy_y = np_utils.to_categorical(encoded_Y)
Define the Neural Network Model

The Keras library provides wrapper classes to allow you to use neural network models developed
with Keras in scikit-learn.

There is a KerasClassifier class in Keras that can be used as an Estimator in scikit-learn, the base
type of model in the library. The KerasClassifier takes the name of a function as an argument.
This function must return the constructed neural network model, ready for training.

Below is a function that will create a baseline neural network for the iris classification problem.
It creates a simple, fully connected network with one hidden layer that contains eight neurons.

The hidden layer uses a rectifier activation function which is a good practice. Because you used a
one-hot encoding for your iris dataset, the output layer must create three output values, one for
each class. The output value with the largest value will be taken as the class predicted by the
model.
The network topology of this simple one-layer neural network can be summarized as follows:

1 4 inputs -> [8 hidden nodes] -> 3 outputs

Note that a “softmax” activation function was used in the output layer. This ensures the output
values are in the range of 0 and 1 and may be used as predicted probabilities.
Finally, the network uses the efficient Adam gradient descent optimization algorithm with a
logarithmic loss function, which is called “categorical_crossentropy” in Keras.
1 ...
2 # define baseline model
3 def baseline_model():
4 # create model
5 model = Sequential()
6 [Link](Dense(8, input_dim=4, activation='relu'))
7 [Link](Dense(3, activation='softmax'))
8 # Compile model
9 [Link](loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
10 return model
You can now create your KerasClassifier for use in scikit-learn.

You can also pass arguments in the construction of the KerasClassifier class that will be passed
on to the fit() function internally used to train the neural network. Here, you pass the number of
epochs as 200 and batch size as 5 to use when training the model. Debugging is also turned off
when training by setting verbose to 0.

1 ...
2 estimator = KerasClassifier(build_fn=baseline_model, epochs=200, batch_size=5, verbose=0)
Evaluate the Model with k-Fold Cross Validation

You can now evaluate the neural network model on our training data.

The scikit-learn has excellent capability to evaluate models using a suite of techniques. The gold
standard for evaluating machine learning models is k-fold cross validation.

First, define the model evaluation procedure. Here, you set the number of folds to 10 (an
excellent default) and shuffle the data before partitioning it.

1 ...
2 kfold = KFold(n_splits=10, shuffle=True)
Now, you can evaluate your model (estimator) on your dataset (X and dummy_y) using a 10-fold
cross-validation procedure (k-fold).
 Evaluating the model only takes approximately 10 seconds and returns an object that describes
the evaluation of the ten constructed models for each of the splits of the dataset.

1 ...
2 results = cross_val_score(estimator, X, dummy_y, cv=kfold)
3 print("Baseline: %.2f%% (%.2f%%)" % ([Link]()*100, [Link]()*100))
Save Model Weights and Architecture Together

Keras separates the concerns of saving your model architecture and saving your model weights.

Model weights are saved to an HDF5 format. This grid format is ideal for storing multi-
dimensional arrays of numbers.
The model structure can be described and saved using two different formats: JSON and YAML.

Three ways of saving and loading your model to a file:

 Save Model to JSON

 Save Model to YAML
 Save Model to HDF5
The first two examples save the model architecture and weights separately. The model weights
are saved into an HDF5 format file in all cases.

Keras also supports a simpler interface to save both the model weights and model architecture
together into a single H5 file.

Saving the model in this way includes everything you need to know about the model, including:

 Model weights
 Model architecture
 Model compilation details (loss and metrics)
 Model optimizer state
This means that you can load and use the model directly without having to re-compile it

# save model and architecture to single file

[Link]("model.h5")
print("Saved model to disk")

Note that in the Keras library, there is another function doing the same, as follows:

1 ...
2 # equivalent to: [Link]("model.h5")
3 from [Link] import save_model
4 save_model(model, "model.h5")
How to Load a Keras Model and Summarize

Your saved model can then be loaded later by calling the load_model() function and passing the
filename. The function returns the model with the same architecture and weights.
from [Link] import load_model

# load model
model = load_model('model.h5')
# summarize model.
[Link]()