Machine Learning

Machine Learning
Lecture: Recap

Ted Scully
RECAP

Mini-Batch Processing
 If your batch size is the same as your training set size then you are using batch GD.
 Most of time you don’t just adopt batch GD. We have already outlined the
problem with this option. Too slow for large training sets.
 It has been observed in practice that when using a larger batch there is a
degradation in the quality of the model (Keshar et al 2016).

 If the batch size = 1 we are using what is called stochastic GD.

 The drawback of this option is it can be very noisy and difficult to converge to
a true minimum
 We lose the speed advantage provided by vectorization

 If batch size > 1 and < m then we are using what is called mini-batch gradient
descent.
 Typically sizes for mini-batch is 32, 64, 128, 256, 512, 1024.
 The advantage of this technique is that we can achieve a relatively high
frequency of weight updates while still retaining the benefit of vectorised
speed up advantages.
RECAP Gradient Descent with Momentum

gradient
gradient
RECAP
Gradient Descent with Momentum
 Notice in the code we maintain an EMA for the gradients for both trainable
parameters. It is this value that now becomes core to the update of these
parameters.
 Notice if there is a high level of oscillation for one parameter such as bias then
it will smoothen out this value and slow down the rate of update.
 In turn this can allow us to use a large learning rate and obtain convergence
faster.
repeat

𝐸𝜆1 = 𝛽𝐸𝜆1 + (1 − 𝛽)𝑑𝜆1

𝐸𝑏 = 𝛽𝐸𝑏 + (1 − 𝛽)𝑑𝑏
𝜆1 = 𝜆1 − 𝛼 (𝐸𝜆1 )
𝑏 = 𝑏 − 𝛼 (𝐸𝑏 )
RECAP
Tensors
 A tensor is typically a multidimensional array (like what we have used with NumPy
ndarray). One minor point is that a tensor can also hold a single scalar value
(unlike NumPy).

import tensorflow as tf

t1 = [Link]((4, 5))
t2 = [Link](3, 15, 2)
t3 = [Link]((2,2))
t4 = [Link](shape=[2,2],dtype=tf.float32)

import tensorflow as tf

v = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) A [Link] is a special type of tensor
meant to hold a mutable state such as the
print (v) weights of a neural network.
print ([Link])

[Link](v-2)
print (v)
RECAP

Automatic Differentiation in TensorFlow

• We initially create an instance of [Link], which records all operations once it
has been created.

• To determine the gradients of the loss function with respect to some variables, we
call the method gradient.
• The first argument is the “target” for the calculation, i.e. typically the output of
the loss function (or whatever function you are trying to optimize)
• The second argument is the “source”, these are the variables values you can alter
in order to optimize the target loss function.

• The gradient function will return the partial derivatives of the (loss) function with
respect to each of the listed trainable variables.

• By default, the resources held by a GradientTape are released as soon as

[Link]() method is called.

• We can either update the variables ourselves (using the gradient descent update rule)
or more commonly we can use an existing optimizer to update the gradients for us.
RECAP def predict(x):
# x2 – 20x + 100
firstTerm = x**2
secondTerm = -20.0* x
quadratic = firstTerm + secondTerm +100.0
return quadratic

learning_rate = 0.1
iterations = 50

x = [Link](20.0, tf.float32)

for i in range(iterations):

with [Link]() as tape:

prediction = predict(x)

gradient = [Link](prediction, x)

x.assign_sub(gradient * learning_rate)

print (x)
Deep Learning

Deep Learning
Lecture: Keras – Sequential API

Ted Scully
 Keras – Layers
1. The fundamental data structure in Keras is a layer.
2. A layer is a data-processing module that takes as
input one or more tensors and outputs one or
more tensors.
3. Typically, each layer has a current state that is
defined by the layer’s weights, one or several
tensors learned with gradient descent (some layers
are stateless).
4. Different types of layers are defined for different
types of networks and problem types.
a. For example, 2D tensors of shape (samples,
features), are used with densely connected
layers.
b. Sequence data, stored in 3D tensors of shape
(samples, time, features), is typically
processed by recurrent layers, such as an
LSTM layer, or 1D convolution layers
(Conv1D).
c. Image data, stored in 4D tensors, is usually
processed by 2D convolution layers (Conv2D).
 Keras – Sequential Model and Dense Layer
1. In Keras, a model, is a way to organize layers.
2. The standard type of model is the Sequential model, a linear stack of layers
3. The most common type of layer in Keras is a Dense layer ([Link]),
which is a fully connected neural network layer.
a) In the code below we first add a fully connected layer of neurons containing 512
neurons each with a ReLu activation function (note the dense layer in this
example assumes a flattened input shape).
b) We then add a fully connected Softmax layer.
c) You can add as many layers as you want to your network.

from tensorflow import keras

from tensorflow import keras
from [Link] import layers
from [Link] import layers
# create instance of Sequential model
model = [Link]()
model = [Link]( [
[Link]([Link](512, activation=[Link])) [Link](512, activation=[Link]),
[Link](10, activation=[Link])]
[Link]([Link](10, activation=[Link])) )
 Input Shape
1. Your Keras model needs to know what input shape it
should expect (all subsequent layers can infer the input
shape form the previous layer).

2. For this reason, you can pass an input_shape argument

to the first layer in your model. This is just a tuple of
integers that specify the shape (The presence of a None
as a dimension indicates that any positive number may
be expected along that dimension).

Notice in this example we specify that the first layer should

expect a rank 1 tensor array that contains 784 values (it is
important to understand that we don’t consider the batch size
or number of instances when creating the model).

from tensorflow import keras

from [Link] import layers

model = [Link]( [
[Link](shape=(784,)),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])
])
 Model Compilation
1. Before training a model, you need to set up the details around the loss function, the
optimizer and the metrics you want to use. In Keras this is done via the compile
method. It receives three arguments:

○ A loss function. This is the cost function that the model will try to minimize. It
can be the string identifier of an existing loss function (such as
CategoricalCrossentropy, SparseCategoricalCrossentropy, BinaryCrossentropy or
MeanSquaredError) or it can be an instance of an objective function. Full list of
loss functions available at [Link]

○ An optimizer. This could be the string identifier of an existing optimizer or an

instance of the Optimizer class. You can find a list of the available optimizers at
[Link] (SGD, Adam, RMSProp, etc).

○ A list of metrics. For any classification problem you will want to set this to
metrics=['accuracy']. A metric could be the string identifier of an existing metric
or a custom metric function. Again a full list of metrics is available at
[Link].
 Multi-class Classification on MINST Dataset
 The MNIST database (Modified National Institute of
Standards and Technology database) is a large database
of handwritten digits that is commonly used for
training various image processing systems.

 Each image is a 28 x 28 pixel image, flattened to be a 1-

d tensor of size 784. Each comes with a label.
 Each pixel has a value between 0(B) – 255(W)

 MNIST is available as a TensorFlow Dataset object. It

has
 60,000 instances of training data ([Link]),
 10,000 instances of test data ([Link]), and

 This is a very basic dataset and it is easy to obtain

accuracy values in the high 90’s (often referred to as a
“Hello World” of Deep Learning)
The original shape of the training import tensorflow as tf
feature data is (60000, 28, 28). from [Link] import layers

We need to reshape this to be a

2D data structure. We reshape mnist = [Link]
the training data so that it is now (x_train, y_train),(x_test, y_test) = mnist.load_data()
(60000, 784). x_train, x_test = x_train / 255.0, x_test / 255.0

# Reshape so that each individual row is an image

The training data now contains x_train = x_train.reshape(x_train.shape[0], 784)
60000 rows and 784 columns. x_test = x_test.reshape(x_test.shape[0], 784)
Therefore, the input layer to the
neural network will have 784 model = [Link]( [
values. [Link](shape=(784,)),
[Link](512, activation=[Link]),
Notice when we compile our [Link](10, activation=[Link])
model we select the adam ])
optimizer, accuracy is our metric
and our loss function is [Link](optimizer='adam',
sparse_categorical_crossentropy. loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
So the next question is how do
we train our model?
 Training in Keras
1. The fit function in Keras facilitates the training of our model and allows us to train our
model for a fixed number of iterations.

1. The main arguments are as follows:

○ x: Array of feature training data.
○ y: Array of the target (label) data.
○ batch_size: Integer or None. Number of samples per gradient update. If unspecified,
batch_size will default to 32.
○ epochs: Integer. Number of epochs to train the model. Remember an epoch is an
iteration over the entire x and y data provided.
○ validation_split: A float value between 0 and 1. Fraction of the training data to be
used as validation data. The model will set apart this fraction of the training data, will
not train on it, and will evaluate the loss and any model metrics on this data at the
end of each epoch.
○ validation_data: Data on which to evaluate the loss and any model metrics at the
end of each epoch. The model will not be trained on this data. validation_data will
override validation_split. validation_data could be: - tuple (x_val, y_val)
import tensorflow as tf
Keras
from [Link] import layers

mnist = [Link]
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Reshape so that each individual row is an image

x_train = x_train.reshape(x_train.shape[0], 784)
x_test = x_test.reshape(x_test.shape[0], 784)

model = [Link]( [
[Link](512, activation=[Link], input_shape= (784,)),
[Link](10, activation=[Link])
]) In this example, we have specified a
validation split of 0.1. We will see this
[Link](optimizer='adam', reflected in the output from the
loss='sparse_categorical_crossentropy', training process. Finally once the
model has been trained we can then
metrics=['accuracy']) use the evaluate function to determine
the loss value & metrics values for the
[Link](x_train, y_train, epochs=5, validation_split=0.1) model on the test data.

results = [Link](x_test, y_test) Notice the last line prints out the loss
print (results) on the test data and the accuracy on
the test data.
import tensorflow as tf
Keras
from [Link] import layers

mnist = [Link]
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# Reshape so that each individual row is an image

x_train = x_train.reshape(x_train.shape[0], 784)
x_test = x_test.reshape(x_test.shape[0], 784)

model = [Link]( [
[Link](shape=(784,)),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])
])

[Link](optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']) The summary() provides a useful
overview of the current state of
[Link](x_train, y_train, epochs=5, validation_split=0.1) your model broken down by
layers and the overall number of
print ([Link]()) parameters.
Predict Function import tensorflow as tf
from [Link] import layers
1. Rather than using the Import numpy as np
[Link] function we
can use the [Link] mnist = [Link]
function. (x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
# Reshape so that each individual row is an image
x_train = x_train.reshape(x_train.shape[0], 784)
x_test = x_test.reshape(x_test.shape[0], 784)

model = [Link]( [
[Link](512, activation=[Link], input_shape= (784,)),
[Link](10, activation=[Link])
])
The [Link] function will
output the probability scores for
[Link](optimizer='adam',
each class for each test instance. In
loss='sparse_categorical_crossentropy',
this example the predict function
metrics=['accuracy'])
would have produced an array
10000 * 10
[Link](x_train, y_train, epochs=5, validation_split=0.1)

We can obtain the label with the results = [Link](x_test)

maximum probability using print (results[0])
[Link] (Returns the indices of print ("Predicted Class is ",[Link](results[0]))
Cork Institute of Technology 18
the maximum values along an axis.).
 Metrics import tensorflow as tf
from [Link] import layers
import numpy as np
• It is worth noting that we can still from [Link] import confusion_matrix
use our metrics package from
Scikit Learn to obtain more detail ….
on individual results. ….

[Link](x_train, y_train, epochs=5,

• For example, in this code on the validation_split=0.1)
next slide we can easily generate resultsProb = [Link](x_test)
a confusion matrix for the result
produced by our neural network. # calculate predicted results from probabilities
#(horizontal axis)
results = [Link](resultsProb, axis =1)
print(confusion_matrix(y_test, results))
 Dense Layers
1. As previously mentioned the Dense layer provides the densely-connected NN layer.

2. The following are the most common parameters for the Dense Layer:
○ units: Specify the number of units(neurons) in this layer.
○ activation: Activation function to use. You can use a range of standard activation
functions here such as ReLu, Sigmoid, Tanh, etc (see [Link])
○ use_bias: Boolean, whether the layer uses a bias vector.
○ kernel_initializer: Initializer for the kernel weights matrix . The Initializations
define the way to set the initial random weights of Keras layers (see
[Link]).
○ bias_initializer: Initializer for the bias vector.
○ kernel_regularizer: Regularizer function applied to the kernel weights matrix
bias_regularizer: Regularizer function applied to the bias vector (see
[Link]).
 Dropout and Flatten
• As we have seen Dropout consists of randomly setting a
fraction of output activations to 0 at each update during
training time, which helps prevent overfitting.
• Dropout is as an additional layer added to our model
[Link]
• The dropout layer will apply dropout to the values that
are inputted to it
• The main parameters are
• rate: float between 0 and 1. Fraction of the input
units to drop.

• The Flatten layer is a straight-forward layer, it’s only

purpose is to flatten whatever input it receives into a flat
1D tensor.
• It can be found at [Link]().
• If a flatten layer is the first layer of your network
then you should specify the input_shape
 Visualizing Accuracy and Loss in Keras

1. The fit function returns a History object.

2. It records training metrics for each epoch. This includes

the loss and the accuracy for the training set as well as the
loss and accuracy for the validation dataset, if one is set.

3. History is a dictionary data structure.

4. You can easily see the data available to you by printing out
print([Link]())

dict_keys(['loss', ‘accuracy', 'val_loss', 'val_accuracy'])

 Fashion MNist
1. Fashion-MNIST is a dataset that
contains a training set of 60,000
examples and a test set of 10,000
examples.
2. Each example is a 28x28 grayscale
image, associated with a label
from 10 classes.
3. Fashion-MNIST serves as a
replacement for the original
MNIST dataset for benchmarking
machine learning algorithms,
which is considered too easy.

Label Class
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot
import tensorflow as tf
import [Link] as keras The shape of the training
from [Link] import layers data is (60000, 28, 28)
import [Link] as plt
import numpy as np
The first layer in the
fashion_mnist = [Link].fashion_mnist network we will build will
be [Link],
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images / 255.0 which transforms the
test_images = test_images / 255.0 format of the images from a
2d-array (of 28 by 28 pixels),
model = [Link]([ to a 1d-array of 28 * 28 =
[Link](shape=(28,28)), 784 pixels.
[Link](),
[Link](128, activation=[Link]), [Link](10, activation=[Link]) ])

num_epochs = 20
[Link](optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = [Link](train_images, train_labels, epochs=num_epochs, validation_split=0.1)

[Link]("ggplot")
[Link]()
[Link]([0, num_epochs, 0, 1])
[Link]([Link](0, num_epochs), [Link]["loss"], label="train_loss")
[Link]([Link](0, num_epochs), [Link]["val_loss"], label="val_loss")
[Link]([Link](0, num_epochs), [Link]["accuracy"], label="train_acc")
[Link]([Link](0, num_epochs), [Link]["val_accuracy"], label="val_acc")
[Link]('Model accuracy')
Cork Institute of Technology 24
[Link]('Accuracy')
import tensorflow as tf
import [Link] as keras The shape of the training
from [Link] import layers data is (60000, 28, 28)
import [Link] as plt
import numpy as np
The first layer in the
fashion_mnist = [Link].fashion_mnist network we will build will
be [Link],
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
train_images = train_images / 255.0 which transforms the
test_images = test_images / 255.0 format of the images from a
2d-array (of 28 by 28 pixels),
model = [Link]([ to a 1d-array of 28 * 28 =
[Link](shape=(28,28)), 784 pixels.
[Link](),
[Link](128, activation=[Link]), [Link](10, activation=[Link]) ])

num_epochs = 20
[Link](optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = [Link](train_images, train_labels, epochs=num_epochs, validation_split=0.1)

[Link]("ggplot")
[Link]()
[Link]([0, num_epochs, 0, 1])
[Link]([Link](0, num_epochs), [Link]["loss"], label="train_loss")
[Link]([Link](0, num_epochs), [Link]["val_loss"], label="val_loss")
[Link]([Link](0, num_epochs), [Link]["accuracy"], label="train_acc")
[Link]([Link](0, num_epochs), [Link]["val_accuracy"], label="val_acc")
[Link]('Model accuracy')
Cork Institute of Technology 25
[Link]('Accuracy')
 CIFAR 10 Dataset
▪ When it comes to computer vision and machine learning, the
MNIST dataset is the classic definition of a “benchmark” dataset,
one that is too easy to obtain high accuracy results on, and not
representative of the images we’ll see in the real world.

▪ While the Fashion MNIST dataset is a little more challenging it is

still not really reflective of real world image classification.

▪ For a more challenging benchmark dataset, we can use CIFAR-10,

a collection of 60,000, 32 × 32 RGB images, thus implying that
each image in the dataset is represented by 32 × 32 × 3 = 3,072
integers.

▪ As the name suggests, CIFAR-10 consists of 10 classes, including

airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and
truck.

▪ Each class is evenly represented with 6,000 images per class.

▪ When training and evaluating a machine learning model on CIFAR-

10, it’s typical to use the predefined data splits by the authors and
use 50,000 images for training and 10,000 for testing.
CIFAR 10 Dataset

27
 CIFAR 10 Dataset
1. CIFAR-10 is substantially harder than the MNIST dataset.

2. The challenge comes from the dramatic variance in how objects appear. For
example, we can no longer assume that an image containing a green pixel at a given
(x, y)-coordinate is a frog. This pixel could be a background of a forest that contains a
deer. Or it could be the colour of a green car or truck.

3. These assumptions are a stark contrast to the MNIST dataset, where the network
can learn assumptions regarding the spatial distribution of pixel intensities. For
example, the spatial distribution of foreground pixels of a 1 is substantially different
than a 0 or a 5.

4. This type of variance exhibited in object appearance in CIFAR10 makes applying a

series of fully-connected layers much more challenging.

5. As you’ll see in the following code, standard fully-connected layer networks are not
suited for this type of image classification.

28
import tensorflow as tf
import [Link] as keras
from [Link] import layers
import [Link] as plt
import numpy as np
1. This code is similiar to
what we= have
num_epochs 50 looked
cifar =[Link].cifar10
previously.
(x_train,
2. However, y_train),(x_test, y_test) = cifar.load_data()
in this case
x_train, x_test = x_train / 255.0, x_test / 255
we iterate for a larger
number of epochs.
model = [Link]([
[Link](shape=(32,32,3)),
[Link](), [Link](1024, activation=[Link]),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])])

[Link](optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = [Link](x_train, y_train, epochs=num_epochs, validation_data=(x_test, y_test))

[Link]("ggplot")
[Link]()
[Link]([Link](0, num_epochs), [Link]["loss"], label="train_loss")
[Link]([Link](0, num_epochs), [Link]["val_loss"], label="val_loss")
[Link]([Link](0, num_epochs), [Link]["acc"], label="train_acc")
[Link]([Link](0, num_epochs), [Link]["val_acc"], label="val_acc")
[Link]("Training Loss and Accuracy")
[Link]("Epoch #")
[Link]("Loss/Accuracy")
[Link]
Clearly the network we have trained is not performing well. Validation accuracy flattens out
at approximately 50% accuracy. Also we can clearly see that the network is overfitting on the
training data. Overfitting is beginning to occur as early as epoch 8 or 9.

We could certainly consider optimizing our hyperparameters further and introducing

regularization and dropout in order to mitigate against overfitting. However, our overall
performance will likely not improve very much.
 Saving and Loading Keras Models to Disk
1. A Keras model consists of multiple components:

• The architecture, or configuration, which specifies what layers the model

contain, and how they're connected.
• A set of weights values (the "state of the model").
• An optimizer (defined by compiling the model).
• A set of losses and metrics (defined by compiling the model).

2. The Keras API saves all of these pieces together in a single format, marked by
the .keras extension. This is a zip archive consisting of the following:

• A JSON-based configuration file ([Link]): Records of model, layer, and other

configuration setting.
• A H5-based state file, such as [Link].h5 (for the whole model), with
directory keys for layers and their weights.
• A metadata file in JSON, storing things such as the current Keras version.
import tensorflow as tf
from tensorflow import keras
from [Link] import layers

cifar = [Link].cifar10
(x_train, y_train),(x_test, y_test) = cifar.load_data()
x_train, x_test = x_train / 255.0, x_test / 255

model = [Link]([
[Link](shape=(32,32,3)),
[Link](),
[Link](64, activation='relu'),
[Link](32, activation='relu'),
[Link](10, activation='softmax') # Output layer for 3 classes
])

# Compile the model

[Link](
optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy']
)

[Link]()
num_epochs = 2
history = [Link](x_train, y_train, epochs=num_epochs, validation_data=(x_test, y_test))
 Saving and Loading Keras Models to Disk
1. [Link]() The saved .keras file contains:

• The model's configuration (architecture)

• The model's weights
• The model's optimizer's state (if any)

2. [Link].load_model will allow us to load back up a file and continue using

as normal.

# Save the entire model to a .keras zip archive

[Link]('[Link]')
print("Model saved to '[Link]’”)

reloaded_model = [Link].load_model('[Link]')

print("Model reloaded successfully!")

reloaded_model.summary()

results = [Link](x_test, y_test)

 Checkpointing Models
• In the example below we leave the model train for a significant number of iterations.
• We can clearly identify the point where the model begins to overfit.
• For example below we can see the model is overfitting after approx. 5 epochs.
• However, one significant problem of course is that the model we end up with is the model after the
final iteration. In the case below this model is seriously overfitting on the training data. Clearly it would
be preferable to capture the model that has the lowest validation loss (which is circled in green below)
rather than the model that will be returned at the end of the training process (highlighted in purple).
 Checkpointing Models
• Checkpointing is a process that allows us to save the current state of a model to disk each
time there is an improvement during the training process.
• An improvement above means an increase in accuracy or a decrease in loss.
• In the example below, as we train the model we could save it’s weights if the new model
(for the current epoch) achieves a lower validation loss than any model we have seen in
the training process so far.

• Checkpointing is also useful

if you have a large model
that takes a very long time
to train. For example, if
something goes wrong
during the training process
then you don’t have to
restart from scratch.
 What is a Callback?
A callback is a set of functions that can be provided to the fit method. These functions
can then be applied at specific stages of the training procedure.

• For example, one callback that is automatically called is the history callback. After
each epoch the history callback is called and it updates the history object with the
loss and accuracy for the current epoch.

• Another alternative is an early stopping callback that will terminate the training
process if a monitored metric such as validation loss has stopped improving.

• The learning rate scheduler callback allows to update the learning rate used after a
fixed epoch or a certain number of epochs.

• You can pass a list of callbacks (using the keyword argument callbacks) to the .fit()
method of the Sequential or Model classes. A full list of callbacks is available here.
 Checkpointing Models – What are Callbacks
There is also a ModelCheckpoint callback that allows you to checkpoint your model during the
training process.

The following are the main arguments for this class.

• filepath: A string that can contain formatting options such as the epoch number. For example, the
following is a common filepath( weights.{epoch:02d}-{val_loss:.2f}. weights.h5)
• For large models this can consume a lot of space!!!

• monitor: (typically ‘val_loss’or‘val_accuracy’)

• mode: Should be minimizing or maximizing the monitor value (typically either ‘min’ or ‘max’)

• save_weights_only: if True, then only the model's weights will be saved. The extension of the
name file should be end in .weights.h5 (if False then it should be .keras)

• save_best_only: If this is set to true then it will only save the model weights for the current epoch
if its metric value is better than what has gone before. However, if you set save_best_only to false
it will save every model after each epoch (regardless of whether that model was better then
previous models or not).
 model = [Link]([
Notice here we create an
[Link](shape=(32, 32, 3)),
instance of a
[Link](),
ModelCheckpoint, which
[Link](1024, activation=[Link]),
will monitor the validation
[Link](512, activation=[Link]),
loss and only save the state
[Link](10, activation=[Link])])
of the current model if the
current validation is lower
fname = "weights.{epoch:02d}-{val_loss:.2f}.weights.h5”
than an preceding loss
value.
checkpoint = [Link](fname, monitor="val_loss",
mode="min", save_best_only=True, save_weights_only=True, verbose=1)

[Link](optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy’])

history = [Link](x_train, y_train, epochs=NUM_EPOCHS,

validation_data=(x_test, y_test), callbacks=[checkpoint])
Associated
Colab
Notebook
 import tensorflow as tf
import [Link] as plt
import [Link] as keras
from [Link] import layers
import numpy as np
In this example we
create the architecture cifar = [Link].cifar10
of the model again from (x_train, y_train),(x_test, y_test) = cifar.load_data()
scratch and load up the x_train, x_test = x_train / 255.0, x_test / 255
weights saved during the
training process earlier. best_model = [Link]([
[Link](shape=(32, 32, 3)),
[Link](),
[Link](1024, activation=[Link]),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])])

Associated best_model.load_weights("[Link].h5")
Colab
Notebook [Link](optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

results = [Link](x_test, y_test)

 Early Stopping
The early stopping callback will terminate the training process if a
monitored metric such as validation loss has stopped improving.

The following are the main arguments for this class.

• monitor: Quantity to be monitored.

• min_delta: Minimum change in the monitored quantity to qualify as
an improvement, i.e. an absolute change of less than min_delta, will
count as no improvement.
• patience: Number of epochs with no improvement after which training
will be stopped.
• mode: One of {"auto", "min", "max"}.
• restore_best_weights: Whether to restore model weights from the
epoch with the best value of the monitored quantity. If False, the
model weights obtained at the last step of training are used. An epoch
will be restored regardless of the performance relative to the baseline.
If no epoch improves on baseline, training will run for patience epochs
and restore weights from the best epoch in that set.
best_model = [Link]([
[Link](shape=(32, 32, 3)),
[Link](),
[Link](1024, activation=[Link]),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])])

earlyStop = [Link](monitor='val_loss', patience=4, mode='min',

restore_best_weights=True)

NUM_EPOCHS = 50

[Link](optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

history = [Link](x_train, y_train, epochs=NUM_EPOCHS, validation_data=(x_test, y_test),

callbacks=[earlyStop])

final_num_epochs = len([Link]['val_loss'])

[Link]("ggplot")
[Link]()
[Link]([Link](0, final_num_epochs), [Link]["loss"], label="train_loss")
[Link]([Link](0, final_num_epochs), [Link]["val_loss"], label="val_loss")
[Link]([Link](0, final_num_epochs), [Link]["accuracy"], label="train_acc")
[Link]([Link](0, final_num_epochs), [Link]["val_accuracy"], label="val_acc")
 Learning Rate Scheduler
After each epoch the Learning Rate Scheduler will call a function named schedule which
will modify the current learning rate.

The following are the main arguments for this class.

• schedule: a function that takes an epoch index (integer, indexed from 0) and current
learning rate (float) as inputs and returns a new learning rate as output (float).
import tensorflow as tf
import [Link] as plt
import [Link] as keras
from [Link] import layers
import numpy as np

def scheduler(epoch, lr):

if epoch>0 and epoch%2==0:
return lr * 0.5
else:
return lr

NUM_EPOCHS = 10

cifar = [Link].cifar10
(x_train, y_train),(x_test, y_test) = cifar.load_data()
x_train, x_test = x_train / 255.0, x_test / 255

model = [Link]([
[Link](shape=(32, 32, 3)),
[Link](),
[Link](1024, activation=[Link]),
[Link](512, activation=[Link]),
[Link](10, activation=[Link])])
learning_rate_callback = [Link](schedule = scheduler)

opt= [Link](learning_rate=0.1)
[Link](optimizer=opt, loss='sparse_categorical_crossentropy', metrics=['accuracy'])

print ("Original Learning Rate ", round([Link].learning_rate.numpy(), 5))

history = [Link](x_train, y_train, epochs=NUM_EPOCHS, validation_data=(x_test, y_test),

callbacks=[learning_rate_callback])

final_num_epochs = len([Link]['val_loss'])
print ("Final Learning Rate ", round([Link](), 5))

[Link]("ggplot")
[Link]()
[Link]([Link](0, NUM_EPOCHS), [Link]["loss"], label="train_loss")
[Link]([Link](0, NUM_EPOCHS), [Link]["val_loss"], label="val_loss")
[Link]([Link](0, NUM_EPOCHS), [Link]["accuracy"], label="train_acc")
[Link]([Link](0, NUM_EPOCHS), [Link]["val_accuracy"], label="val_acc")
[Link]("Training Loss and Accuracy")
[Link]("Epoch #")
[Link]("Loss/Accuracy")
[Link]
Original Learning Rate 0.1
Epoch 1/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 7s 4ms/step - accuracy: 0.2653 - loss: 2.0295 - val_accuracy:
0.3824 - val_loss: 1.7026 - learning_rate: 0.1000
Epoch 2/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.3874 - loss: 1.7043 - val_accuracy:
0.3876 - val_loss: 1.7230 - learning_rate: 0.1000
Epoch 3/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 10s 3ms/step - accuracy: 0.4544 - loss: 1.5220 - val_accuracy:
0.4158 - val_loss: 1.6477 - learning_rate: 0.0500
Epoch 4/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.4774 - loss: 1.4605 - val_accuracy:
0.4681 - val_loss: 1.4880 - learning_rate: 0.0500
Epoch 5/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.5110 - loss: 1.3686 - val_accuracy:
0.5022 - val_loss: 1.4035 - learning_rate: 0.0250
Epoch 6/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.5306 - loss: 1.3244 - val_accuracy:
0.4770 - val_loss: 1.4414 - learning_rate: 0.0250
Epoch 7/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 4s 3ms/step - accuracy: 0.5470 - loss: 1.2669 - val_accuracy:
0.5265 - val_loss: 1.3483 - learning_rate: 0.0125
Epoch 8/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.5593 - loss: 1.2331 - val_accuracy:
0.5232 - val_loss: 1.3542 - learning_rate: 0.0125
Epoch 9/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.5701 - loss: 1.2105 - val_accuracy:
0.5338 - val_loss: 1.3237 - learning_rate: 0.0063
Epoch 10/10
1563/1563 ━━━━━━━━━━━━━━━━━━━━ 5s 3ms/step - accuracy: 0.5804 - loss: 1.1836 - val_accuracy:
0.5291 - val_loss: 1.3252 - learning_rate: 0.0063
Final Learning Rate 0.00625
Deep Learning

Deep Learning
Lecture: Keras – Functional API

Ted Scully
 Non Standard Models Types
1. So far, we have been using the Sequential API in Keras, which is suited for creating
standard models in a linear fashion. We create an instance of our Sequential model
and sequentially add individual layers. However, many model architectures don’t
follow this basic structure.
2. For example, some models can be designed to take a diversity of different input
types.
3. Let’s assume we wanted to predict the selling price of a secondhand car we might
have an image of the car as well as numerical data such as standard feature data
such as age, make, etc. We might push the feature data through a dense network
and the picture through a convolution network, the outputs of these networks are
then fused and pushed through a final network in order to make some prediction.
 Non Standard Models Types
1. Some models can be designed so that they have multiple outputs.
2. For example, if we do object localization we want our model to
predict both the bounding box of an object and the class of object
within the bounding box.
3. More specifically we might want out model to output the x,y pixel
location of an object, the height and width of the bounding box
and also the class of object contain therein.
 Inception Network
The model below is taken from the original paper on inception
networks (Going Deeper with Convolutions 2014).
This is the original inception network (also called GoogLeNet)

You may notice that there is a repetitive pattern in the model.

module. We take a closer look at this over the next few slides.
 ResNet
The original ResNet (Residual Network) architecture, introduced by Kaiming He et al.
in 2015, was designed with varying depths, starting at 18 layers and scaling up to 152
layers in the main ImageNet studies..
 Functional API Keras
1. The Sequential API in Keras, creates models in a linear fashion and cannot be
used to create the non-standard models we saw in the previous slides.

2. The functional API in Keras allows us more flexibility to create non-standard

models. It enables us to build models that exhibit a non-linear topology,
shared layers as well as models that might have multiple inputs and outputs.

3. There are three basic steps to creating a model using the functional API:

• Create an input Keras tensor

• Connect layers by passing the output of one layer to the next layer.

• Create an instance of the model specifying the input and output layers
that will populate the model.
 Layers are callable
1. Before we talk about the functional API in more detail let’s revisit the idea of a Keras layer.
2. As we have seen before a layer is a fundamental component of a neural networks in
Keras.
3. An instance of a layer is callable. Notice below we create a basic dense layer. We
subsequently call the layer and pass it a tensor. We can then collect the resulting tensor in
outputs.

import tensorflow as tf
from [Link] import layers

basicLayer = [Link](16, activation='relu’)

#create a tensor
inputs = [Link](shape=(4, 5))

# call the dense layer and pass it a tensor, results in stored in outputs
outputs = basicLayer(inputs)

print ([Link]) (4, 16)

Functional API Keras – Step 1. Creating the Input
With the functional API the first step is to create an input that specifies the shape of
the incoming data instance.

The Input object is not a layer. Instead, it returns a symbolic tensor (think of it as a
placeholder).

In the code example we will look at the Functional API implementation of the same
model we implemented previously (with the Sequential API) for tackling the Fashion
MNIST problem.

# First step is creating a input layer that specifies the incoming data instance shape

# create out input tensor

input = [Link](shape=(28,28))
 Functional API Keras – Step 2. Calling Layers
The second step is to create individual layers and to feed the output from one layer to the
input of the next layer (think of it as forming a connected chain/graph).

In the example, we create a flatten layer (because this is just a densely connected network we
are going to flatten the incoming instance). As we have seen previously layers are callable,
which means I can call this layer and pass it an incoming tensor, which will return an output
tensor. This is exactly what we do below.
We take the flat layer and call it by pass it the input tensor. This returns a resulting tensor called
layer1 in this case. (In essence we have added the first node to our network graph)

# First step is creating a input tensor that specifies the incoming data instance shape
input = [Link](shape=(28,28))

# create a layer (the purpose of which is to flatten incoming tensor)

layer1 = [Link]()

# call the flatten layer (just created) and pass it incoming tensor
layer1_output = layer1(input)
(None, 784)
print (layer1_output.shape)
Functional API Keras – Step 2. Calling Layers
We can then continue in this same way. Notice
below we create a densely connected layer and
call it by passing it the tensor output of the
previous flat layer. We then create a Softmax layer
and call it by passing it the tensor outputted from
the densely connected layer.

input = [Link](shape=(28,28))

layer1 = [Link]()
layer1_output = layer1(input)

layer2 = [Link](128, activation=[Link])

layer2_output = layer2(layer1_output)

layer3 = [Link](10, activation=[Link])

layer3_output = layer3(layer2_output)
Functional API Keras – Step 3 Create a Model
Once we have defined our
input = [Link](shape=(28,28)) graph the final step is to create
an instance of a Model.
layer1 = [Link]()
layer1_output = layer1(input) When we create the model we
must specify the inputs and
layer2 = [Link](128, activation=[Link]) outputs from the graph of
layer2_output = layer2(layer1_output) layers.

layer3 = [Link](10, activation=[Link]) Once this step is complete we

layer3_output = layer3(layer2_output) can compile and training out
model in the usual way. Full
model = [Link](inputs=input, outputs=layer3_output) code available here.

NUM_EPOCHS= 10

[Link](optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

history = [Link](train_images, train_labels, epochs=NUM_EPOCHS, validation_split=0.1)

Functional API Keras – Shorter Representation
Please note that you can also commonly see the functional API applied in an a
more succinct (albeit more confusing form) where the layer is created and
called in the same line.

# Shortened example

input = [Link](shape=(28,28))

layer1_output = [Link]()(input)

layer2_output = [Link](128, activation=[Link])(layer1_output)

layer3_output = [Link](10, activation=[Link])(layer2_output)

model = [Link](inputs=input, outputs=layer3_output)

Functional API Keras - Branching
In the next example we are going
to build the convolutional
network shown with internal
branching (again we use the
Fashion MNIST for illustration).

The input image is fed into two

separate convolutional networks.
The output of each network is
pooled, flattened and then
merged.

The resulting tensor is pushed

through a densely connected
network and then finally a
softmax layer.
Full code here.
 Functional API Keras - Branching
input = [Link](shape=(28,28,1))

covn1_output = [Link].Conv2D(32, kernel_size=4, activation='relu')(input)

pool1_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn1_output)
flat1_output = [Link]()(pool1_output)

covn2_output = [Link].Conv2D(64, kernel_size=2, activation='relu')(input)

pool2_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn2_output)
flat2_output = [Link]()(pool2_output)

merge_output = [Link]([flat1_output, flat2_output])

dense1_output = [Link](32, activation='relu')(merge_output)

final_output = [Link](10, activation=[Link])(dense1_output)

model = [Link](inputs=input, outputs=final_output)

 Functional API Keras - Branching

In this example, we modify the architecture slightly Here,

we add an additional convolutional and pooling layer.
It’s important to note that previously we flattened the
tensors before we concatenated. In this example, we just
concatenate two 3D feature maps.
Full code here.
 Functional API Keras - Concatenate

In the code we saw on the previous slide we used concatenate to merge two linear
tensors together. However, there are many cases where we want to take two multi-
dimensional structures and concatenate them together.

(14, 14, 16)

(14, 14, 32)

As long as the resulting

data structures different
only along one dimension
then concatenate will
append along that
dimension.
 Functional API Keras - Branching
input = [Link](shape=(28,28,1))

covn1_output = [Link].Conv2D(16, kernel_size=3, activation='relu', padding="same")(input)

pool1_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn1_output)

covn2_output = [Link].Conv2D(32, kernel_size=5, activation='relu', padding="same")(input)

pool2_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn2_output)
pool1_output shape (None, 14, 14, 16)
print ("pool1_output shape ", pool1_output.shape) pool2_output shape (None, 14, 14, 32)
print ("pool2_output shape ", pool2_output.shape) Shape after merge (None, 14, 14, 48)

merge_output = [Link]([pool1_output, pool2_output])

print("Shape after merge ", merge_output.shape)

covn3_output = [Link].Conv2D(16, kernel_size=3, activation='relu')(merge_output)

pool3_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn3_output)

flat1_output = [Link]()(pool3_output)
dense1_output = [Link](32, activation='relu')(flat1_output)
final_output = [Link](10, activation=[Link])(dense1_output)

model = [Link](inputs=input, outputs=final_output)

Functional API Keras – Multiple
Inputs
It may be the case that rather than
branching within the network you
originally have two or more inputs
leadings to different parts of the
graph. This is easy to achieve, we can
specify a list of inputs (or outputs
when we create our Model)

On the right this network takes in two

separate images, feeds each image
into a different convolutional, pooling
and flatten layer.

The results are then concatenated and

pushed through a dense layer,
followed by a Softmax layer.

Full code here.

 Functional API Keras – Multiple Inputs
• Notice we create two separate inputs and specify these as the list of
inputs when creating the model.
input1 = [Link](shape=(28,28,1))
input2 = [Link](shape=(28,28,1))

covn1_output = [Link].Conv2D(32, kernel_size=4, activation='relu')(input1)

pool1_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn1_output)
flat1_output = [Link]()(pool1_output)

covn2_output = [Link].Conv2D(64, kernel_size=2, activation='relu')(input2)

pool2_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn2_output)
flat2_output = [Link]()(pool2_output)

merge_output = [Link]([flat1_output, flat2_output])

dense1_output = [Link](32, activation='relu')(merge_output)
final_output = [Link](10, activation=[Link])(dense1_output)

model = [Link](inputs=[input1, input2], outputs=final_output)

 Functional API Keras – Plugging Models Together
A model is callable in the same way as a layer is callable

• I can use a model in the same manner as I can use an

individual layer.
• In other words I can create a model by connecting together
multiple pre-existing models together as part of a larger
model.
• Code is available here.

from [Link] import Image

from [Link] import plot_model

#FIRST MODEL
input1 = [Link](shape=(28,28,1))

covn1_output = [Link].Conv2D(32, kernel_size=4, activation='relu')(input1)

pool1_output = [Link].MaxPooling2D(pool_size=(2, 2))(covn1_output)
flat1_output = [Link]()(pool1_output)

covd_model = [Link](inputs=input1, outputs=flat1_output)

Functional API Keras – Plugging Models Together
# SECOND MODEL
input2 = [Link](shape=(4608,))
dense1_output = [Link](32, activation='relu')(input2)
final_output = [Link](10, activation=[Link])(dense1_output)

dense_model = [Link](inputs=input2, outputs=final_output)

In this example we create a model that consists of a

dense layer of neurons followed by a Softmax layer
Functional API Keras – Plugging Models Together
• Notice in this code we use the functional API to link
together the two models we created in the previous
slides.
• A model is callable in the same way that an individual
layer is callable.

• We feed the input tensor into the first convolutional

model, the tensor outputted from this (covd_output) is
then inputted to the densely connected model.

#MERGED MODEL
input = [Link](shape=(28,28,1))

covd_output = covd_model(input)
final_output = dense_model(covd_output)

model = [Link](inputs=input, outputs=final_output)