Advanced Keras API
Prof. Kuan-Ting Lai
2019/4/2
Going Beyond Sequential Model
Multi-input Model Multi-output Model
Inception Module
• Christian Szegedy et al., “Going
Deeper with Convolutions,”
CVPR, 2014.
[Link]
Residual Connection
• Kaiming He et al., “Deep Residual
Learning for Image Recognition,” CVPR
(2015), [Link]
Functional API
from keras import Input, layers
input_tensor = Input(shape=(32,))
dense = [Link](32, activation='relu') A layer is a function
output_tensor = dense(input_tensor)
A layer may be called on a
tensor, and it returns a tensor
Functional API vs. Sequential Model
• Create a Model object using only an input tensor and an output tensor
input_tensor = Input(shape=(64,))
x = [Link](32, activation='relu')(input_tensor)
x = [Link](32, activation='relu')(x)
output_tensor = [Link](10, activation='softmax')(x)
model = Model(input_tensor, output_tensor)
Functional API vs. Sequential Model
• Create a Model object using only an input tensor and an output tensor
input_tensor = Input(shape=(64,))
x = [Link](32, activation='relu')(input_tensor)
x = [Link](32, activation='relu')(x)
output_tensor = [Link](10, activation='softmax')(x)
model = Model(input_tensor, output_tensor)
seq_model = Sequential()
seq_model.add([Link](32, activation='relu’,
input_shape=(64,)))
seq_model.add([Link](32, activation='relu'))
seq_model.add([Link](10, activation='softmax'))
Question-answering Model
• Two inputs:
1. A natural-language question
2. Reference text snippet (such as a news
article)
• One output: answer
One-word answer obtained via a
SoftMax over some predefined vocabulary
Multi-output Model
• Predict age, income, gender based on the contents of posts
Directed Acyclic Graph of Layers
• Graph can’t have cycles!
The Purpose of 1x1 Convolutions
• Reduce the channel dimension
Residual Connection
from keras import layers
x = ...
y = layers.Conv2D(128, 3,
activation='relu',
padding='same')(x)
y = layers.Conv2D(128, 3,
activation='relu',
padding='same')(y)
y = layers.Conv2D(128, 3,
activation='relu',
padding='same')(y)
y = [Link]([y, x])
Vanishing Gradients in Deep Learning
• A signal becomes smaller after propagated through multi-
layers, and may be lost (vanished)
• Solutions:
LSTM: using carry track to propagate signal parallel to main track
Residual: simple jump connection
Monitoring Model Training
• Model checkpoint saving
Saving the current weights of the model during training
• Early stopping
• Dynamically adjusting parameters
Adaptive learning rate during training
• Visualizing the model and data
Using Callbacks
• EarlyStopping - interrupts training when accuracy has stopped
improving for more than one epoch
• ModelCheckpoint - Saves the current weights after every epoch
from keras import callbacks
callbacks_list = [
[Link](monitor='acc', patience=1),
[Link](filepath='my_model.h5', monitor='val_loss',
save_best_only=True)
]
[Link](optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
[Link](x, y, epochs=10, batch_size=32, callbacks=callbacks_list,
validation_data=(x_val, y_val))
ReduceLROnPlateau Callback
• factor – the learning rate is multiplied by factor after pre-defined epochs
• patience – epochs before callback is triggered
callbacks_list = [
[Link](monitor='val_loss', factor=0.1, patience=10)
]
[Link](x, y, epochs=10, batch_size=32, callbacks=callbacks_list,
validation_data=(x_val, y_val))
Implement Your Own Callback Function
• Inherit [Link] and implement any number of the
following methods
on_epoch_begin
on_epoch_end
on_batch_begin
on_batch_end
on_train_begin
on_train_end
Tensor Board
• Add TensorBoard callback function and assign log_dir
callbacks = [
[Link](log_dir='my_log_dir', histogram_freq=1,
embeddings_freq=1)
]
history = [Link](x_train, y_train, epochs=20, batch_size=128,
validation_split=0.2, callbacks=callbacks)
• Run command => $ tensorboard --logdir=my_log_dir
TensorBoard: Accuracy and Loss
TensorBoard: Activation Histograms
TensorBoard: Word-embedding Visualization
TensorBoard: Network Graph Visualization
Keras plot_model
from [Link] import plot_model
plot_model(model, show_shapes=True,
to_file='[Link]')
Batch Normalization
• Sergey Ioffe and Christian Szegedy, “Batch Normalization: Accelerating
Deep Network Training by Reducing Internal Covariate Shift,” ICML,
2015 ([Link]
• Normalizing data after every transformation
• Enhance back propagation
• Some deep networks can only be trained with batch normalization
conv_model.add(layers.Conv2D(32, 3, activation='relu'))
conv_model.add([Link]())
dense_model.add([Link](32, activation='relu'))
dense_model.add([Link]())
Batch Renormalization
• Sergey Ioffe, “Batch Renormalization: Towards Reducing Minibatch
Dependence in Batch-Normalized Models,” 2017,
[Link]
• Günter Klambauer et al., “Self-Normalizing Neural Networks,” NIPS,
2017, [Link] .
Depthwise Separable Convolution
• Separating the learning of spatial
features and channel-wise
features
• Less parameters, slightly better
accuracy
• Francois Chollet, “Xception: Deep
Learning with Depthwise
Separable Convolutions.”
References
• Francois Chollet, “Deep Learning with Python,” Chapter 7