C H A P T E R

2
Neural networks

OUTLINE
• Principles of neural networks
• Optimization algorithms for training neural networks
• Getting to know the Kaggle platform
• Using a fully connected neural network to complete a
house price-prediction system
• Getting to know TensorBoard
• The problem of overfitting

2.1 Introduction to neural networks

2.1.1 A brief history of neural networks

In 1943, Warren McCulloch and Walter Pitts [1] introduced a computational model based on a threshold logic algo-
rithm, paving the way for the development of artificial neural networks (ANNs). Inspired from McCullock and Pitts
[1], numerous methods have been proposed for neural networks which can be categorized into two approaches: (1)
focusing on biological process [2,3] and (2) focusing on application [4,5]. However, around 1980–2000, simpler models
such as linear classifiers or support vector machines (SVMs) [6,7] for classification and regression analysis became
more popular. The reason can be explained as follows. Deep networks with many intermediate layers called hidden
layers can obtain better performance than that of shallow neural networks with few hidden layers, as shown in Fig. 2.1.
In a deep network, the hidden layers extract features from input data and these features are used on the following
layers for computation. Therefore, the more hidden layers, the more detailed features of data can be obtained. For
example, a human face recognition model takes face images as inputs, and then the first hidden layer takes raw data
from the input layer and extracts simple features such as edges, lines, and so on. In the following hidden layers, the
high-level specific features of the input image such as nose, eyes, hair, and others can be extracted for face recognition.
Unfortunately, machine learning algorithms like gradient-based learning methods and backpropagation [8] do not
work well for deep neural networks because of a vanishing gradient problem during the training process. Further-
more, since deep neural networks have high computational cost and memory footprint, they require large datasets
for training and development.

Principles and Labs for Deep Learning 27 Copyright © 2021 Elsevier Inc. All rights reserved.
[Link]
28 2. Neural networks

FIG. 2.1 The architecture of a neural network.

From 2000 to today, the development of various aspects of computer technology and machine learning algorithms
has made it possible to build and train deep neural networks, and there have been major breakthroughs.
▪ Computing ability
In the past, central processing units (CPUs) were used to perform the calculation of neural networks, but now
graphic processing units (GPUs) and tensor processing units (TPUs) are used because of their improved performance.
In 2012, Krizhevsky et al. used a NVIDIA GPU to train an eight-layer network called “AlexNet” [9]. After winning the
2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) competition [10], everyone realized the amazing
computing power of GPUs and the abilities of deep neural networks.
▪ Dataset and storage equipment
A deep neural network cannot be trained efficiently without sufficient training data. Because data storage devices
were expensive and had limited capacity, large datasets were impossible 20–30 years ago. Nowadays, with the devel-
opment of science and technology, the price of storage devices has decreased, and Internet and cloud devices, such as
Kaggle ([Link]/) and TensorFlow hub ([Link]/hub) are prevalent; there are many available
databases for users to download, research, and develop. Thus the problems of lacking of large datasets and data stor-
age devices have been solved.
▪ Activation function
Sigmoid was a very popular activation function for neural networks. However, using the sigmoid activation func-
tion in hidden layers of deep networks with random weight initialization leads to vanishing gradients during training,
which makes the weights of the networks difficult to update, resulting in ineffective training. In 2011, Glorot et al. [11]
introduced the rectified linear unit (ReLU) function, which has proved effective at improving the problem of vanishing
gradients and the learning speed of various deep neural networks. In 2015, parametric rectified linear unit (PReLU)
activation was introduced by He et al. [12], allowing for effectively investigating and training deeper and wider deep
neural network models.

2.1.2 Principle of neural networks

A neural network is composed of three types of layers: input layer, hidden layer, and output layer. Each layer in the
network has many neurons for connection and computation, as shown in Fig. 2.2. The neurons of the input layer, hid-
den layer, and output layer are called the input neuron, hidden neuron, and output neuron, respectively. The input
layer provides initial data from outside to the network without any computations for further processing by hidden
 2.1 Introduction to neural networks 29
layers. Hidden layers are located between the input and output layers in the neural network and are responsible for
performing computation on the input data through hidden neurons and passing the results to the output layer. The
output layer takes the results from the last hidden layer as the input data and uses its neurons to compute and produce
the final result of the network.

Connections

(1)
a1

(2)
x1 a1
(1)
a2
(2)
x2 a2
(1)
a3
Neurons

Input layer Hidden layer Output layer

FIG. 2.2 Example of a two-layer neural network. Note that the number of layers of the neural network is equal to the number of hidden layers plus
one, a(li ) represents the output of i-th neuron in l-th layer, the biases are hidden.

Sigmoid Function
1.0
1 b
s (z) =
0.8 1 + e–z

0.6
w1 z = z1w1 + z2w2 + b a
s x1
0.4
a = s (z)
w2
0.2

0.0 x2
–6 –4 –2 0 2 4 6
z
FIG. 2.3 Illustration of a neuron of a neural network. xi, wi, and b represent inputs, weights, and bias of the neuron, respectively. σ is an activation
function.

Except for input neurons, each neuron of the neural network can receive one or more inputs with separate weights,
and these inputs are summed to produce an output. Normally, after summing the inputs, the resulting sum is sent to a
non-linear function, which is known as the activation function for generating output, as shown in Fig. 2.3. The common
activation functions include ReLU, sigmoid, and tanh, which we discuss in later chapters.
When the neural network adopts activation functions to perform a nonlinear transformation of the inputs, it can
help the neural networks describe and learn complex tasks.

2.1.3 Training neural networks

Training a neural network is the process of using training data to find the appropriate weights of the network for
creating a good mapping of inputs and outputs. As shown in Fig. 2.4, the training procedure for a neural network
consists of four parts: preparing the dataset, building a network model, loss function, and optimization.
30 2. Neural networks

FIG. 2.4 Schematic diagram of the neural network training procedure.

1. Preparing the dataset

The dataset for training neural networks is divided into three types of data: training data, validation data, and test
data.
▪ Training data: A set of examples, which is used for fitting the weights of connections between neurons in neural
networks. The training data often contains pairs of samples (input sample, ground truth label). The ground truth
label is also called the expected output.
▪ Validation data: A set of examples, which is used to estimate the model fit during tuning the hyperparameters of the
network model, such as the number of hidden layers, the number of neurons in each layer, and so on.
▪ Test data: A set of examples, which is used to estimate the final network model fit on the training data. For most
competitions, training data and validation data are published, while the test data is kept as the basis for the final
evaluation of the model.

Supplementary explanation
The test data is used to evaluate the accuracy and generalization ability of a neural network in the real world. If the hyper-
parameters of the neural network are directly tuned through the results on the test data, the network can often obtain good
performance on both training data and test data, but it will perform poorly when applied to the real world [13]. In order to
avoid this problem, the validation data is used instead of the test set, so that the performance of the model on the test data will
be more in line with that of the model operating in the real world.

2. Building a network model

As introduced in Section 2.1.2, neural networks consist of three main components: input layers, hidden layers, and
output layers. For different tasks, the network models are built with different hyperparameters, such as the number of
hidden layers, the number of neurons in each layer, and so on. Because the choice of hyperparameters directly affects
the training results of the network models, we provide the TensorBoard tuning toolkit in Chapter 7 to assist in adjust-
ing these hyperparameters.
In the neural network, if each neuron in a layer is connected to all neurons in the following layer, this network is
called a fully connected neural network (FCNN). Fig. 2.5 shows a FCNN with three hidden layers that employ the
ReLU activation function for non-linear transformation. Because of full connection, all the combinations of the infor-
mation from the previous layer can be used to compute in the next layer; this helps the FCNN learn better input data
 2.1 Introduction to neural networks 31
[14]. However, FCNNs are often extremely computationally expensive and are challenged by the problem of overfit-
ting, a phenomenon of modeling the training data too well during the training process, which leads to poor perfor-
mance of the network, especially deep neural networks [15]. In the next sections of this chapter, we discuss the
application of an FCNN for house price-prediction, as well as provide the solution to the overfitting problem.

FIG. 2.5 Fully connected neural network.

3. Loss function
A neural network is trained through an optimization process that uses a loss function to calculate an error between
the predicted value of the model and the expected output. For the different purposes of training, the optimization pro-
cess may minimize or maximize the loss function, which means it needs to evaluate a suitable solution such as a set of
parameters to reach the lowest or highest error score, respectively. Typically, minimizing the loss function is applied
when training the neural networks. There are many loss functions and it can be challenging to choose a suitable one for
a specific problem. The following introduces commonly used loss functions for main problems in machine learning,
including the linear regression problem, binary classification problem, and multiclass classification problem.
▪ Linear regression problem: Neural networks are designed with one neuron in the output layer for each possible
desired value. An example of a regression model is the house price-prediction model, which is based on information
such as the number of bedrooms, bathrooms, and floors in the house, the age of the house, and so on, to predict the
price of a house. The most common loss functions for linear regression models are mean squared error (MSE) and
mean absolute error (MAE), which calculate the average of the squared or absolute difference between the predicted
value and the expected output.
▪ Binary classification problem: Neural networks are designed with one neuron in the output layer for predicting an
input sample as belonging to one of two classes. Medical testing to determine whether a person has a certain disease
or not is a typical binary classification problem. For training binary classification models, binary cross-entropy loss
function, also known as log loss function, is commonly used. We discuss this in more detail in Chapter 3.
32 2. Neural networks

▪ Multiclass classification problem: Neural networks are designed with many neurons in the output layers, where
each neuron with softmax activation functions is responsible for predicting one class. Categorical cross-entropy loss
function, also known as softmax loss function, is widely used for training multiclass classification models. We
discuss categorical cross-entropy in more detail in Chapter 4.
Because house price-prediction introduced in this chapter is a linear regression problem, MSE or MAE can be
employed as loss functions. Here, we present the formulas and differences of these two loss functions.
MSE:
XN  2
y  ^yi
i¼1 i
MSE ¼
N
MAE:
XN  
yi  ^y 
i
MAE ¼ i¼1
N
where y is the expected output, ^y is the predicted value of the neural network model, and N is the amount of data in a
batch.
Both MSE and MAE calculate the error between the predicted value (y_pre) of the model and the expected output
(y_true), but MSE calculates the average of squared error and MAE calculates the average of absolute error. Fig. 2.6
presents a comparison between MSE and MAE. As shown, when the value of y_true - y_pre is between 1 and 1, the
MSE loss value is lesser than that of MAE. In contrast, the MSE loss value is greater than MAE loss value when the
value of y_true - y_pre is greater than 1 or less than 1. Using these two methods as the loss function will produce
different training results.

Mean square error vs Mean absolute error

4.0

3.5

3.0

2.5
Loss

2.0

1.5

1.0

0.5

MAE
0.0 MSE

–2.0 –1.5 –1.0 –0.5 0.0 0.5 1.0 1.5 2.0

Y_true - Y_pred
FIG. 2.6 MSE and MAE loss value.

The house price-prediction model in this chapter uses MSE as the training loss function. The reader can change MSE
to MAE to compare the results between these two methods.
 2.1 Introduction to neural networks 33
4. Optimization
In training neural networks, the optimization process is to find a set of parameters, also called a set of weights, to
make the loss value of the loss function as small as possible. Gradient descent (GD) is the most commonly used opti-
mization algorithm for training neural networks. To find a minimum of the loss function with GD, take steps propor-
tional to the opposite direction of the gradient of the loss function at the current point. This algorithm can be likened to
a person who is looking for a path to get down a mountain where the path down is not visible. Based on the current
position of the person, they look at the steepness of the mountain, then go downhill in the direction of the steepest
descent. This step is repeated until they reach the bottom of the mountain. For example, given a neural network model
with uninitialized weights W and a loss function L which is used to calculate the error between the predicted output of
the network and the expected output. The network is trained using the GD algorithm to minimize L. The steps of find-
ing W to reach a minimum loss value of L are as follows:
▪ Step 1. Randomly initialize W
▪ Step 2. Update W with a learning rate η through the formula:
∂L
W ¼W η
∂W
∂L
where represents the gradient of L at point W.
∂W
▪ Step 3. Calculate the value of L with new W, called “loss,” and repeat Step 2 until reaching a minimum loss, as shown
in Fig. 2.7
The loss value reaches minimum loss, which means the predicted results of the model are closest to the expected
outputs.

FIG. 2.7 Gradient descent diagram.

In fact, not every update of GD is updated towards the minimum value of the loss function, but rather is updated
towards the direction that can reduce the error of the loss function at that time. Thus, when the model is trained until
the loss value cannot be reduced, the reached point is usually the local minimum instead of the global minimum, as
shown in Fig. 2.8.

FIG. 2.8 The weights are updated in the direction of converg-

ing to a local minimum instead of a global minimum.
34 2. Neural networks

Learning rate is a configurable hyperparameter that determines step size at each iteration in training neural
networks. A smaller learning rate results in small changes in the weights of each update and requires many train-
ing iterations, while a larger learning rate makes quick changes and requires fewer training iterations. If the learn-
ing rate is too large, the changes in the weights of each update are too large; it is likely to jump over the minimum
value and produce oscillations, as shown in Fig. 2.9A. Conversely, if the learning rate is too small, the optimization
efficiency may be poor, and the optimal value cannot be found after a long training, as shown in Fig. 2.9B. Thus,
the learning rate is one of the most crucial hyperparameters to be carefully selected when training the network
model.

FIG. 2.9 Optimization process with Loss Loss

different learning rates.

Start Point End Point Start Point End Point

Weights Weights

(A) Too large learning rate. (B) Too small learning rate.

There are many kinds of gradient-based optimization algorithms. The aforementioned GD uses all training data to
calculate the gradient of the loss function and update the weights once. If the neural network is updated N times, it
needs to calculate the entire training data N times. Using GD is very time-consuming and inefficient. Therefore, the
stochastic gradient descent (SGD) algorithm is introduced. At each time, SDG calculates the gradient of the loss func-
tions based only on a random sample from the training dataset, and then updates the weights based on this gradient.
This makes SDG suitable for training a huge training dataset. In addition, there are also many optimization methods,
such as Momentum [16], AdaGrad [17], Adam [18], and others. Momentum adapts the concept of momentum while
AdaGrad adjusts the learning rate according to the gradient for optimization. Adam can be seen as a combination of
Momentum and AdaGrad that utilizes estimations of first and second moments of gradient to adapt learning rates for
different weights. By using Adam as an optimizer, the weights of the network model can be updated through the fol-
lowing formula:
m^t
wt ¼ wt1  α pffiffiffiffi
^vt + ε
where w is the weights of the model, α is step size parameter (α ¼ 0.001), ε is set to 108, t ¼ 0 is for initialization of time
^ t + 1 and ^
step, and m vt + 1 are defined as:
mt
^t ¼
m
1  βt1
vt
^vt ¼
1  βt2
Here, β1, β2  [0, 1) are hyperparameters for controlling the exponential decay rates of the moving average of the gra-
dient mt and the squared gradient vt.
mt ¼ β1 mt1 + gt ð1  β1 Þ
vt ¼ β2 vt1 + ðgt ⨀ gt Þð1  β2 Þ

where gt represents gradient on mini-batch at timestep t, and ⨀ is the element-wise operation.

Because of the efficiency in optimization, Adam has been widely applied for training deep neural networks in recent
years.
 2.2 Introduction to Kaggle 35

2.2 Introduction to Kaggle

2.2.1 Kaggle platform

Kaggle is a world-famous competition platform where companies and researchers can publish datasets and orga-
nize competitions to solve the challenges of data science. The competition is open to anyone. As shown in Fig. 2.10, the
Kaggle contest page has as many as 19 contests in progress. It is also worth noting that Kaggle is not only a competition
website but also a community platform, allowing users to work, discuss, team up, or share research results with each
other.
Another great thing about Kaggle is that it has a datasets area, in which datasets have been sorted and made avail-
able for download, as shown in Fig. 2.11. Experimental data in this chapter is a dataset downloaded from the Kaggle
website.

FIG. 2.10 Kaggle competition page.

2.2.2 House sales in King County dataset

In this section, a “House Sales in King County, USA” dataset from Kaggle is introduced for training and evaluating
the house price-prediction model. To download the dataset, please access the URL:[Link]
harlfoxem/housesalesprediction, as shown in Fig. 2.12.
This dataset has 21,613 housing data, and each house sample has 21 items of information. The codes indicate the
following meanings:
▪ id: identification code of the house
▪ date: date the house was sold
▪ price: housing price (target)
▪ bedrooms: number of bedrooms
▪ bathrooms: number of bathrooms
▪ sqft_living: area of the interior living space (square feet)
▪ sqft_lot: area of the land space (square feet)
▪ floors: total floors of the house
▪ waterfront: a variable for whether or not the apartment overlooks the waterfront
36 2. Neural networks

FIG. 2.11 Public datasets on Kaggle.

FIG. 2.12 House sales in King County dataset.

 2.3 Experiment 1: House price prediction 37
▪ view: an index of how good the view of the property was
▪ condition: an index on the condition of the house
▪ grade: an index for rating building construction and design (according to the King County scoring system)
▪ sqft_above: area of the interior housing space that is above ground level (square feet)
▪ sqft_basement: area of the interior housing space that is below ground level (square feet)
▪ yr_built: building time
▪ yr_renovated: timing of last renovation
▪ zipcode: ZIP code that the house is in
▪ lat: latitude coordinates
▪ long: longitude coordinates
▪ sqft_living15: square footage of living space recorded in 2015 (implies some renovations)
▪ sqft_lot15: square footage of land lots recorded in 2015

2.3 Experiment 1: House price prediction

In this section, an FCNN is built and trained on the “House Sales in King County, USA” dataset to predict the price
of houses. The network model takes the information of the houses, such as the number of bedrooms, bathrooms, floors,
and so on, as the input, and then outputs the price of the house. MSE and Adam are used as the loss function and
optimizer of the model, respectively. Fig. 2.13 shows the flowchart of the source code for the house price-prediction
model.

FIG. 2.13 The flowchart of the source code for the

1. Preparing Data 2. Building and training the 3. Displaying house price-prediction model.
network model training results
- Importing packages
- Reading and - Build a fully connected - Training loss
converting data neural network model and valid loss
(model-1)
- Data division (training, - The average
validation, and test) - Set optimizer, loss function error percentage
- Training model-1 on the test data

2.3.1 Preparing dataset

1. Import necessary packages

import os
import numpy as np
import pandas as pd
import tensorflow as tf
import [Link] as plt
from tensorflow import keras
from [Link] import layers

2. Reading and converting data

▪ Read information
38 2. Neural networks

data = pd.read_csv(".\dataset\kc_house_data.csv")
# Display the shape of the dataset, a total of 21613 samples, each sample has 21

kinds of information.
[Link]
Result: (21613, 21)

▪ Display data
# Set the number of rows to 25
[Link].max_columns = 25
# display the first five lines (default)
[Link]()
Result:

▪ Check the data type

There are five types of data: object (string), boolean, inte ger, float, and categorical.

[Link]
Result:
id int64
date object
price float64
bedrooms int64
bathrooms float64
sqft_living int64
sqft_lot int64
floors float64
waterfront int64
view int64
condition int64
grade int64
sqft_above int64
sqft_basement int64
yr_built int64
yr_renovated int64
zipcode int64
lat float64
long float64
sqft_living15 int64
sqft_lot15 int64
dtype: object
 2.3 Experiment 1: House price prediction 39
▪ Convert data type
Because the date data in the dataset is in a string type and the input of the model only accepts a numeric type, date
data including year, month, and day are converted into numeric values through the following code:
# convert them to numeric values
data['year'] = pd.to_numeric(data['date'].[Link](0, 4))
data['month'] = pd.to_numeric(data['date'].[Link](4, 6))
data['day'] = pd.to_numeric(data['date'].[Link](6, 8))
#Delete useless data, inplace is to save the updated data to the original place
[Link](['id'], axis="columns", inplace=True)
[Link](['date'], axis="columns", inplace=True)
[Link]()
Result:

3. Data division

▪ Split data: Divide dataset into three sets: training data, validation data, and test data

data_num = [Link][0]
# Get a random index equal to the number of data,
indexes = [Link](data_num)
#Randomly divide data into Train, validation and test. The division ratio here is 6:2:2
train_indexes = indexes[:int(data_num *0.6)]
val_indexes = indexes[int(data_num *0.6):int(data_num *0.8)]
test_indexes = indexes[int(data_num *0.8):]
# Retrieve training data, validation data and test data
train_data = [Link][train_indexes]
val_data = [Link][val_indexes]
test_data = [Link][test_indexes]

▪ Data normalization
The main function of normalization is to scale different data to the same scale. For example: “The number of bed-
rooms or bathrooms in the house is about 1 to 5, and the area of the house is about 1500 m2 to 2500 m2.” Because of the
large difference in data scale, it may cause the prediction model to pay more attention to the data with larger values
and ignore the data with smaller values. In order to solve this problem, the input data is usually scaled between 0 and 1
or between 1 and 1; this process is called data normalization.
In this experiment, the standard score is used to standardize the data, which is formulated as follows:
ðx  meanÞ
xnorm ¼
std
where, x is a raw score, mean is the mean of the population, and std is the standard deviation of the population.
40 2. Neural networks

The source code for data normalization:

train_validation_data = [Link]([train_data, val_data])

mean = train_validation_data.mean()
std = train_validation_data.std()
train_data = (train_data - mean) / std
val_data = (val_data - mean) / std

Create the training data in Numpy array format

x_train = [Link](train_data.drop('price', axis='columns'))

y_train = [Link](train_data['price'])
x_val = [Link](val_data.drop('price', axis='columns'))
y_val = [Link](val_data['price'])
There are a total of 12967 training samples, and each sample has 21 kinds of
information.

x_train.shape
Result: (12967, 21)

2.3.2 Building and training network model

1. Build a FCNN named Model-1
In this example, we construct a network model with three fully connected layers, in which ReLU is used as the acti-
vation function in the hidden layers. Since a linear output is required, the output layer does not use any activation
function.
# Create a fully connected neural network
model = [Link](name='model-1')
# The first fully connected layer is set to 64 neurons, and the input shape is set to (21, ),
but in fact the shape of the data we input is (batch_size, 21)
[Link]([Link](64, activation='relu', input_shape=(21,)))
# The second fully connected layer (64 neurons)

[Link]([Link](64, activation='relu'))
# The output fully connected layer ( 1 neuron).
[Link]([Link](1))
# Display network model structure
[Link]()
Result:
 2.3 Experiment 1: House price prediction 41
2. Set the optimizer, loss function, metric function, and callback function.

Set the optimizer, loss function, metric function

[Link]([Link](0.001),
loss=[Link](),
metrics=[[Link]()])

Create a directory to save Model

model_dir = 'lab2-logs/models/'
[Link](model_dir) # for creating a folder to save model

Set the callback function:

# TensorBoard callback function helps record training information and save as

TensorBoard log file
log_dir = [Link]('lab2-logs', 'model-1')
model_cbk = [Link](log_dir=log_dir)
# ModelCheckpoint helps to save the network model,
model_mckp = [Link](model_dir + '/Best-model-1.h5',
monitor='val_mean_absolute_error',
save_best_only=True,
mode='min')

3. Training model

history = [Link](x_train, y_train, # training data

batch_size=64, # Batch size is set to 64
epochs=300, # Train the entire dataset 300 times
validation_data=(x_val, y_val), # Verification information
callbacks=[model_cbk, model_mckp])
Result:

2.3.3 Displaying training results

1. History

[Link]() # View what information is saved in history

Result dict_keys(['loss', 'val_loss', 'val_mean_absolute_error', 'mean_absolute_error'])
42 2. Neural networks

2. Draw a line chart of the loss

In “[Link],” the loss function is MSE, so the “loss” and val_loss recorded in the history are the loss values
calculated by the MSE.

[Link]([Link]['loss'] , label='train')
[Link]([Link]['val_loss'] , label='validation')
[Link]('loss')
[Link]('epochs')
[Link](loc='upper right')
Result:

3. Draw a line chart of metrics

In “[Link],” the metric function has been set to MAE, so the network calculates the MAE between the
predicted value and the expected output. The mean_absolute_error and val_mean_absolute_error values will be
recorded in history.

[Link]([Link]['mean_absolute_error'] , label='train')
[Link]([Link]['val_mean_absolute_error'] , label='validation')
[Link]('metrics')
[Link]('epochs')
[Link](loc='upper right')
Result:
 2.4 Introduction to TensorBoard 43
4. The average percentage error on test data
Predict house price on test data and calculate the average percentage error.
#Load model
model.load_weights('lab2-logs/models/Best-model-1.h5')
#take out the house price
y_test = [Link](test_data['price'])
# data normalization
test_data = (test_data - mean) / std
# Save the input data in Numpy format
x_test = [Link](test_data.drop('price', axis='columns'))
# Predict on test data
y_pred = [Link](x_test)
# Convert the prediction results back
y_pred = [Link](y_pred * std['price'] + mean['price'], y_test.shape)
# Calculate the mean percentage error
percentage_error = [Link]([Link](y_test - y_pred)) / [Link](y_test) * 100
# Display percentage error
print("Model_1 Percentage Error: {:.2f}%".format(percentage_error))
Result: Model_1 Percentage Error: 14.08%

2.4 Introduction to TensorBoard

TensorBoard is a TensorFlow toolkit that provides the visualization and measurement needed for machine learning
experimentation such as tracking loss values and the accuracy of the model during training, visualizing the model
graph, viewing histograms, and so on. There are two common ways to use TensorBoard: (1) adding “[Link]-
[Link]” function to create and store logs when training with [Link] of Keras, and (2) using “tf.
summary” API to log information when training with “[Link]()” or other methods. The graphic interface
of the TensorBoard is shown in Fig. 2.14. As shown, there are four main tools in TensorBoard: the Scalars dashboard,
Graphs dashboard, Distributions dashboard, and Histograms dashboard.

FIG. 2.14 The graphic interface of TensorBoard.

44 2. Neural networks

▪ The Scalars dashboard: helps to track scalar values such as learning rate, loss, accuracy, and so on during training
neural networks
▪ Graphs dashboard: helps to visualize the models built by TensorFlow
▪ The Distributions and Histograms dashboards: help to display the distribution of the tensor. They are widely used
for visualizing weights and biases of the TensorFlow models
The advantages and disadvantages of TensorBoard include:
▪ Advantages: The information during training the model such as changes in the loss, accuracy, the histograms of
weights, biases, and so on, can be tracked and viewed in real time, without having to wait until the training is
completed.
▪ Disadvantages: The information will be written to the log file many times during training the model. If a lot of
information is recorded, training time is increased.
When training the house price-prediction model, the TensorBoard callback function, namely, “[Link].
TensorBoard,” has been added for creating and storing the log. There are two ways to open the log file. The first
way is to directly open the log file on Jupyter Notebook, and the second way is to run TensorBoard through a terminal
and then observe results through the browser.
▪ Open log file with Jupyter Notebook (results are shown in Fig. 2.15)

FIG. 2.15 Visualizing metrics (loss and accuracy) on TensorBoard (1).

 2.4 Introduction to TensorBoard 45

# Loading TensorBoard directly on the jupyter notebook

%load_ext tensorboard
# Run TensorBoard and specify the log file folder as lab2-logs
%tensorboard --logdir lab2-logs
Result:
▪ Open log file with Command line
- Please go to the location where the TensorBoard log file is stored and run the command below. Note that the result
is observed through URL: [Link] as shown in Fig. 2.16.

tensorboard --logdir lab2-logs

- The port number can be specified for displaying the result; following the command below, the result can be
observed through URL: [Link] as shown in Fig. 2.16.

tensorboard --port 9527 --logdir lab2-logs

FIG. 2.16 Visualizing metrics (loss and accuracy) on TensorBoard (2).

In addition to metrics such as loss and accuracy, the model graph is also visualized, as shown in Fig. 2.17. We discuss
the other visualization functions of TensorBoard such as Images, Text, Audio, and so on in Chapter 7.
46 2. Neural networks

FIG. 2.17 Visualizing the model graph on TensorBoard.

2.5 Experiment 2: Overfitting problem

2.5.1 Introduction to overfitting

Overfitting refers to the network model that obtained very good performance on the training data but that had poor
performance on the validation data. The training loss curve is usually used to observe whether or not there is an over-
fitting problem. Fig. 2.18 presents an overfitting phenomenon, where the loss value of the training data (training error)
continues to decrease after a period of training, while the loss value of the verification data (validation error) gradually
increases.

Error

Underfitting zone Overfitting zone

Validation error
optimism
Training error

optimal Capacity
Capacity
FIG. 2.18 Overfitting phenomenon.

The training result of the house price-prediction model in Section 2.3 is shown in Fig. 2.19, and the overfitting
phenomenon can also be observed from the loss curve graph.
 2.5 Experiment 2: Overfitting problem 47

FIG. 2.19 Overfitting problem when training house price-prediction model.

Overfitting usually occurs when the training data is too small in scale or the complexity of the model is too great. As
such, adding training data or simplifying the model may improve the problem of overfitting. The three methods to
prevent overfitting without increasing the amount of data are:
▪ Reduce the size of the model: When the number of parameters of the model is reduced, the model with fewer
parameters will not be able to easily fit all training data. The model must learn how to use limited parameters to learn
an effective feature representation.
▪ Apply weight regularization: When training a neural network model, the size of the network weights will increase. The
longer the network is trained, the larger the network weights will become. The neural network with large weights is
usually unstable because even small variation on the inputs can lead to large changes in output [19]. This can be a sign of
overfitting training data of the neural network. To solve the overfitting problem, the core idea of weight regularization
is to limit the size of the network weights during the training process. To penalize large weights, the first weight size is
calculated, and then the calculated result is added to the loss function when training the model. There are two main
approaches to calculate weight size: L1 regularization and L2 regularization, also known as weight decay [20].
L1 regularization:
Weight size

Loss = Loss +

L2 regularization:
Weight size

Loss = Loss +

where λ is a regularization parameter for controlling the penalty, λ  [0, 1], w is the weights of the model, M is the total
amount of parameters of the model, and LossMSE is MSE loss function.
▪ Apply dropout technique [15]: The dropout technique refers to randomly discarding neurons in the neural network
to prevent complex co-adaptions during the training process. When discarding neurons, they are temporarily taken
out of the network, and their connections with other neurons are removed as well, as shown in Fig. 2.20. The dropout
48 2. Neural networks

technique has proved effective in addressing the problem of overfitting and improving the performance of neural
networks in many applications such as speech recognition, image classification, and others.

FIG. 2.20 Random dropout.

2.5.2 Code examples

This section continues using the house price-prediction model (Model-1) for testing overfitting. We explore the
effect of the three methods for preventing overfitting problems using three modified models based on model-1: (1)
model of reducing model size, named Model-2, (2) model of adding weight regularization, named Model-3, and
(3) model of adding dropout technique, named Model-4. Table 2.1 shows the architecture of Model-1, Model-2,
Model-3, and Model-4.

TABLE 2.1 The architecture of house price-prediction models.

Name Architecture Description

Model-1 - Input layer with input shape (,21). House price-prediction model in
- Two hidden layers (fully connected layers); each layer has 64 neurons. Section 2.3
- One output layer (fully connected layer) with one neuron.

Model-2 - Input layer with input shape (,21). Model of reducing the Model-1 size
- Two hidden layers (fully connected layers); each layer has 16 neurons.
- One output layer (fully connected layer) with one neuron.
Model-3 - Input layer with input shape (,21). Model of adding weights’
- Two hidden layers (fully connected layers); each layer has 64 neurons and L2 regularization. regularization
- One output layer (fully connected layer) with one neuron.
Model-4 - Input layer with input shape (,21). Model of adding dropout
- Two hidden layers (fully connected layers); each layer has 64 neurons; randomly discard 30% of technique
neurons in each layer.
- One output layer (fully connected layer) with one neuron.
 2.5 Experiment 2: Overfitting problem 49
1. Model-2: model of reducing the Model-1 size

# Create model-2
model_2 = [Link](name='model-2')
# first hidden fully connected layer with 16 neurons
model_2.add([Link](16, activation='relu', input_shape=(21,)))
# second hidden fully connected layer with 16 neurons
model_2.add([Link](16, activation='relu'))
# output fully connected layer with 1 neurons
model_2.add([Link](1))

# Set the optimizer, loss function and metrics function for training
model_2.compile([Link](0.001),
loss=[Link](),
metrics=[[Link]()])

# Set callback function

log_dir = [Link]('lab2-logs', 'model-2')
model_cbk = [Link](log_dir=log_dir)
model_mckp = [Link](model_dir + '/Best-model-2.h5',
monitor='val_mean_absolute_error',
save_best_only=True,
mode='min')
# Train model-2
model_2.fit(x_train, y_train,
batch_size=64,
epochs=300,
validation_data=(x_val, y_val),
callbacks=[model_cbk, model_mckp])
50 2. Neural networks

2. Model-3: model of adding weight regularization

# Create a network model

model_3 = [Link](name='model-3')
# first hidden fully connected layer with 64 neurons, adding L2 regularization
model_3.add([Link](64, kernel_regularizer=[Link].l2(0.001),
activation='relu', input_shape=(21,)))
# Second hidden fully connected layer with 64 neurons, adding L2 regularization
model_3.add([Link](64, kernel_regularizer=[Link].l2(0.001),
activation='relu'))
# hidden output fully connected layer with 1 neuron
model_3.add([Link](1))

# Set the optimizer, loss function and metric function for training
model_3.compile([Link](0.001),
loss=[Link](),
metrics=[[Link]()])

# Set callback function

log_dir = [Link]('lab2-logs', 'model-3')
model_cbk = [Link](log_dir=log_dir)
model_mckp = [Link](model_dir + '/Best-model-3.h5',
monitor='val_mean_absolute_error',
save_best_only=True,
mode='min')

# Train model-3
model_3.fit(x_train, y_train,
batch_size=64,
epochs=300,
validation_data=(x_val, y_val),
callbacks=[model_cbk, model_mckp])
 2.5 Experiment 2: Overfitting problem 51
3. Model-4: model of adding dropout

# Create model-4
model_4 = [Link](name='model-4')
# first hidden fully connected layer with 64 neurons,
model_4.add([Link](64, activation='relu', input_shape=(21,)))
#randomly discard 30% neurons
model_4.add([Link](0.3))
# second hidden fully connected layer with 64 neurons
model_4.add([Link](64, activation='relu'))
# randomly discard 30% neurons
model_4.add([Link](0.3))
#Output fully connected layer with 1 neurons
model_4.add([Link](1))

# Set the optimizer, loss function and indicator function for training
model_4.compile([Link](0.001),
loss=[Link](),
metrics=[[Link]()])

# Set callback function

log_dir = [Link]('lab2-logs', 'model-4')
model_cbk = [Link](log_dir=log_dir)
model_mckp = [Link](model_dir + '/Best-model-4.h5',
monitor='val_mean_absolute_error',
save_best_only=True,
mode='min')

# Train model-4
model_4.fit(x_train, y_train,
batch_size=64,
epochs=300,

validation_data=(x_val, y_val),
callbacks=[model_cbk, model_mckp])
52 2. Neural networks

After training, the trained Model-2, Model-3, and Model-4 are verified on the test data.
1. Model-2:

model_2.load_weights('lab2-logs/models/Best-model-2.h5')
y_pred = model_2.predict(x_test)
y_pred = [Link](y_pred * std['price'] + mean['price'], y_test.shape)
percentage_error = [Link]([Link](y_test - y_pred)) / [Link](y_test) * 100
print("Model_2 Percentage Error: {:.2f}%".format(percentage_error))
Result: Model_2 Percentage Error: 13.15%

2. Model-3:

model_3.load_weights('lab2-logs/models/Best-model-3.h5')
y_pred = model_3.predict(x_test)
y_pred = [Link](y_pred * std['price'] + mean['price'], y_test.shape)
percentage_error = [Link]([Link](y_test - y_pred)) / [Link](y_test) * 100
print("Model_3 Percentage Error: {:.2f}%".format(percentage_error))
Result: Model_3 Percentage Error: 12.89%

3. Model-4:

model_4.load_weights('lab2-logs/models/Best-model-4.h5')
y_pred = model_4.predict(x_test)
y_pred = [Link](y_pred * std['price'] + mean['price'], y_test.shape)
percentage_error = [Link]([Link](y_test - y_pred)) / [Link](y_test) * 100
print("Model_4 Percentage Error: {:.2f}%".format(percentage_error))
Result: Model_4 Percentage Error: 13.33%

2.5.3 Visualization with TensorBoard

In this section, we use TensorBoard to observe and analyze the training results of Model-1, Model-2, Model-3, and
Model-4 in Section 2.5.2.
▪ Uncheck all the training data records in the lower-left corner of TensorBoard, displaying only the validation data
records, as shown in Fig. 2.21. In Fig. 2.21, model-1/validation with the orange line chart represents an overfitting
model; model-2/validation with the maroon line chart represents the model of reducing parameters; model-3/
validation with the cyan line chart represents the model of adding L2 regularization; and model-4/validation with
the green line chart represents the model of using the dropout technique.
▪ Adjust the smoothing ratio to zero to display the most original data without modification; note that setting the
smoothing to zero is more convenient for finding the lowest point, as shown in Fig. 2.22.
The experimental results prove that three methods are capable of solving the overfitting problem of the original
model and improving the performance of the model. Among them, Model-4 applying the dropout technique has
the lowest loss value, while Model-3 using L2 normalization has the lowest percentage error.
 2.5 Experiment 2: Overfitting problem 53

FIG. 2.21 TensorBoard Scalars (1).

54 2. Neural networks

FIG. 2.22 TensorBoard Scalars (2).

 References 55

References
[1] W.S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys. 5 (4) (1943) 115–133.
[2] N. Rochester, J. Holland, L. Haibt, W. Duda, Tests on a cell assembly theory of the action of the brain, using a large digital computer, IRE Trans.
Inform. Theory 2 (3) (1956) 80–93.
[3] B.W.A.C. Farley, W. Clark, Simulation of self-organizing systems by digital computer, Trans. IRE Prof. Group Inform. Theory 4 (4) (1954) 76–84.
[4] F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain, Psychol. Rev. 65 (6) (1958) 386–408.
[5] J.J. Weng, N. Ahuja, T.S. Huang, Learning recognition and segmentation of 3-D objects from 2-D images, in: (4th) International Conference on
Computer Vision, IEEE, 1993, pp. 121–128.
[6] T. Joachims, Text categorization with support vector machines: learning with many relevant features, in: European Conference on Machine
Learning, Springer, Berlin, Heidelberg, 1998, pp. 137–142.
[7] C. Cortes, V. Vapnik, Support-vector networks, Mach. Learn. 20 (3) (1995) 273–297.
[8] P. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, PhD thesis Harvard University, Cambridge,
MA, 1974.
[9] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information
Processing Systems, 2012, pp. 1097–1105.
[10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, Imagenet: a large-scale hierarchical image database, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, 2009, pp. 248–255.
[11] X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: International Conference on Artificial Intelligence and Statistics, 2011,
pp. 315–323.
[12] K. He, X. Zhang, S. Ren, J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in: Proceedings of
the IEEE International Conference on Computer Vision, 2015, pp. 1026–1034.
[13] B. Recht, R. Roelofs, L. Schmidt, V. Shankar, Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451, (2018).
[14] J. Janke, M. Castelli, A. Popovic, Analysis of the proficiency of fully connected neural networks in the process of classifying digital images.
benchmark of different classification algorithms on high-level image features from convolutional layers. Expert Syst. Appl. 135 (2019)
12–38, [Online]. Available [Link]
[15] N. Srivastava, G.E. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhustdinov, Dropout: a simple way to prevent neural networks from overfitting,
J. Mach. Learn. Res. 15 (1) (2014) 1929–1958.
[16] I. Sutskever, J. Martens, G. Dahl, G. Hinton, On the importance of initialization and momentum in deep learning, in: International Conference on
Machine Learning, 2013, pp. 1139–1147.
[17] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization, J. Mach. Learn. Res. (2011)
2121–2159.
[18] D. Kinga, J.B. Adam, A method for stochastic optimization, in: International Conference on Learning Representations (ICLR), vol. 5, 2015.
[19] R. Reed, R.J. Marks II, Neural Smithing: Supervised Learning in Feedforward Artificial Neural Networks, MIT Press, 1999, 269.
[20] A. Krogh, J.A. Hertz, A simple weight decay can improve generalization, in: Advances in Neural Information Processing Systems, 1992,
pp. 950–957.