6

CHAPTER 2
BACKGROUND THEORY

2.1. Machine Learning

According to Arthur Samuel, “Machine learning is a subfield of computer
science which gives computers the ability to learn without being explicitly
programmed”. This study helps in predicting and learning from the data imported
with the help of algorithms implemented. Machine learning is used where there is
difficulty in programming tasks instead machine learning algorithms are used to
achieve the task. Some of these tasks include identity fraud detection, computer
vision, population growth prediction, email filtering, Weather forecasting, Optical
Character Recognition (OCR), diagnostics, real-time decisions etc.

2.1.1. Deep Learning

According to Arthur Andrew Ng, “Deep Learning is a superpower, with it, can
make a computer see, translate languages, render a medical diagnosis, or build pieces
of a car that can drive itself. If that is not a superpower, do not know what is”. Deep
learning is a broader family of machine learning methods based on learning data
representations, as opposed to task-specific algorithms. Learning can be supervised or
unsupervised. It is a set of algorithms in machine learning to learn multiple levels of
representation, corresponding to different layers of abstraction that help to make sense
of data.
Many layers are used to compute nonlinear functions with highly complex
data. Each layer gets its input from a preceding layer, then it computes and transforms
the data and sends it to the further layers. Each layer in a network consists of neurons
and has various modes of connections to other neurons in the same layer as well as to
those of other layers depending on the type of network.
The whole idea of deep learning is using brain simulations, helping to make
learning algorithms more efficient to use and revolutionary advances in machine
learning and Artificial Intelligence. Nowadays deep learning gets more attention
 7

with development of modern technologies and easy execute it [4].

2.2. Artificial Neural Network (ANN)

Artificial Neural Network (ANN) is an efficient computing system whose
central theme is borrowed from the analogy of biological neural networks. Artificial
Neural Network (ANNs) are also named as “artificial neural systems,” or “parallel
distributed processing systems,” or “connectionist systems.” Artificial Neural
Network (ANN) acquires a large collection of units that are interconnected in some
pattern to allow communication between the units. These units, also referred to as
nodes or neurons, are simple processors which operate in parallel.
Every neuron is connected with other neuron through a connection link. Each
connection link is associated with a weight that has information about the input signal.
This is the most useful information for neurons to solve a particular problem because
the weight usually excites or inhibits the signal that is being communicated. Each
neuron has an internal state, which is called an activation signal. Output signals,
which are produced after combining the input signals and activation rule, may be sent
to other units [2]. Machine learning concepts are classified into three categories:
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning

2.2.1. Supervised Learning

There is, a dataset is given as input and assumptions can be made on the
output data how it looks like. In supervised learning, there is a relationship between
the input data and the output data. The output can be predicted with the input given.
The artificial neural network solutions have been trained with supervision. In
this mode, the actual output of a neural network is compared to the desired output.
Weights, which are usually randomly set to begin with, are then adjusted by the
network so that the next iteration, or cycle, it will produce a closer match between the
desired and the actual output. The learning method tries to minimize the current errors
of all processing elements. This global error reduction is created over time by
continuously modifying the input weights until acceptable network accuracy is
reached.
 8

In supervised learning, the artificial neural network must be trained before it

becomes useful. Training consists of presenting input and output data to the network.
This data is often referred to as the training set. That is, for each input set provided to
the system, the corresponding desired output set is provided as well. In most
applications, actual data must be used. This training phase can consume a lot of time.
In prototype systems, with inadequate processing power, learning can take weeks.
This training is considered complete when the neural network reaches a user
defined performance level. This level signifies that the network has achieved the
desired statistical accuracy as it produces the required outputs for a given sequence of
inputs. When no further learning is necessary, the weights are typically frozen for the
application. Some network types allow continual training, at a much slower rate,
while in operation. This helps a network to adapt to gradually changing conditions.
Training sets need to be fairly large to contain all the needed information if the
network is to learn the features and relationships that are important. Not only do the
sets have to be large but the training sessions must include a wide variety of data. If
the network is trained just one example at a time, all the weights set so meticulously
for one fact could be drastically altered in learning the next fact. The previous facts
could be forgotten in learning something new.
As a result, the system has to learn everything together, finding the best
weight settings for the total set of facts. For example, in teaching a system to
recognize pixel patterns for the ten digits, if there were twenty examples of each digit,
all the examples of the digit seven should not be presented at the same time.
How the input and output data is represented, or encoded, is a major
component to successfully instructing a network. Artificial networks only deal with
numeric input data. Therefore, the raw data must often be converted from the external
environment. Additionally, it is usually necessary to scale the data, or normalize it to
the network paradigm.
This pre-processing of real-world stimulation, be they cameras or sensors, into
machine readable format is already common for standard computers. Many
conditioning techniques which directly apply to artificial neural network
implementations are readily available. It is then up to the network designer to find the
best data format and matching network architecture for a given application.
After a supervised network performs well on the training data, then it is
 9

important to see what it can do with data it has not seen before. If a system does not
give reasonable outputs for this test set, the training period is not over. Indeed, this
testing is critical to ensure that the network has not simply memorized a given set of
data but has learned the general patterns involved within an application [3].

2.2.2. Unsupervised Learning

Unsupervised learning is an approach where the algorithm has to identify the
hidden patterns in the given input. So, the algorithm works without any guidance as
the input data is not labeled or classified.
Unsupervised learning is the great promise of the future. It shouts that
computers could someday learn on their own in a true robotic sense. Currently, this
learning method is limited to networks known as self-organizing maps. These kinds of
networks are not in widespread use. They are basically an academic novelty. Yet, they
have shown they can provide a solution in a few instances, proving that their promise
is not groundless. They have been proven to be more effective than many algorithmic
techniques for numerical aerodynamic flow calculations. They are also being used in
the lab where they are split into a front-end network that recognizes short, phoneme-
like fragments of speech which are then passed on to a back-end network. The second
artificial network recognizes these strings of fragments as words.
This promising field of unsupervised learning is sometimes called self-
supervised learning. These networks use no external influences to adjust their
weights. Instead, they internally monitor their performance. These networks look for
regularities or trends in the input signals, and makes adaptations according to the
function of the network. Even without being told whether it is right or wrong, the
network still must have some information about how to organize itself. This
information is built into the network topology and learning rules.
An unsupervised learning algorithm might emphasize cooperation among
clusters of processing elements. In such a scheme, the clusters would work together. If
some external input activated any node in the cluster, the cluster activity as a whole
could be increased. Likewise, if external input to nodes in the cluster was decreased,
that could have an inhibitory effect on the entire cluster.
Competition between processing elements could also form a basis for learning.
Training of competitive clusters could amplify the responses of specific groups to
 10

specific stimuli. As such, it would associate those groups with each other and with a
specific appropriate response. Normally, when competition for learning is in effect,
only the weights belonging to the winning processing element will be updated.
At the present state of the art, unsupervised learning is not well understood
and is still the subject of research. This research is currently of interest to the
government because military situations often do not have a data set available to train a
network until a conflict arises [3].

2.2.3. Reinforcement Learning

Reinforcement learning is a suitable action to maximize reward in a particular
situation. It is to find the best possible behavior or path it should take in a specific
situation.

2.3. Preprocessing of Images

Users are required to take photos of hand-written numbers on light-colored
paper or board. However, with the real-world lighting, the shadows and specular
highlights make it difficult to segment and recognize the digits directly. The intensity
values of the digits are close to the shadow, so applying a global threshold to image
cannot segment the digits from the background effectively. Address the issue by first
running a preprocessing procedure on the taken image, found the preprocessing step
useful since it can eliminate the noise comes with the paper and light and only
amplify the digits.
In the preprocessing, Canny edge features computed on the image are fed into
contour finder to draw the bounding box of each feature. The result of the bounding
box is shown in image. To improve the speed of preprocessing step, the input image is
resized to 640_480 at the beginning, and the color is also inverted, converting the
light-colored background to dark.

2.4. Activation Function

The Activation function is important for an Artificial Neural Network (ANN)
to learn and make sense of something really complicated. Their main purpose is to
convert an input signal of a node in an Artificial Neural Network (ANN) to an output
signal. This output signal is used as the input value to the next layer in the stack.
 11

Activation function decides whether a neuron should be activated or not by

calculating the weighted sum and further adding bias to it. The motive is to introduce
non-linearity into the output of a neuron.
If do not apply activation function then the output signal would be simply
linear function (one-degree polynomial). Now, a linear function is easy to solve but
that are limited in their complexity, have less power. Without activation function,
model cannot learn and model complicated data such as images, videos, audio,
speech, etc.
Non-Linear functions are those which have a degree more than one and it have
a curvature. Now, need a neural network to learn and represent almost anything and
any arbitrary complex function that maps an input to output.
Neural Network is considered “Universal Function Approximators”. It means
they can learn and compute any function at all [5].

2.4.1. Types of Activation Functions

There are four types of activation functions.
[Link]. Threshold activation function (binary step function)
A Binary step function is a threshold-based activation function. If the input
value is above or below a certain threshold, the neuron is activated and sends exactly
the same signal to the next layer.

Figure 2.1. Binary step-function

The problem with this function is for creating a binary classifier (1 or 0), but if
you want multiple such neurons to be connected to bring in more classes, Class1,
Class2, Class3, etc. In this case, all neurons will give one, so cannot decide.
 12

[Link]. Sigmoid activation function (logistic function)

A Sigmoid function is a mathematical function having a characteristic “S”-
shaped curve or sigmoid curve which ranges between 0 and 1, therefore it is used for
models where need to predict the probability as an output. The Sigmoid function is
differentiable, means which can find the slope of the curve at any two points.

Figure 2.2. Sigmoid activation function

Sigmoid activation function is that it can cause the drawback of the neural
network to get stuck at training time if strong negative input is provided.

[Link]. Hyperbolic tangent function (tanh)

It is similar to Sigmoid but better in performance. It is nonlinear in nature, so
great which can stack layers. The function ranges between (-1, 1).

Figure 2.3. Hyperbolic tangent function

 13

The main advantage of this function is that strong negative inputs will be
mapped to negative output and only zero-valued inputs are mapped to near-zero
outputs. So less likely to get stuck during training.

[Link]. Rectified linear units (ReLU)

Rectified Linear Units (ReLU), is the most used activation function in
Convolutional Neural Networks (CNNs) and Artificial Neural Network (ANN) which
ranges from zero to infinity [0, ∞).

Figure 2.4. Rectified linear unit function

It gives an output ‘x’ if x is positive and 0 otherwise. It looks like having the
same problem of linear function as it is linear in the positive axis. Relu is non-linear in
nature and a combination of ReLu is also non-linear. In fact, it is a good approximator
and any function can be approximated with a combination of Relu. ReLu is 6 times
improved over hyperbolic tangent function. It should only be applied to hidden layers
of a neural network. So, for the output layer use softmax function for classification
problem and for regression problem use a Linear function.

2.5. Convolutional Neural Networks

Regular neural networks use vectors as input and have fully connected layers,
which mean that each element is connected to all nodes in the next layer. Images,
then, need to be reshaped to vectors and the number of weights, between the input
layer and one node in the hidden layer, are the same as the number of pixels. Larger
images would, therefore, result in an unreasonable number of weights. Convolutional
Neural Networks (CNNs) use images as input and convolution kernels, made up out
 14

of weights, as connections to the next layer. This means that weights are shared on
different spatial positions and the needed amounts of weights are lowered. These
weights are arranged in 3D volumes and transform the input image into an output of
node activations [5].

2.5.1. Layers
Convolutional Neural Networks (CNNs) consist of three different types of
layers: convolutional, max pooling and fully connected layers. Convolutional layers
have weights arranged in kernels that are convolved with the input. Each pixel in the
2D convolution output is mapped through an activation function, such as Rectified
Linear Units (ReLU), to produce a 2D output volume of node activations. The use of
convolution can both keep and decrease the size of the input image. Usually,
convolutional layers keep the size which is accomplished by the use of zero padding.
It adds zeros around the image before the convolution and then keeps only the central
part of the result with the same size as the input.
Max pooling layers are used to down sample the 2D volume. A max pooling
layer takes an image region and preserves only the highest value inside the region. It
can be seen as a convolution kernel, with a specific stride, extracting max values
instead of calculating dot products. A stride of 1 would preserve the volume size
while a stride of two would divide the width and height dimension by two [4].
The fully connected layers are exactly the same as the ones in a regular neural
network. Each node in the previous layer has a connection to each node in the fully
connected layer. Such layers are usually placed at the end of a Convolutional neural
networks (CNN) [5].

2.5.2. Convolutional Neural Networks (CNNs) Architecture

Convolutional Neural Networks (CNNs) usually contain repetitions of
convolutional layer(s) followed by max pooling. The final layers are one or two fully
connected layer(s) followed by a Softmax layer. It is designed for gray images with
28 pixels in both width and height. Zero padding is used in all the convolutional and
max pooling layers making the stride determine the output size [5].