Module 1

Introduction to Deep Learning

Introduction
• Supervised learning adapts a system such that for a given input data it
produces a target output. The learning data is made up of tuples
(attributes, label) where “attributes” represent the input data and
“label” represents the target output.
• The goal here is to adapt the system so that for a new input the
system can predict the target output. Supervised learning can use
both continuous and discrete types of input data
Unsupervised learning
• Unsupervised learning involves data that comprises of input vectors
without any target output. There are different objectives in
unsupervised learning, such as clustering, density estimation, and
visualization.
• The goal of clustering is to discover groups of similar data items on
the basis of measured or perceived similarities between the data
items. The purpose of density estimation is to determine the
distribution of the data within the input space.
• In visualization, the data is projected down from a high-dimensional
space to two or three dimensions to view the similar data items.
Semi-supervised learning
• Semi-supervised learning first uses unlabeled data to learn a feature
representation of the input data and then uses the learned feature
representation to solve the supervised task.
• The training dataset can be divided into two parts: the data samples with
corresponding labels and the data samples where the labels are not known.
• Semisupervised learning can involve not providing with an explicit form of error
at each time but only a generalized reinforcement is received giving indication of
how the system should change its behavior, and this is sometimes referred to as
reinforcement.
• Reinforcement learning has been successful in applications as diverse as
autonomous helicopter flight, robot legged locomotion, cell-phone network
routing, marketing strategy selection, factory control and efficient webpage
indexing.
1.2 Shallow Learning
• Shallow architectures are well understood and perform good on many
common machine learning problems, and they are still used in a vast
majority of today’s machine learning applications.
• However, there has been an increased interest in deep architectures
recently, in the hope to find means to solve more complex real-world
problems (e.g., image analysis or natural language understanding) for
which shallow architectures are unable to learn models adequately
Deep Learning
• Deep learning is a new area of machine learning which has gained
popularity in recent past.
• Deep learning refers to the architectures which contain multiple
hidden layers (deep networks) to learn different features with
multiple levels of abstraction.
• Deep learning algorithms seek to exploit the unknown structure in
the input distribution in order to discover good representations, often
at multiple levels, with higher level learned features defined in terms
of lower level features.
• Conventional machine learning techniques are restricted in the way
they process the natural data in its raw form.
• For decades, constructing a pattern recognition or machine learning
system required considerable domain expertise and careful hand
engineering to come up with a feature extractor that transformed the
raw data (such as pixel values of an image) into suitable internal
representation or feature vector from which the learning system, such
as a classifier, could detect or classify patterns in the input.
• Deep learning allows inputting the raw data (pixels in case of image
data) to the learning algorithm without first extracting features or
defining a feature vector.
• Deep learning algorithms can learn the right set of features, and it
does this in a much better way than extracting these features using
hand-coding.
• Instead of handcrafting a set of rules and algorithms to extract
features from raw data, deep learning involves learning these features
automatically during the training process
• In deep learning, a problem is realized in terms of hierarchy of concepts, with each
concept built on the top of the others.
• The lower layers of the model encode some basic representation of the problem,
whereas higher level layers build upon these lower layers to form more complex
concepts
• Given an image, the pixel intensity values are fed as inputs to the deep learning system.
A number of hidden layers then extract features from the input image. These hidden
layers are built upon each other in a hierarchal fashion. At first, the lower level layers of
the network detect only edge-like regions.
• These edge regions are then used 1.3 Deep Learning 3 to define corners (where edges
intersect) and contours (outlines of objects). The layers in the higher level combine
corners and contours to lead to more abstract “object parts” in the next layer. The key
aspect of deep learning is that these layers of features are not handcrafted and designed
by human engineers; rather, they are learnt from data gradually using a general-purpose
learning procedure.
• Finally, the output layer classifies the image and obtains the output class label—
the output obtained at the output layer is directly influenced by every other node
available in the network.
• This process can be viewed as hierarchical learning as each layer in the network
uses the output of previous layers as “building blocks” to construct increasingly
more complex concepts at the higher layers.
• Figure compares traditional machine learning approach based on handcrafted
features to deep learning approach based on hierarchical representation learning.
Why to Use Deep Learning
• In particular, deep learning has made good impact in historically difficult areas of
machine learning:
• Near-human-level image classification;
• Near-human-level speech recognition;
• Near-human-level handwriting transcription;
• Improved self-driving cars;
• Digital assistants such as Google Now, Microsoft Cortana, Apple’s Siri, and
Amazon Alexa;
• Improved ad targeting, as used by Google, Baidu, and Bing;
• Improved search results on the web;
• Ability to answer natural language questions; and
• Superhuman Go, Shogi, and Chess playing
• The exceptional performance of deep models can be mainly attributed to their
flexibility in representing a rich set of highly nonlinear functions as well as the
devised methods for efficient training of these powerful networks. Furthermore,
employing various regularization techniques ensured that deep models with huge
numbers of free parameters are statistically desirable in the sense that they will
generalize well to unseen data.
• The automatic and generic approach of feature learning in deep models enables
one to use them across different applications (e.g., image classification, speech
recognition, language modeling, and information retrieval) with relatively little
adjustments.
• Therefore, deep models seem to be domain-oblivious in the sense that in order
to use it across different applications, only a small amount of domain specific
customizations is required. Ideally, the domain-obliviousness of deep networks is
advantageous, as having access to a universal and generic model reduces the
hassles of adapting for new applications.
• Deep learning is still in its infancy, but it is likely that deep learning will have many
successes in the near future as it requires little hand engineering and thus can
take advantage of vast amount of data and computation power.
• Deep learning has succeeded in previously unsolved problems which were quite
difficult to resolve using machine learning as well as other shallow networks.
• The dramatic progress of deep learning has sparked such a burst of activity that
venture capitalists who did not even know what deep learning was all about
some years back, today are suspicious of the startups that do not have it.
• In near future, deep learning may herald an age where it may assist humans in
software development, science, and many more. Integrating deep learning with
the whole toolbox of other artificial intelligence techniques may accomplish
startling things that will have great impact in the field of technology.
How Deep Learning Works
• Deep networks map input to target via a sequence of layered transformations, and that
these layered transformations are learned by exposure to the training examples. The
transformations that a layer applies to its input are determined by the layer’s weights,
which are basically a bunch of numbers.
• In other words, transformations implemented by a layer are parameterized by its
weights. In this context, learning can be defined as the process of finding the values of
the weights of all layers in the network in such a manner that input examples can be
correctly mapped to their associated targets.
• A deep learning network contains thousands of parameters, and finding the right values
of these parameters is not an easy task, particularly when the value of one parameter
has an impact on the value of another parameter.
• In order to train a deep network one needs to find out how far the calculated output of
the network is from the desired value. This measure is obtained by using a loss function,
also called as objective function. The objective function calculates the difference
between the predicted output obtained from the network and the true target value for a
specific example
• This gives a measure of how well the network has learnt a specific example. The
objective of the training is to find the values for the weights that minimize the chosen
error function.
• The difference obtained is then used as a feedback signal to adjust the weights of the
network, in a way that loss score for the current example is lowered. This adjustment is
done by the optimizer—backpropagation algorithm, the central algorithm in deep
learning. Backpropagation algorithm involves assigning random values to the weight
vectors initially, so that the network just implements a series of random transformations.
• Initially, the output obtained from the network can be far from what it should be, and
accordingly the loss score may be very high. With every example that is fed to the
network, the weights are adjusted in such a direction that makes the loss score to
decrease.
• This process is repeated a number of times, until the weight values that minimize the
loss function are obtained. A network is said to have learned when the output values
obtained from the network are as close as they can be to the target values.
Deep Learning Challenges
• Deep learning networks have brought their own set of problems and challenges which
outweighed the benefits of deep architectures for several decades. Training these
architectures for general use was impractically slow. With limited computational power,
deep learning networks were already overtaken by other approaches such as kernel
methods.
• With the significant growth in computational power (particularly in GPUs and distributed
computing) and access to large labeled datasets paved the way for its return. However,
despite the remarkable advances in this area, training deep models with a huge number
of free parameters is an intricate and ill-posed optimization problem.
• Many research works have been dedicated to creating efficient training methods for
deep architectures. The strategies reported in the literature that deal with the difficulties
of training deep networks include developing better optimizers, using well-designed
initialization strategies, using activation functions based on local competition and using
skip connections between layers with the aim to improve the flow of information.
• However, deep network training still faces problems which are caused by
the stacking of several nonlinear transformations and need to be
addressed. Moreover, deep learning involves using large amounts of data
to learn progressively. While large amounts of data are available in many
applications, however, in some areas copious amount of data are rarely
available. More flexible models are required to achieve an enhanced
learning ability when only a limited amount of data is available
• Deep learning networks are very good at solving one problem; however,
using deep networks to solve a very similar problem requires retraining and
reassessment. Although there are many advancements in this aspect, more
work is required in developing deep learning models which can perform
multitasks without the need of reworking on the whole architecture