MEAP Edition

Manning Early Access Program

Deep Learning with Python
Version 5

Copyright 2017 Manning Publications

For more information on this and other Manning titles go to

[Link]

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 Welcome
Thank you for purchasing the MEAP for Deep Learning with Python. If you are looking
for a resource to learn about deep learning from scratch and to quickly become able to use
this knowledge to solve real-world problems, you have found the right book. *Deep
Learning with Python* is meant for engineers and students with a reasonable amount of
Python experience, but no significant knowledge of machine learning and deep learning. It
will take you all the way from basic theory to advanced practical applications. However, if
you already have experience with deep learning, you should still be able to find value in
the latter chapters of this book.
Deep learning is an immensely rich subfield of machine learning, with powerful
applications ranging from machine perception to natural language processing, all the way up
to creative AI. Yet, its core concepts are in fact very simple. Deep learning is often
presented as shrouded in a certain mystique, with references to algorithms that “work like
the brain”, that “think” or “understand”. Reality is however quite far from this science-
fiction dream, and I will do my best in these pages to dispel these illusions. I believe that
there are no difficult ideas in deep learning, and that’s why I started this book, based on
premise that all of the important concepts and applications in this field could be taught to
anyone, with very few prerequisites.
This book is structured around a series of practical code examples, demonstrating on real-
world problems every the notions that gets introduced. I strongly believe in the value of
teaching using concrete examples, anchoring theoretical ideas into actual results and
tangible code patterns. These examples all rely on Keras, the Python deep learning library.
When I released the initial version of Keras almost two years ago, little did I know that it
would quickly skyrocket to become one of the most widely used deep learning frameworks.
A big part of that success is that Keras has always put ease of use and accessibility front and
center. This same reason is what makes Keras a great library to get started with deep
learning, and thus a great fit for this book. By the time you reach the end of this book, you
will have become a Keras expert.
I hope that you will this book valuable —deep learning will definitely open up new
intellectual perspectives for you, and in fact it even has the potential to transform your
career, being the most in-demand scientific specialization these days. I am looking forward
to your reviews and comments. Your feedback is essential in order to write the best possible
book, that will benefit the greatest number of people.

— François Chollet

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 brief contents
PART 1: THE FUNDAMENTALS OF DEEP LEARNING
1 What is Deep Learning?
2 Before we start: the mathematical building blocks of neural networks
3 Getting started with neural networks
4 Fundamentals of machine learning
PART 2: DEEP LEARNING IN PRACTICE
5 Deep learning for computer vision
6 Deep learning for text and sequences
7 Advanced deep learning best practices
8 Generative deep learning
9 Conclusions

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 1

What is Deep Learning?

1.1 Artificial intelligence, machine learning and deep learning

In the past few years, Artificial Intelligence (AI) has been a subject of intense media
1
hype. Machine learning, deep learning, and AI come up in countless articles, often
outside of technology-minded publications. We are being promised a future of intelligent
chatbots, self-driving cars, and virtual assistants—a future sometimes painted in a grim
light, and sometimes as an utopia, where human jobs would be scarce and most economic
activity would be handled by robots or AI agents.
As a future or current practitioner of machine learning, it is important to be able to
recognize the signal in the noise, to tell apart world-changing developments from what
are merely over-hyped press releases. What is at stake is our future, and it is a future in
which you have an active role to play: after reading this book, you will be part of those
who develop the AIs. So let’s tackle these questions—what has deep learning really
achieved so far? How significant is it? Where are we headed next? Should you believe
the hype?
First of all, we need to define clearly what we are talking about when we talk about
AI. What is artificial intelligence, machine learning, and deep learning? How do they
relate to each other?

Figure 1.1 Artificial Intelligence, Machine Learning and Deep Learning

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 2

1.1.1 Artificial intelligence

Artificial intelligence was born in the 1950s, as a handful of pioneers from the nascent
field of computer science started asking if computers could be made to "think"—a
question whose ramifications we are still exploring today. A concise definition of the
field would be: the effort to automate intellectual tasks normally performed by humans.
As such, AI is a very general field which encompasses machine learning and deep
learning, but also includes many more approaches that do not involve any learning. Early
chess programs, for instance, only involved hard-coded rules crafted by programmers,
and did not qualify as "machine learning". In fact, for a fairly long time many experts
believed that human-level artificial intelligence could be achieved simply by having
programmers handcraft a sufficiently large set of explicit rules for manipulating
knowledge. This approach is known as "symbolic AI", and it was the dominant paradigm
in AI from the 1950s to the late 1980s. It reached its peak popularity during the "expert
systems" boom of the 1980s.
Although symbolic AI proved suitable to solve well-defined, logical problems, such
as playing chess, it turned out to be intractable to figure out explicit rules for solving
more complex, fuzzy problems, such as image classification, speech recognition, or
language translation. A new approach to AI arose to take its place: machine learning.

1.1.2 Machine Learning

In Victorian England, Lady Ada Lovelace was a friend and collaborator of Charles
Babbage, the inventor of the "Analytical Engine", the first known design of a
general-purpose computer—a mechanical computer. Although visionary and far ahead of
its time, the Analytical Engine wasn’t actually meant as a general-purpose computer
when it was designed in the 1830s and 1840s, since the concept of general-purpose
computation was yet to be invented. It was merely meant as a way to use mechanical
operations to automate certain computations from the field of mathematical
analysis—hence the name "analytical engine". In 1843, Ada Lovelace remarked on the
invention:
"The Analytical Engine has no pretensions whatever to originate anything. It can do
whatever we know how to order it to perform… Its province is to assist us in making
available what we are already acquainted with."
This remark was later quoted by AI pioneer Alan Turing as "Lady Lovelace’s
objection" in his landmark 1950 paper "Computing Machinery and Intelligence", which
introduced the "Turing test" as well as key concepts that would come to shape AI. Turing
was quoting Ada Lovelace while pondering whether general-purpose computers could be
capable of learning and originality, and he came to the conclusion that they could.
Machine learning arises from this very question: could a computer go beyond "what
we know how to order it to perform", and actually "learn" on its own how to perform a
specified task? Could a computer surprise us? Rather than crafting data-processing rules
by hand, could it be possible to automatically learn these rules by looking at data?
©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 3

This question opens up the door to a new programming paradigm. In classical

programming, the paradigm of symbolic AI, humans would input rules (a program), data
to be processed according to these rules, and out would come answers. With machine
learning, humans would input data as well as the answers expected from the data, and out
would come the rules. These rules could then be applied to new data to produce original
answers.

Figure 1.2 Machine learning: a new programming paradigm

A machine learning system is "trained" rather than explicitly programmed. It is

presented with many "examples" relevant to a task, and it finds statistical structure in
these examples which eventually allows the system to come up with rules for automating
the task. For instance, if you wish to automate the task of tagging your vacation pictures,
you could present a machine learning system with many examples of pictures already
tagged by humans, and the system would learn statistical rules for associating specific
pictures to specific tags.
Although machine learning only started to flourish in the 1990s, it has quickly
become the most popular and most successful subfield of AI, a trend driven by the
availability of faster hardware and larger datasets. Machine learning is tightly related to
mathematical statistics, but it differs from statistics in several important ways. Unlike
statistics, machine learning tends to deal with large, complex datasets (e.g. a dataset of
millions of images, each consisting of tens of thousands of pixels) for which "classical"
statistical analysis such as bayesian analysis would simply be too impractical to be
possible. As a result, machine learning, and especially deep learning, exhibits
comparatively little mathematical theory—maybe too little—and is very
engineering-oriented. It is a hands-on discipline where ideas get proven empirically much
more often than theoretically.

1.1.3 Learning representations from data

To define deep learning, and understand the difference between deep learning and other
machine learning approaches, first we need to get some idea of what machine learning
algorithms really do. We just stated that machine learning discovers rules to execute a
data-processing task, given examples of what is expected. So, to do machine learning, we
need three things:

Input data points. For instance, if the task is speech recognition, these data points could
be sound files of people speaking. If the task is image tagging, they could be picture files.
Examples of the expected output. In a speech recognition task, these could be
©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
4

human-generated transcripts of our sound files. In an image task, expected outputs could
tags such as "dog", "cat", and so on.
A way to measure if the algorithm is doing a good job, to measure the distance between
its current output and its expected output. This is used as a feedback signal to adjust the
way the algorithm works. This adjustment step is what we call "learning".

A machine learning model transforms its input data into a meaningful output, a
process which is "learned" from exposure to known examples of inputs and outputs.
Therefore, the central problem in machine learning and deep learning is to meaningfully
transform data, or in other words, to learn useful "representations" of the input data at
hand, representations that get us closer to the expected output. Before we go any further:
what’s a representation? At its core, it’s a different way to look at your data—to
"represent", or "encode" your data. For instance, a color image can be encoded in the
RGB format ("red-green-blue") or in the HSV format ("hue-saturation-value"): these are
two different representations of the same data. Some tasks that may be difficult with one
representation can become easy with another. For example, the task "select all red pixels
in the image" is simpler in the RBG format, while "make the image less saturated" is
simpler in the HSV format. Machine learning models are all about finding appropriate
representations for their input data, transformations of the data that make it more
amenable to the task at hand, such as a classification task.
Let’s make this concrete. Let’s consider an x axis, and y axis, and some points
represented by their coordinates in the (x, y) system: our data, as illustrated in figure 3
1.3.

Figure 1.3 Some sample data

As you can see we have a few white points and a few black points. Let’s say we want
to develop an algorithm that could take the coordinates (x, y) of a point, and output
whether the point considered is likely to be black or to be white. In this case:

The inputs are the coordinates of our points.

The expected outputs are the colors of our points.
A way to measure if our algorithm is doing a good job could be, for instance, the
percentage of points that are being correctly classified.

What we need here is a new representation of our data that cleanly separates the

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 5

white points from the black points. One transformation we could use, among many other
possibilities, would be a coordinate change, illustrated in figure 1.4.

Figure 1.4 Coordinate change

In this new coordinate system, the coordinates of our points can be said to be a new
"representation" of our data. And it’s a good one! With this representation, the
black/white classification problem can be expressed as a simple rule: black points are
such that x 0 or "white points are such that x < 0". Our new representation basically
solves the classification problem.
In this case, we defined our coordinate change by hand. But if instead we tried
systematically searching for different possible coordinate changes, and used as feedback
the percentage of points being correctly classified, then we would be doing machine
learning. "Learning", in the context of machine learning, describes an automatic search
process for better representations.
All machine learning algorithms consist of automatically finding such
transformations that turn data into more useful representations for a given task. These
operations could sometimes be coordinate changes, as we just saw, or could be linear
projections (which may destroy information), translations, non-linear operations (such as
select all points such that x 0), etc. Machine learning algorithms are not usually very
creative in finding these transformations, they are merely searching through a predefined
set of operations, called an "hypothesis space".
So that’s what machine learning is, technically: searching for useful representations
of some input data, within a pre-defined space of possibilities, using guidance from some
feedback signal. This simple idea allows for solving a remarkably broad range of
intellectual tasks, from speech recognition to autonomous car driving.
Now that you understand what we mean by learning, let’s take a look at what makes
deep learning special.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
6

1.1.4 The "deep" in deep learning

Deep learning is a specific subfield of machine learning, a new take on learning
representations from data which puts an emphasis on learning successive "layers" of
increasingly meaningful representations. The "deep" in "deep learning" is not a reference
to any kind of "deeper" understanding achieved by the approach, rather, it simply stands
for this idea of successive layers of representations—how many layers contribute to a
model of the data is called the "depth" of the model. Other appropriate names for the field
could have been "layered representations learning" or "hierarchical representations
learning". Modern deep learning often involves tens or even hundreds of successive
layers of representation—and they are all learned automatically from exposure to training
data. Meanwhile, other approaches to machine learning tend to focus on learning only
one or two layers of representation of the data. Hence they are sometimes called "shallow
learning".
In deep learning, these layered representations are (almost always) learned via models
called "neural networks", structured in literal layers stacked one after the other. The term
"neural network" is a reference to neurobiology, but although some of the central
concepts in deep learning were developed in part by drawing inspiration from our
understanding of the brain, deep learning models are not models of the brain. There is no
evidence that the brain implements anything like the learning mechanisms in use in
modern deep learning models. One might sometimes come across pop-science articles
proclaiming that deep learning works "like the brain", or was "modeled after the brain",
but that is simply not the case. In fact, it would be confusing and counter-productive for
new-comers to the field to think of deep learning as being in any way related to the
neurobiology. You don’t need that shroud of "just like our minds" mystique and mystery.
So you might as well forget anything you may have read so far about hypothetical links
between deep learning and biology. For our purposes, deep learning is merely a
mathematical framework for learning representations from data.

Figure 1.5 A deep neural network for digit classification

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 7

What do the representations learned by a deep learning algorithm look like? Let’s
look at how a 3-layer deep network transforms an image of a digit in order to recognize
what digit it is:

Figure 1.6 Deep representations learned by a digit classification model

As you can see, the network transforms the digit image into representations that are
increasingly different from the original image, and increasingly informative about the
final result. You can think of a deep network as a multi-stage information distillation
operation, where information goes through successive filters and comes out increasingly
"purified" (i.e. useful with regard to some task).
So that is what deep learning is, technically: a multi-stage way to learn data
representations. A simple idea—but as it turns out, very simple mechanisms, sufficiently
scaled, can end up looking like magic.

1.1.5 Understanding how deep learning works in three figures

At this point, you know that machine learning is about mapping inputs (e.g. images) to
targets (e.g. the label "cat"), which is done by observing many examples of input and
targets. You also know that deep neural networks do this input-to-target mapping via a
deep sequence of simple data transformations (called "layers"), and that these data
transformations are learned by exposure to examples. Now let’s take a look at how this
learning happens, concretely.
The specification of what a layer does to its input data is stored in the layer’s
"weights", which in essence are a bunch of numbers. In technical terms, you would say
that the transformation implemented by a layer is "parametrized" by its weights. In fact,
weights are also sometimes called the "parameters" of a layer. In this context, "learning"
will mean finding a set of values for the weights of all layers in a network, such that the
network will correctly map your example inputs to their associated targets. But here’s the
thing: a deep neural network can contain tens of millions of parameters. Finding the
©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
8

correct value for all of them may seem like a daunting task, especially since modifying
the value of one parameter will affect the behavior of all others!

Figure 1.7 A neural network is parametrized by its weights

To control something, first, you need to be able to observe it. To control the output of
a neural network, you need to be able to measure how far this output is from what you
expected. This is the job of the "loss function" of the network, also called "objective
function". The loss function takes the predictions of the network and the true target (what
you wanted the network to output), and computes a distance score, capturing how well
the network has done on this specific example.

Figure 1.8 A loss function measures the quality of the network’s output

The fundamental trick in deep learning is to use this score as a feedback signal to
adjust the value of the weights by a little bit, in a direction that would lower the loss
score for the current example. This adjustment is the job of the "optimizer", which
implements what is called the "backpropagation" algorithm, the central algorithm in deep
learning. In the next chapter we will explain in more detail how backpropagation works.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 9

Figure 1.9 The loss score is used as a feedback signal to adjust the weights

Initially, the weights of the network are assigned random values, so the network
merely implements a series of random transformations --naturally its output is very far
from what it should ideally be, and the loss score is accordingly very high. But with
every example that the network processes, the weights get adjusted just a little in the right
direction, and the loss score decreases. This is the "training loop", which, repeated a
sufficient number of times (typically tens of iterations overs thousands of examples),
yields weight values that minimize the loss function. A network with a minimal loss is
one for which the outputs are as close as they can be to the targets: a trained network.
Once again: a very simple mechanism, which once scaled ends up looking like magic.

1.1.6 What deep learning has achieved so far

Although deep learning is a fairly old subfield of machine learning, it only rose to
prominence in the early 2010s. In the few years since, it has achieved nothing short of a
revolution in the field, with remarkable results on all perceptual problems, such as
"seeing" and "hearing"—problems which involve skills that seem very natural and
intuitive to humans but have long been elusive for machines.
In particular, deep learning has achieved the following breakthroughs, all in
historically difficult areas of machine learning:

Near-human level image classification.

Near-human level speech recognition.
Near-human level handwriting transcription.
Improved machine translation.
Improved text-to-speech conversion.
Digital assistants such as Google Now or Amazon Alexa.
Near-human level autonomous driving.
Improved ad targeting, as used by Google, Baidu, and Bing.
Improved search results on the web.
Answering natural language questions.
Superhuman Go playing.

In fact, we are still just exploring the full extent of what deep learning can do. We
©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
10

have started applying it to an even wider variety of problems outside of machine

perception and natural language understanding, such as formal reasoning. If successful,
this might herald an age where deep learning assists humans in doing science, developing
software, and more.

1.1.7 Don’t believe the short-term hype

Although deep learning has led to remarkable achievements in recent years, expectations
for what the field will be able to achieve in the next decade tend to run much higher than
what will actually turn out to be possible. While some world-changing applications like
autonomous cars are already within reach, many more are likely to remain elusive for a
long time, such as believable dialogue systems, human-level machine translation across
arbitrary languages, and human-level natural language understanding. In particular, talk
of "human-level general intelligence" should not be taken too seriously. The risk with
high expectations for the short term is that, as technology fails to deliver, research
investment will dry up, slowing down progress for a long time.
This has happened before. Twice in the past, AI went through a cycle of intense
optimism followed by disappointment and skepticism, and a dearth of funding as a result.
It started with symbolic AI in the 1960s. In these early days, projections about AI were
flying high. One of the best known pioneers and proponents of the symbolic AI approach
was Marvin Minsky, who claimed in 1967: "Within a generation […] the problem of
creating 'artificial intelligence' will substantially be solved". Three years later, in 1970, he
also made a more precisely quantified prediction: "in from three to eight years we will
have a machine with the general intelligence of an average human being". In 2016, such
an achievement still appears to be far in the future, so far in fact that we have no way to
predict how long it will take, but in the 1960s and early 1970s, several experts believed it
to be right around the corner (and so do many people today). A few years later, as these
high expectations failed to materialize, researchers and government funds turned away
from the field, marking the start of the first "AI winter" (a reference to a nuclear winter,
as this was shortly after the height of the Cold War).
It wouldn’t be the last one. In the 1980s, a new take on symbolic AI, "expert
systems", started gathering steam among large companies. A few initial success stories
triggered a wave of investment, with corporations around the world starting their own
in-house AI departments to develop expert systems. Around 1985, companies were
spending over a billion dollar a year on the technology, but by the early 1990s, these
systems had proven expensive to maintain, difficult to scale, and limited in scope, and
interest died down. Thus began the second AI winter.
It might be that we are currently witnessing the third cycle of AI hype and
disappointment—and we are still in the phase of intense optimism. The best attitude to
adopt is to moderate our expectations for the short term, and make sure that people less
familiar with the technical side of the field still have a clear idea of what deep learning
can and cannot deliver.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 11

1.1.8 The promise of AI

Although we might have unrealistic short-term expectations for AI, the long-term picture
is looking bright. We are only just getting started in applying deep learning to many
important problems in which it could prove transformative, from medical diagnoses to
digital assistants. While AI research has been moving forward amazingly fast in the past
five years, in large part due to a wave of funding never seen before in the short history of
A.I, so far relatively little of this progress has made its way into the products and
processes that make up our world. Most of the research findings of deep learning are not
yet applied, or at least not applied to the full range of problems that they can solve across
all industries. Your doctor doesn’t yet use AI, your accountant doesn’t yet use AI.
Yourself, you probably don’t use AI technologies in your day-to-day life. Of course, you
can ask simple questions to your smartphone and get reasonable answers. You can get
fairly useful product recommendations on [Link]. You can search for "birthday"
on Google Photos and instantly find those pictures of your daughter’s birthday party from
last month. That’s a far cry from where such technologies used to stand. But such tools
are still just accessory to our daily lives. AI has yet to transition to become central to the
way we work, think and live.
Right now it may seem hard to believe that AI could have a large impact on our
world, because at this point AI is not yet widely deployed—much like it would have been
difficult to believe in the future impact of the Internet back in 1995. Back then most
people did not see how the Internet was relevant to them, how it was going to change
their lives. The same is true for deep learning and AI today. But make no mistake: AI is
coming. In a not so distant future, AI will be your assistant, even your friend; it will
answer your questions, it will help educate your kids, and it will watch over your health.
It will deliver your groceries to your door and it will drive you from point A to point B. It
will be your interface to an increasingly complex and increasingly information-intensive
world. And even more importantly, AI will help humanity as a whole move forwards, by
assisting human scientists in new breakthrough discoveries across all scientific fields,
from genomics to mathematics.
On the road to get there, we might face a few setbacks, and maybe a new AI
winter—in much the same way that the Internet industry got overhyped in 1998-1999 and
suffered from a crash that dried up investment throughout the early 2000s. But we will
get there eventually. AI will end up being applied to nearly every process that makes up
our society and our daily lives, much like the Internet today.
Don’t believe the short-term hype, but do believe in the long-term vision. It may take
a while for AI to get deployed to its true potential—a potential the full extent of which no
one has yet dared to dream—but AI is coming, and it will transform our world in a
fantastic way.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
12

1.2 Why deep learning, why now?

The two key ideas of deep learning for computer vision, namely convolutional neural
networks and backpropagation, were already well-understood in 1989. The LSTM
algorithm, fundamental to deep learning for time series, was developed in 1997 and has
barely changed since. So why did deep learning only take off after 2012? What changed
in these two decades?
In general, there are three technical forces that are driving advances in machine
learning:

Hardware.
Datasets and benchmarks.
Algorithmic advances.

Because the field is guided by experimental findings rather than by theory,

algorithmic advances only become possible when appropriate data and hardware is
available to try new ideas (or just scale up old ideas, as is often the case). Machine
learning is not mathematics or physics, where major advances can be done with a pen and
a piece of paper. It is an engineering science.
So the real bottleneck throughout the 1990s and 2000s was data and hardware. But
here is what happened during that time: the Internet took, and high-performance graphics
chips were developed for the needs of the gaming market.

1.2.1 Hardware
Between 1990 and 2010, off-the shelf CPUs have gotten faster by a factor of
approximately 5,000. As a result, nowadays it’s possible to run small deep learning
models on your laptop, whereas this would have been intractable 25 years ago.
However, typical deep learning models used in computer vision or speech recognition
require orders of magnitude more computational power than what your laptop can
deliver. Throughout the 2000s, companies like NVIDIA and AMD have been investing
billions of dollars into developing fast, massively parallel chips (graphical processing
units, GPUs) for powering the graphics of increasingly photorealistic video games.
Cheap, single-purpose supercomputers designed to render complex 3D scenes on your
screen, in real-time. This investment came to benefit the scientific community when, in
2007, NVIDIA launched CUDA, a programming interface for its line of GPUs. A small
number of GPUs started replacing massive clusters of CPUs in a number of various
highly-parallelizable applications, starting with physics modeling. Deep neural networks,
consisting mostly of many small matrix multiplications, are also highly parallelizable,
and around 2011, some researchers started writing CUDA implementations of neural
nets—Dan Ciresan and Alex Krizhevsky were some of the first among them.
So what happened is that the gaming market has subsidized supercomputing for the
next generation of artificial intelligence applications. Sometimes, big things start as
games. Today, the NVIDIA Titan X, a gaming GPU that cost $1000 at the end of 2015,

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 13

can deliver a peak of 6.6 TLOPS in single-precision, i.e. 6.6 trillion of float32
operations per second. That’s about 350 times more than what you can get out of a
modern laptop. On a Titan X, it only takes a couple of days to train an ImageNet model
of the sort that would have won the competition a few years ago. Meanwhile, large
companies train deep learning models on clusters of hundreds of GPUs of a type
developed specifically for the needs of deep learning, such as the NVIDIA K80. The
sheer computational power of such clusters is something that would never have been
possible without modern GPUs.
What’s more, the deep learning industry is even starting to go beyond GPUs, and is
investing into increasingly specialized and efficient chips for deep learning. In 2016, at
its annual I/O convention, Google revealed its "TPU" project (tensor processing unit), a
new chip design developed from the ground-up to run deep neural networks, reportedly
10x faster and far more energy-efficient than top-of-line GPUs.

1.2.2 Data
Artificial Intelligence is sometimes heralded as the new industrial revolution. If deep
learning is the steam engine of this revolution, then data is its coal. The raw material that
powers our intelligent machines, without which nothing would be possible. When it
comes to data, besides the exponential progress in storage hardware over the past twenty
years, following Moore’s law, the game-changer has been the rise of the Internet, making
it feasible to collect and distribute very large datasets for machine learning. Today, large
companies work with image datasets, video datasets, and natural language datasets that
could not have been collected without the Internet. User-generated image tags on Flickr,
for instance, have been a treasure trove of data for computer vision. So were YouTube
videos. And Wikipedia is a key dataset for natural language processing.
If there is one dataset that has been a catalyst for the rise of deep learning, it is the
ImageNet dataset, consisting in 1.4 million images hand-annotated with 1000 images
categories (one category per image). But what makes ImageNet special is not just its
large size, but also the yearly competition associated with it. As [Link] as been
demonstrating since 2010, public competitions are an excellent way to motivate
researchers and engineers to push the envelope. Having common benchmarks that
researchers compete to beat has greatly helped the recent rise of deep learning.

1.2.3 Algorithms
Besides hardware and data, up until the late 2000s, we were still missing a reliable way to
train very deep neural networks. As a result, neural networks were still fairly shallow,
leveraging only one or two layers of representations, and so they were not able to shine
against more refined shallow methods such as SVMs or Random Forests. The key issue
was that of "gradient propagation" through deep stacks of layers. The feedback signal
used to train neural networks would fade away as the number of layers increased.
This changed around 2009-2010 with the development of several simple but
important algorithmic improvements that allowed for better gradient propagation:

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
14

Better "activation functions" for neural layers.

Better "weight initialization schemes". It started with layer-wise pre-training, which was
quickly abandoned.
Better "optimization schemes", such as RMSprop and Adam.

It is only when these improvements started allowing for training models with ten or
more layers that deep learning really started to shine.
Finally, in 2014, 2015 and 2016, even more advanced ways to help gradient
propagation were discovered, such as batch normalization, residual connections, and
depthwise separable convolutions. Today we can train from scratch models that are
thousands of layers deep.

1.2.4 A new wave of investment

As deep learning became the new state of the art for computer vision in 2012-2013, and
eventually for all perceptual tasks, industry leaders took note. What followed was a
gradual wave of industry investment far beyond anything previously seen in the history
of AI.
In 2011, right before deep learning started taking the spotlight, the total venture
capital investment in AI was around $19M, going almost entirely to practical applications
of shallow machine learning approaches. By 2014, it had risen to a staggering $394M.
Dozens of startups launched in these 3 years, trying to capitalize on the deep learning
hype. Meanwhile, large tech companies such as Google, Facebook, Baidu and Microsoft
have invested in internal research departments in amounts that would most likely dwarf
the flow of venture capital money. Only a few numbers have surfaced. In 2013, Google
acquired the deep learning startup DeepMind for a reported $500M—the largest
acquisition of an AI company in history. In 2014, Baidu started a deep learning research
center in Silicon Valley, investing $300M in the project. The deep learning hardware
startup Nervana Systems was acquired by Intel in 2016 for over $400.
In fact, machine learning and in particular deep learning have become central to the
product strategy of these tech giants. In late 2015, Sundar Pichai, Google CEO, stated:
"Machine learning is a core, transformative way by which we’re rethinking how
we’re doing everything. We are thoughtfully applying it across all our products, be it
search, ads, YouTube, or Play. And we’re in early days, but you will see us?-- in a
systematic way—apply machine learning in all these areas."
As a result of this wave of investment, the number of people working on deep
learning went in just 5 years from a few hundreds, to tens of thousands, and research
progress has reached a frenetic pace. There are currently no signs that this trend is going
to slow anytime soon.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
 15

1.2.5 The democratization of deep learning

One the key factors driving this inflow of new faces in deep learning has been the
democratization of the toolsets used in the field. In the early days, doing deep learning
required significant C++ and CUDA expertise, which few people possessed. Nowadays,
basic Python scripting skills suffice to do advanced deep learning research. This has been
driven most notably by the development of Theano and then TensorFlow, two symbolic
tensor manipulation frameworks for Python that support auto-differentiation, greatly
simplifying the implementation of new models, and by the rise of user-friendly libraries
such as Keras, which makes deep learning as easy as manipulating Lego bricks. After its
release early 2015, Keras has quickly become the go-to deep learning solution for large
numbers of new startups, grad students, and for many researchers pivoting into the field.

1.2.6 Will it last?

Is there anything special about deep neural networks that makes them the "right"
approach for companies to be investing in and for researchers to flock to? Or is deep
learning just a fashion that might not last? Will we still be using deep neural networks in
20 years?
The short answer is yes—deep learning does have several properties that justify its
status as an AI revolution, and it is here to stay. We may not still be using neural
networks two decades from now, but whatever we use will directly inherit from modern
deep learning and its core concepts.
These important properties can be broadly sorted into 3 categories:

Simplicity. Deep learning removes the need for feature engineering, replacing complex,
brittle and engineering-heavy pipelines with simple end-to-end trainable models typically
built using only 5 or 6 different tensor operations.
Scalability. Deep learning is highly amenable to parallelization on GPUs or TPUs,
making it capable of taking full advantage of Moore’s law. Besides, deep learning models
are trained by iterating over small batches of data, allowing them to be trained on datasets
of arbitrary size (the only bottleneck being the amount of parallel computational power
available, which thanks to Moore’s law is a fast-moving barrier).
Versatility and reusability. Contrarily to many prior machine learning approaches, deep
learning models can be trained on additional data without restarting from scratch, making
them viable for continuous online learning, an important property for very large
production models. Furthermore, trained deep learning models are repurposable and thus
reusable: for instance it is possible to take a deep learning model trained for image
classification and drop it into a video processing pipeline. This allows us to reinvest
previous work into increasingly complex and powerful models. This also makes deep
learning applicable to fairly small datasets.

Deep learning has only been in the spotlight for a few years, and we haven’t yet
established the full scope of what it can do. Every passing month we still come up with
new use cases, or with engineering improvements lifting previously known limitations.
Following a scientific revolution, progress generally follows a sigmoid curve: it starts
with a period of fast progress than gradually stabilizes, as researchers start hitting against
hard limitations and further improvements become more incremental. With deep learning
©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]
16

in 2017, it seems that we are still in the first half of that sigmoid, and there is a lot more
progress yet to come in the next few years.

©Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and
other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.
[Link]