Deep Learning

Chapter 1 Introduction
Dr. Minhhuy Le
EEE, Phenikaa University
Chapter 1: Course info & programming review
1. Course introduction and grades
2. History of Deep Learning
3. Deep learning applications
4. Materials
 1. Course introduction and grades
Course introduction

Học phần “Học sâu” cung cấp những kiến thức cốt lõi của công nghệ học sâu, bao gồm:
môhì nh mạng nơ ron truyền thẳng; các kỹ thuật tổng quát hoávàtối ưu hóa các mô
hình; mở rộng môhình để làm việc với dữ liệu lớn; mạng CNN, RNN. Khóa học này
trang bị các kỹ năng liên quan đến việc thiết kế, xây dựng vàlập trì
nh môhình học sâu.
Học phần cũng trang bị kiến thức cần thiết để học viên cóthể sử dụng thư viện học sâu
như Tensorflow để xây dựng một số ứng dụng cơ bản của công nghệ học sâu.

Minhhuy Le, ICSLab, Phenikaa Uni. 4

 1. Course introduction and grades
Goals
• Tổng hợp lại kiến thức cơ bản về học sâu.
• Vận hành được các môhì nh học sâu vàứng dựng trong một số bài toán thực tế.

Outcome requirements
• Phân tí ch được các kiến thức về môhì nh mạng nơ ron truyền thẳng, các kỹ thuật
tổng quát hoávàtối ưu hóa môhì nh, môhình học sâu CNN, RNN.
• Thiết kế được các môhì nh học sâu CNN, RNN trong các bài toán thực tế với dữ
liệu ảnh vàdữ liệu chuỗi.

Minhhuy Le, ICSLab, Phenikaa Uni. 5

 1. Course introduction and grades
Book
• Ian, Goodfellow; Yoshua, Bengio; Aaron, Courville (2016), Deep Learning, The
MIT Press.

References
• Raschka, Sebastian (2019), Python Machine Learning :, Packt,, 9781789955750.
• Cs231n Stanford University
• Deep learning – [Link]

Grades
• Attendant + Homework: 10%
• Midterm project: 20%
• Final project: 70%

Minhhuy Le, ICSLab, Phenikaa Uni. 6

 1. Course introduction and grades

Code on: Python, Google Colab, Tensorflow 2.0, Sklearn

MIT - Massachusetts Institute of Technology Stanford University Harvard University

Minhhuy Le, ICSLab, Phenikaa Uni. 7

 1. Course introduction and grades
AI’s Heroes

Yan Lecun Yoshua Bengio Geoffrey Hinton

Andrew Ng. Fei Fei Li

Minhhuy Le, ICSLab, Phenikaa Uni. 8

 1. Course introduction and grades
Chapter 1: Course Infor & Programming review - Chapter 3: Convolutional Neural Network - week 8-10
week 1 1. Convolutional operator
1. Course introduction and grades 2. History of CNN
2. History of Deep learning 3. Deep Convolutional Models
3. Deep learning applications 4. Layers in CNN
5. Applications of CNN
Chapter 2: Building Neural Network from Scratch – 6. Practice
week 2-7 Midterm summary
1. Shallow neural network Chapter 4: TensorFlow Library - week 11-13
2. Deep neural network 1. Introduction to TensorFlow
3. Building neural network: step-by-step (modulation) 2. Building a deep neural network with TensorFlow
4. Regularization 3. Applications
5. Dropout 4. Practice
6. Batch Normalization Chapter 5: Recurrent Neural Network - week 14-15
7. Optimizers 1. Unfolding Computational Graphs
8. Hyper-parameters 2. Building a Recurrent Neural Networks
9. Practice 3. Long Short-Term Memory
Midterm 4. Vision with Language Processing
5. Application of RNN
6. Practice
Minhhuy Le, ICSLab, Phenikaa Uni. 9
 1. Course introduction and grades

45 hours at Classes:
Theory + Coding practice

90 hours shelf-study at home:

Theory + Coding practice

Minhhuy Le, ICSLab, Phenikaa Uni. 10

 2. History of Deep Learning

• Deep learning is a Subset

of Machine Learning in
which Artificial Neural
Network adapt and learn
from vast amounts of data

Minhhuy Le, ICSLab, Phenikaa Uni. 11

 2. History of Deep Learning
• Deep learning is a Subset of Machine Learning in which Artificial
Neural Network adapt and learn from vast amounts of data.

Training Data Deep Learning algorithm

Training
Prediction

New Data Learned model Prediction

Methods that can learn from and make predictions on data

Minhhuy Le, ICSLab, Phenikaa Uni. 12

 2. History of Deep Learning
Supervised: Learning with a labeled training set of data
Example: learn the classification of images based on image labels (dogs/cats, day time, numbers, etc.)

Unsupervised: Discover patterns in unlabeled data class

Example: cluster similar documents based on text A
class
Reinforcement learning: learn to act based on feedback/reward A
Example: learn to play Go, reward: win or lose Classification

Regression Clustering

Sources: [Link]
[Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 13

 2. History of Deep Learning
Supervised: Learning with a labeled training set of data
Example: learn the classification of images based on image labels (dogs/cats, day time, numbers, etc.)

Unsupervised: Discover patterns in unlabeled data class

Example: cluster similar documents based on text A
class
Reinforcement learning: learn to act based on feedback/reward A
Example: learn to play Go, reward: win or lose Classification

Regression Clustering

Sources: [Link]
[Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 14

 2. History of Deep Learning
Most deep learning methods work well because of human-designed
representations and input features
DL becomes just optimizing weights to best make a final prediction

NER: Named Entity Recognition

POS: Part of Speech

Minhhuy Le, ICSLab, Phenikaa Uni. 15

 2. History of Deep Learning

• Hierarchy of representations with increasing levels of abstraction

• Image recognition
• Pixel ➔ edge ➔ texton ➔ motif ➔ part ➔ object

• Text
• Character ➔ word ➔ word group ➔ clause ➔ sentence ➔ story

• Speech
• Sample ➔ spectral band ➔ sound ➔ … ➔ phone ➔ phoneme ➔
word

Minhhuy Le, ICSLab, Phenikaa Uni. 16

 2. History of Deep Learning
• A sub-field of machine learning for learning representations of data.
• Exceptionally effective at learning patterns.
• Deep learning algorithms attempt to learn (multiple levels of) representation by using a hierarchy of
multiple layers
• If you provide the system tons of information, it begins to understand it and respond in useful ways.

[Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 17

 2. History of Deep Learning Why is DL useful?

o Manually designed features are often over-specified, incomplete and take a

long time to design and validate
o Learned Features are easy to adapt, fast to learn
o Deep learning provides a very flexible, (almost?) universal, learnable
framework for representing world, visual and linguistic information.
o Can learn in both unsupervised and supervised ways
o Effective end-to-end joint system learning
o Utilize large amounts of training data

Around 2010, DL started to

outperform other ML techniques,
first in speech and vision, then in
Natural Language Processing (NLP)

Minhhuy Le, ICSLab, Phenikaa Uni. 18

 2. History of Deep Learning

Several big improvements in recent years in NLP Leverage different levels of representation
✓ Machine Translation o words & characters
✓ Sentiment Analysis o syntax & semantics
✓ Dialogue Agents
✓ Question Answering
✓ Text Classification …
Minhhuy Le, ICSLab, Phenikaa Uni. 19
 2. History of Deep Learning

•Minsky & Papert

showed XOR
cannot be solved by
1958~1969 •Perceptrons 1970~1980 Perceptrons (1969)
•Rosenblatt (1958),
(Golden Minsky & Papert
(1st AI •Backpropagation:
Era) Winter) Linnainmaa,
(1969) Werbos
•AI Dreams
shattered

• Backpropagation, • Deep Learning,

LSTM, OCR • Kernel Machines Imagenet,
• Rumelhart, Hinton, 1999~2005 (SVM, etc.) 2006~now Alexnet,
1981~1998 Williams, became popular GoogLeNet etc.
(2nd AI • NN could not
(Thaw of AI • Hinton, Osindero,
(AI Summer) Hochreiter,
Schmidhuber, Winter) improve beyond Winter) Teh, Deng, LeCun,
LeCun, Bottou, few layers Bottou, Bengio,
Bengio, Haffner Haffner

Minhhuy Le, ICSLab, Phenikaa Uni. 20

 2. History of Deep Learning
1958-1969
• Rosenblatt proposed a machine for
binary classifications
• Main idea
• One weight wi per input xi
• Multiply weights with respective
inputs and add bias w0
• If result is larger than threshold δ,
return 1, otherwise 0.

Minhhuy Le, ICSLab, Phenikaa Uni. 21

 2. History of Deep Learning

Training a Perceptron
• Learning Algorithm by Rosenblatt
• Initialize weights randomly
• Take one sample xi and predict yi
• For erroneous predictions, update weights
• If the output was 𝑦ො = 0 and yi = 1, increase
weights
• If the output was 𝑦ො = 1 and yi = 0, decrease
weights
• Repeat until no errors are made

Video (18’54”): [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 22

 2. History of Deep Learning

Multi-layer Perceptron
• One perceptron = one decision
• Question: What about multiple
decisions?
• Eg. Digit classification

• Answer: Neural Network (NN) or

Multi-Layer Perceptron (MLP)
• Stack multiple perceptrons
(neurons) into a single layer
• Connect two or more layers by
feeding output of one layer as
input to the next layer

Minhhuy Le, ICSLab, Phenikaa Uni. 23

 2. History of Deep Learning
1970~1980: 1st AI Winter

• XOR cannot be solved by Perceptron (Minsky)

• Perceptron training method cannot be applied to Neural Networks
• Funding slushed, Neural Networks were damned
• AI WINTER!!!
• Dreams shattered!
• Some significant results
• Backpropagation: training method for NN (1970, 1974)

Minhhuy Le, ICSLab, Phenikaa Uni. 24

 2. History of Deep Learning
1999~2005: 2nd AI Winter
• Kernel Machines (e.g. Support Vector Machines (SVM), etc.) became popular
• Achieved similar accuracies
• Included much fewer heuristics
• Nice proofs on generalization

• Neural networks could not improve beyond a few layers

• Lack of processing power (No GPUs)
• Lack of data (No big, annotated datasets)
• Overfitting (Models could not generalize)
• Vanishing gradients (0.1*0.1*0.1*….*0.1 = 0.000000000001, too small for
learning)
• AI community turned away from Neural Networks
Minhhuy Le, ICSLab, Phenikaa Uni. 25
 2. History of Deep Learning
2006~now: Thaw of AI Winter
• Questions:
• Are 1-2 hidden layers the best NN can do?
• Or, is it the learning algorithm not really mature?
Training
• Deep Learning (2006, Hinton, Osindero, Teh) Layer
• Layer-by-layer training
• Per-layer trained parameters
initialize further training using
contrastive divergence

Minhhuy Le, ICSLab, Phenikaa Uni. 26

 2. History of Deep Learning
Deep Learning is here …

• ImageNet dataset (Deng et al, 2009)

• Collected images for each term of Wordnet (100,000 classes)
• Tree of concepts organized hierarchically
• “Ambulance”, “Dalmatian dog”, “Egyptian cat”, …
• Imagenet Large Scale Visual Recognition Challenge (ILSVRC)
• 1 million images
• 1,000 classes
• Top-5 and top-1 error measured
• Errors reduced drastically in the past 8 years (2010~2017): 28.2% ➔ 2.3%

Minhhuy Le, ICSLab, Phenikaa Uni. 27

 2. History of Deep Learning
ImageNet Large Scale Visual
Recognition Challenge

152 layers

22 layers
Shallow
8 layers

AlexNet ResNet SENet

VGG,
GoogLeNet

Minhhuy Le, ICSLab, Phenikaa Uni. 28

 2. History of Deep Learning
Some FUN now …

• The Neural Network ZOO

• Graphical notations for all kinds of neural networks
• [Link]
• A Neural Network Playground
• An online interactive way to play with different network architectures
• [Link]

• 8 Inspirational Applications of Deep Learning

• Very interesting applications of deep learning
• [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 29

 3. Applications of Deep Learning
(1/8) Automatic Colorization of B&W Images

• Large Convolutional Neural

Networks (CNN)
• Website
[Link]
/
• Video (5 s)
[Link]
tion/

Minhhuy Le, ICSLab, Phenikaa Uni. 30

 3. Applications of Deep Learning
(2/8) Automatically Adding Sounds
• Two types of NN
• Large CNN for images
• Large Long Short-Term Memory
(LSTM) Recurrent Neural
Networks (RNN) for sound
• News
• [Link]
-intelligence-produces-realistic-
sounds-0613
• Visually Indicated Sounds (MIT)
• [Link]

• Video (2.54 s)
• [Link]
Minhhuy Le, ICSLab, Phenikaa Uni. 31
 3. Applications of Deep Learning
(3/8) Automatic Machine Translation

• Automatic Translation of Text

• Large Long Short-Term Memory (LSTM) Recurrent Neural Networks

• Automatic Translation of Images

• CNN + LSTM RNN

Minhhuy Le, ICSLab, Phenikaa Uni. 32

 3. Applications of Deep Learning
(4/8) Object Classification & Detection

• Large deep CNN

• Paper on ImageNet Classification
• [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 33

 3. Applications of Deep Learning
ConvNetJS: CIFAR-10 Demo

• ConvNetJS: CIFAR-10 Demo

• [Link]

• Clarifai: 10,000 images

• [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 34

 3. Applications of Deep Learning
(5/8) Automatic Handwriting Generation

• Usage: used with forensic analysis

DEMO: [Link]
Minhhuy Le, ICSLab, Phenikaa Uni. 35
 3. Applications of Deep Learning
(6/8) Automatic Text Generation

• Large RNN
• Code on Github
• [Link]
• Paul Graham generator
• Shakespeare
• Wikipedia
• Algebraic Geometry (LaTeX)
• Linux source code
• Generating Baby Names

Minhhuy Le, ICSLab, Phenikaa Uni. 36

 3. Applications of Deep Learning
Automatic Music Synthesis

• Using large RNN

• Authors comments:
• This track was made using a RNN. Fed 500 mb guitar tabs in ASCII. It writes
the tabs out in ASCII, I imported into GuitarPro, recorded the output, imported
that into FL Studio, added some filters and a drum loop and got this. The notes
and rhythms themselves are totally unedited.
• Music (5:20):
• [Link]
recurrent-neural-network

Minhhuy Le, ICSLab, Phenikaa Uni. 37

 3. Applications of Deep Learning
(7/8) Automatic Image Caption Generation

• Large CNN → Object Detection

• Large LSTM RNN → Caption Text Generation
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 38

 3. Applications of Deep Learning
(7/8) Automatic Image Caption Generation
• Demo
• [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 39

 3. Applications of Deep Learning
(8/8) Automatic Game Playing

• Vision
• Decision Making [Link]
• Etc.

Minhhuy Le, ICSLab, Phenikaa Uni. 40

 4. Materials

• Python Programming Language

• Deep Learning Frameworks
• Calculus

Minhhuy Le, ICSLab, Phenikaa Uni. 41

 4. Materials

• Python
• Jupyter Notebook
• Google Colab

Minhhuy Le, ICSLab, Phenikaa Uni. 42

 4. Materials

Deep learning Frameworks

Minhhuy Le, ICSLab, Phenikaa Uni. 43

 4. Materials
Deep learning Frameworks [Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 44

 4. Materials

Minhhuy Le, ICSLab, Phenikaa Uni. 45

 4. Materials
- Python programming lecture of mine
- CS229 Python & NumPy

Minhhuy Le, ICSLab, Phenikaa Uni. 46

 4. Materials
- Google Colab: Free GPU & CPU [Link]

[Link]

Minhhuy Le, ICSLab, Phenikaa Uni. 47

 5. Conclusions

• Remind: Python programming & Maths

• More self-study is required
• Coding from scratch
• Project based exam

Minhhuy Le, ICSLab, Phenikaa Uni. 48