Master Course

Machine Learning Techniques

Prof. Fahd Kalloubi
[Link]@[Link]

❖ Associate professor in Machine Learning and Big Data

❖ Ex Adjunct professor at euro-Mediterranean university of Fez

❖ Affiliate professor at UM6P – The Vanguard Center

❖ Ex Professor at ENSIAS

❖ Research interests: NLP, Knowledge Graph, Recommender System, Web Mining

 Lecture Topics
1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for ML/DL
Fahd Kalloubi Machine Learning Techniques 2
 What you will be able to do
after this course: A short
teaser

Fahd Kalloubi Machine Learning Techniques 3

 Audio classification using convolutional Neural
Networks

Fahd Kalloubi Machine Learning Techniques 4

 2D/3D Convolutional Networks

Fahd Kalloubi Machine Learning Techniques 5

 Photographic Style Transfer with Deep Learning

Fahd Kalloubi Machine Learning Techniques 6

 About this course

Fahd Kalloubi Machine Learning Techniques 7

 Topics Planned 1/3
• Part 1: Introduction
• Course overview, introduction to deep learning
• The brief history of Machine/deep learning
• Practical considerations to deliver ML outcomes
• Computational requirements for ML
• Part 2: mathematical and computational foundations
• Linear algebra and Vector calculus for Machine Learning
• Probability and distribution, continuous optimization
• Convex optimization
• Backpropagation and Automatic differentiation
• Empirical Risk Minimization and Basics of model learning
• Part 3: clustering and density estimation
• Clustering methods: K-means, DBSCAN, …
• Parametric Vs Non parametric estimation methods (MLE and MAP)
• Gaussian mixture models
Fahd Kalloubi • Expecatation-MaximizationMachine Learning Techniques 8
 Topics Planned 2/3
• Part 4: Support Vector Machines and Kernel methods
• Support Vector Machines and Nonlinear SVMs
• Kernel methods
• Machine learning models evaluation – metrics and methods
• Part 5: Tree-based methods
• Decision trees
• Ensemble methods
• Part 6: dimensionality reduction - Feature selection and feature extraction
• Principal component analysis, Linear Discriminant Analysis, Multiple Discriminant
Analysis, Isometric feature Mapping, …
• Feature selection methods
• Part 7: linear models
• Linear models for classification
• Linear models for regression
Fahd Kalloubi Machine Learning Techniques 9
 Topics Planned 3/3

• Part 8: Introduction to neural networks

• Multilayer Perceptron and universal approximation theorem
• Regularization, Normalization, weight initialization
• Advanced optimization algorithms for neural networks
• Residual neural networks and variants
• Part 9: Neural Networks Architectures
• Convolutional neural networks (if time allows it)
• Recurrent neural networks (if time allows it)
• Autoencoders and variational autoencoders (if time allows it)
• Generative Adversial Networks (if time allows it)

Fahd Kalloubi Machine Learning Techniques 10

 Questions ?

Fahd Kalloubi Machine Learning Techniques 11

 Grading

• 25% problem Sets

• Assignments
• Quizzes
• 25% Class Project:
• 5% Project proposal
• 10% Project presentation (+ peer review)
• 10% Project report + Github repository (+ peer review)
• 50% final exam

Fahd Kalloubi Machine Learning Techniques 12

 What is machine learning ?
A short overview

1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI
Fahd Kalloubi Machine Learning Techniques 13
 The Traditional Programming Paradigm

Machine learning is a Field of study that gives computers the ability to

learn without being explicitly programmed – Arthur Samuel (1959)

Machine Learning

Fahd Kalloubi Machine Learning Techniques 14

 The connection between fields

Fahd Kalloubi Machine Learning Techniques 15

 Different types of AI
• Artificial Intelligence (AI)
• orig. subfield of computer science, solving tasks humans are good at (natural language,
speech, image recognition, ...)
• Narrow AI
• solving a particular task (playing a game, driving a car, ...)
• Artificial General Intelligence (AGI)
• multi-purpose AI mimicking human intelligence across tasks

Fahd Kalloubi Machine Learning Techniques 16

 What This Course Is About ?

E.g., generalized linear models,

tree-based methods, "shallow"
E.g., symbolic expressions, logic networks, support vector machines,
rules / "handcrafted" nested if-else nearest neighbors, ...
programming statements ...
Main focus of the course
Fahd Kalloubi Machine Learning Techniques 17
 Not all AI systems involve Machine Learning
Deep Blue used custom VLSI chips
to execute the alpha-beta search
algorithm in parallel, an example of
GOFAI (Good Old-Fashioned
Artificial Intelligence).

Fahd Kalloubi Machine Learning Techniques 18

 Examples from the three Related “Areas”

Algorithms that learn

models/representations/ rules
A non-biological system that is automatically from data/examples
Algorithms that parameterize
intelligent through rules
multilayer neural networks that then
learn representations of data with
multiple layers of abstraction
Fahd Kalloubi Machine Learning Techniques 19
 Some applications of Machine Learning/Deep Learning

• Email spam detection

• Fingerprint / face detection & matching (e.g., phones)
• Web search (e.g., DuckDuckGo, Bing, Google)
• Sports predictions
• Post office (e.g., sorting letters by zip codes)
• ATMs (e.g., reading checks)
• Credit card fraud
• Stock predictions
Source: [Link]

Fahd Kalloubi Machine Learning Techniques 20

 Some applications of Machine Learning/Deep Learning
• Email spam detection
• Smart assistants (Apple Siri, Amazon Alexa, ...)
• Product recommendations (e.g., Netflix, Amazon)
• Self-driving cars (e.g., Uber, Tesla)
• Language translation (Google translate)
• Sentiment analysis
• Drug design
• Medical diagnoses
• ...
Source: [Link]
roads-with-no-one-at-the-wheel/

Fahd Kalloubi Machine Learning Techniques 21

 The 3 Broad Categories of ML
This also applies to DL

1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI
Fahd Kalloubi Machine Learning Techniques 22
 The 3 Broad Categories of ML

Source: Simon J.D. Prince (2023). Understanding Deep Learning , MIT Press

Source: Raschka and Mirjalily (2019). Python Machine Learning, 3rd Edition
Fahd Kalloubi Machine Learning Techniques 23
 Supervised Learning
Regression

Source: Raschka and Mirjalili (2019). Python Machine Learning, 3rd Edition

Source: Simon J.D. Prince (2023). Understanding Deep Learning , MIT Press

Fahd Kalloubi Machine Learning Techniques 24

 Supervised Learning
Classification

Source: Raschka and Mirjalili (2019). Python Machine Learning, 3rd Edition

Source: Simon J.D. Prince (2023). Understanding Deep Learning , MIT Press
Fahd Kalloubi Machine Learning Techniques 25
 Supervised Learning

Fahd Kalloubi Machine Learning Techniques 26

 Supervised Learning

Fahd Kalloubi Machine Learning Techniques 27

 Supervised Learning

Fahd Kalloubi Machine Learning Techniques 28

 Supervised Learning
What do these examples have in common?
• Very complex relationship between input and output
• Sometimes may be many possible valid answers
• But outputs (and sometimes inputs) obey rules

Language obeys Natural images also

grammatical rules have “rules”
Fahd Kalloubi Machine Learning Techniques 29
 Supervised Learning
Idea
• Learn the “grammar” of the data from unlabeled examples
• Can use a gargantuan amount of data to do this (as unlabeled)
• Make the supervised learning task earlier by having a lot of
knowledge of possible outputs

Fahd Kalloubi Machine Learning Techniques 30

 The 2 nd Subcategory of ML/DL

Fahd Kalloubi Machine Learning Techniques 31

 unsupervised learning 1:
Representation learning/dimensionality reduction
E.g., Principal Component Analysis (PCA)

Fahd Kalloubi Machine Learning Techniques 32

 unsupervised learning 1:
Representation learning/dimensionality reduction
E.g., Autoencoders

Fahd Kalloubi Machine Learning Techniques 33

 unsupervised learning 2:
Clustering

Clustering of a
website visitors

An example of visualization
with t-SNE showing
semantic clusters

Fahd Kalloubi Machine Learning Techniques 34

 Semi-supervised learning
• mix between supervised and unsupervised learning
• some training examples contain outputs, but some do not
• use the labeled training subset to label the unlabeled
portion of the training set, which we then also utilize for
model training

Fahd Kalloubi Machine Learning Techniques 35

 Reinforcement learning

• A set of states
• A set of actions
• A set of rewards

• Goal: take actions to change the state so that you receive

rewards
• You don’t receive any data – you have to explore the
environment yourself to gather data as you go
Fahd Kalloubi Machine Learning Techniques 36
 Reinforcement learning
Example: chess
• States are valid states of the chess board
• Actions at a given time are valid possible moves
• Positive rewards for taking pieces, negative rewards for losing them

Fahd Kalloubi Machine Learning Techniques 37

 Reinforcement learning
Example: chess
• States are valid states of the chess board
• Actions at a given time are valid possible moves
• Positive rewards for taking pieces, negative rewards for losing them

Fahd Kalloubi Machine Learning Techniques 38

 Reinforcement learning
Why is this difficult ?
• Stochastic
– Make the same move twice, the opponent might not do the same thing
– Rewards also stochastic (opponent does or doesn’t take your piece)
• Temporal credit assignment problem
– Did we get the reward because of this move? Or because we made good
tactical decisions somewhere in the past?
• Exploration-exploitation trade-off
– If we found a good opening, should we use this?
– Or should we try other things, hoping for something better?
Fahd Kalloubi Machine Learning Techniques 39
 The supervised Learning Workflow

1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI
Fahd Kalloubi Machine Learning Techniques 40
 Supervised Learning Workflow

Fahd Kalloubi Machine Learning Techniques 41

 Supervised Learning Workflow
Using a test dataset to evaluate the performance of a predictive model

Fahd Kalloubi Machine Learning Techniques 42

 Structured vs Unstructured Data

Fahd Kalloubi Machine Learning Techniques 43

 Machine Learning vs Deep Learning

Image source: Stevens et al., Deep Learning with PyTorch. Manning, 2020

Fahd Kalloubi Machine Learning Techniques 44

 Machine Learning vs Deep Learning

Image source: Stevens et al., Deep Learning with PyTorch. Manning, 2020

Fahd Kalloubi Machine Learning Techniques 45

Machine Learning Terminology and
notation (this also applies to DL)
1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI

Fahd Kalloubi Machine Learning Techniques 46

 Machine Learning Jargon 1/2
• Supervised learning
– Learn function to map x (features) to output y (targets)
• Structured data
– Databases, Spreadsheets/csv files (tabular data)
• Unstructured data
– features like image pixels, audio signals, text sentences (before DL,
extensive feature engineering was required)

Image source: Adrian Bulat, 2D and 3D Face alignment library build using pytorch
Fahd Kalloubi Machine Learning Techniques 47
 Supervised Learning (more formal Notation)
Training examples

Training set:
Unknown function:
Hypothesis:

Classification Regression

Fahd Kalloubi Machine Learning Techniques 48

 Data representation

Feature vector

Fahd Kalloubi Machine Learning Techniques 49

 Data representation

Feature vector Design Matrix Design Matrix

Fahd Kalloubi Machine Learning Techniques 50

 Machine Learning Jargon 2/2
• Training a model = fitting a model = parameterizing a model = learning from
data
• Training example, synonymous to training record, training instance, training
sample (in some contexts, sample refers to a collection of training examples)
• Feature, synonymous to observation, predictor, variable, independent variable,
input, attribute, covariate
• Target, synonymous to outcome, ground truth, output, response variable,
dependent variable, (class) label (in classification)
• Output / Prediction, use this to distinguish from targets; here, means output from
the model

Fahd Kalloubi Machine Learning Techniques 51

The practical aspects and tools

1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI

Fahd Kalloubi Machine Learning Techniques 52

 Main scientific Python Libraries

Source: [Link]
Main tools for this course
Fahd Kalloubi Machine Learning Techniques 53
 Main scientific Python Libraries
The State of Machine Learning Frameworks in 2019

Source: [Link]
Fahd Kalloubi Machine Learning Techniques 54
 Main scientific Python Libraries
The State of Machine Learning Frameworks in 2019

Source: [Link]
Fahd Kalloubi Machine Learning Techniques 55
 Main scientific Python Libraries
The State of Machine Learning Frameworks over the past 5 years

Source: google trends

Fahd Kalloubi Machine Learning Techniques 56

Computational requirements for AI

1. Course overview
2. What is machine learning ?
3. The broad categories of ML
4. The supervised learning workflow
5. Necessary ML notation and jargon
6. About the practical aspects and tools
7. Computational requirements for AI

Fahd Kalloubi Machine Learning Techniques 57

 Computational
requirements for AI

1. The evolution of AI hardware

2. Challenges & Constraints
3. Future of AI Hardware

Fahd Kalloubi Machine Learning Techniques 58

 Why AI needs hardware ?
Enough
memory?

Enough
compute?
• In 2023: Internet user ~ 15.87 TB of data daily.
• In 2023: YouTube videos ~ 4.3 PB per day.
• By 2024: TikTok videos ~ 7.35 TB per day.
• By 2024: Self-driving car ∼ 20 TB per day
• By 2025: cloud storage ~ 200+ zettabytes.

Fahd Kalloubi Machine Learning Techniques 59

 Key AI Hardware components

Shift from CPUs to GPUs

• Transitioned from general-purpose CPUs to GPUs
for faster, parallel processing in AI, greatly reducing
training times.

Rise of TPUs and NPUs

• TPUs and NPUs, tailored for AI tasks, enhance
efficiency and performance in specialized
applications.

Key AI Hardware Components

• Includes GPUs, TPUs, fast memory (RAM, cache), [Link]
and SSDs, crucial for rapid data transfer in AI ground-evolution-ai-hardware-technology-ritesh-
vajariya-wogke/
systems.

Fahd Kalloubi Machine Learning Techniques 60

 Key AI Hardware components

By 2030, we are likely to

witness microprocessor
products with over 1
trillion transistors in a
package!

[Link]
moores

Fahd Kalloubi Machine Learning Techniques 61

 GPUs: Revolutionizing Deep Learning
A graphics processing unit (GPU) is an electronic circuit that can perform mathematical
calculations at high speed.

Architectural Advantage of GPUs

• Equipped with thousands of cores for efficient
parallel processing, essential for deep learning.

Speeding Up AI Computation with GPUs

• Reduces neural network training time from weeks to
days or hours, facilitating faster AI development.

Expanding AI Applications
• Enhanced GPU power and efficiency enable more
complex models, advancing AI in various sectors
like healthcare and autonomous driving.

Fahd Kalloubi Machine Learning Techniques 62

 GPUs: Revolutionizing Deep Learning

GPUs become Faster

BUT

Ex pensive and Energy consumer

H100 | H200: 700 watts

B100 | B200: 1000 watts

Fahd Kalloubi Machine Learning Techniques 63

 Specialized AI Chips: TPUs and NPUs

Design for AI Specificity

• TPUs and NPUs are specifically engineered for
neural network operations, optimizing AI task
performance.

Enhanced Processing Capabilities

• These specialized chips significantly outperform
traditional units in speed and efficiency for high-
volume AI computations.

Applications in Advanced Projects

• Google's TPUs are integral to their AI
advancements; NPUs enhance consumer devices
with features like image recognition and real-time
language processing.
[Link]

Fahd Kalloubi Machine Learning Techniques 64

 TPUs: Evolution
Tensor Processing Unit (TPU) is a specialized hardware accelerator designed by Google specifically
for accelerating machine learning tasks.

TPUv1 TPUv3 TPUv5

Doubled power, Incorporates deep
Enhancing specific
deployed in larger reinforcement learning
machine learning
pods for 8x in design; latest in TPU
tasks
performance tech.
increase.

2015 2017 2018 2021 2023

TPUv2 TPUv4
Improved memory and More than doubled
power; performance of TPUv3.
supports floating-point
for training and
inference.

Fahd Kalloubi Machine Learning Techniques 65

 NPUs: Evolution
Neural processing units (NPUs) are specialized chips that are designed to accelerate the processing
of neural networks.

• The first neural network accelerators began

appearing in 2014

• NPUs are more widely available than TPUs,

because they are produced by many companies,
such as Intel, Nvidia, Huawei, and Samsung

• NPUs are faster than both GPUs and TPUs to

perform AI tasks

• Used in Edge computing and cloud computing

[Link]
of-ai-hardware-explained-c532c12913fd

Fahd Kalloubi Machine Learning Techniques 66

 Hardware requirements for AI

Comparative analysis

Fahd Kalloubi Machine Learning Techniques 67

 Memory and storage for AI

Memory and Storage: The Backbone

• Critical for rapid data access and handling large AI
workloads, involving fast RAM and efficient SSDs.

RAM: Essential for Fast Data Access

• Vital for quick data access in AI processors, significantly boosting
the speed of AI training and operations.

Cache: Reducing Processing Delays

• Key in minimizing data retrieval times, storing frequently accessed
DRAM => DDR5 with Transfer speed of 38,4 to 51,2 GB/s
data close to the processor to speed up computations.
SRAM/Cache => AMD EPYC 7773X with 768M Cache
SSDs: Accelerating Data Flow
• Solid-State Drives (SSDs) provide faster read-write speeds than PCIe 5.0 NVMe SSDs => Deliver theoretical read speeds
HDDs, essential for efficiently managing large datasets in AI of 14,000 MBps
applications.

Fahd Kalloubi Machine Learning Techniques 68

 Interconnects and Data Transfer for AI

Role of Interconnects
• Critical for enabling high-speed data transfer among
AI hardware components like processors, memory,
and storage.

Reducing Bottlenecks
• Efficient interconnects are crucial for preventing
data bottlenecks, ensuring smooth and fast AI
operations.

Advancements in Technology
• Continuous improvements in interconnect
technology, including faster and more efficient data
transfer protocols, further boost AI system
performance.
[Link]
interconnect-for-high-performance-computing-and-ai/
InfiniBand
Optical fiber
5G
Fahd Kalloubi Machine Learning Techniques 69
 Challenges in AI Hardware

High Power Consumption

• AI hardware consumes significant power during intensive tasks, impacting costs and
sustainability.

Heat Management
• Advanced cooling is required to manage heat from high-performance AI processors
and ensure system stability.

Other Challenges
• Managing physical space for hardware and ensuring reliability during continuous
operation are also key concerns.

Fahd Kalloubi Machine Learning Techniques 70

 The Future of AI Hardware

Advanced Chip Architectures

• New chip designs aim to boost AI processor performance and efficiency, tackling
more complex tasks with reduced power consumption.

Integration of AI into More Devices

• AI hardware is expanding into a broader array of devices, such as cell phones and
home appliances, making AI more prevalent in daily life.

Sustainable and Energy-Efficient Technologies

• Growing environmental concerns are driving the development of energy-efficient AI
hardware that optimizes performance while minimizing power usage.

Fahd Kalloubi Machine Learning Techniques 71

 Next Lecture:
A Brief Summary of the History of
Neural Networks and Deep
Learning

Fahd Kalloubi Machine Learning Techniques 72

 Lecture Topics

1. Artificial neurons
2. Multilayer neural networks
3. Deep learning
4. Current research trends

Fahd Kalloubi Machine Learning Techniques 73

 Neural network and deep learning – A timeline
McCulloch & Pitt's neuron model (1943)
McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. The Bulletin of Mathematical Biophysics, 5(4), 115-133.

Image Source: [Link]

Mathematical formulation of a
biological neuron, could solve
AND, OR, NOT problems

Fahd Kalloubi Machine Learning Techniques 74

 McCulloch & Pitt's neuron model (1943)

Mathematical formulation of a
biological neuron, could solve
AND, OR, NOT problems

Fahd Kalloubi Machine Learning Techniques 75

 Neural network and deep learning – A timeline
Frank Rosenblatt's Perceptron (1957)
A learning algorithm for the neuron model
Rosenblatt, F. (1957). The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory.

Inspired by

McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous
activity. The bulletin of mathematical biophysics, 5(4), 115-133.

Hebb, D. O. (1949). The organization of behavior. A neuropsychological theory.

Image source: [Link]

Perceptron#/media/
File:Mark_I_perceptron.jpeg

Fahd Kalloubi Machine Learning Techniques 76

 Neural network and deep learning – A timeline
Widrow and Hoff's ADALINE (1960)
A nicely differentiable neuron model
Widrow, B., & Hoff, M. E. (1960). Adaptive switching circuits (No. TR-1553-1). Stanford Univ Ca Stanford Electronics Labs.
Widrow, B. (1960). Adaptive" adaline" Neuron Using Chemical" memistors.".

A "single layer neural network"

Image source:
[Link]
publication/265789430/figure/fig2/AS:392335251787780@14
70551421849/ ADALINE-An-adaptive-linear-neuron-
[Link]

Source: Raschka and Mirjalily (2019). Python Machine Learning, 3rd Edition

Fahd Kalloubi Machine Learning Techniques 77

 Neural network and deep learning – A timeline

Minsky and Pappert (1969): "Perceptrons" book

 Problem: Perceptrons (and ADALINEs) could not solve XOR problems!
 Neurons, a dead end? Start of the first "AI Winter"

Minsky, M. L., & Papert, S. A. (1969). Perceptrons: an introduction to computational geometry. MA: MIT Press, Cambridge.

Fahd Kalloubi Machine Learning Techniques 78

 Training neural networks
with multiple layers

1. Artificial neurons
2. Multilayer neural networks
3. Deep learning
4. Current research trends

Fahd Kalloubi Machine Learning Techniques 79

 Multilayer Neural Networks Can Solve XOR Problems

Decision boundaries of two different multilayer perceptrons on simulated data solving the XOR
problem

[Link]

Fahd Kalloubi Machine Learning Techniques 80

 Multilayer Neural Networks – A timeline
• Solution to the XOR problem: hidden layers and non-linear activation functions
• New problem: Hard to train
• Solution: Backpropagation

Note that backpropagation has been independently

formulated many times ...
[Link]

Rumelhart and Hinton (1986) formulated it independently and then showed

that it really works (and formed the basis of all consequent neural network and
DL progress):
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533.

Also, it was later shown that "neural nets" are universal function approximators

Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359-366.

Fahd Kalloubi Machine Learning Techniques 81

 Multilayer Neural Networks – A timeline
Rumelhart and Hinton (1986) formulated it independently and then showed
that it really works (and formed the basis of all consequent neural network and
DL progress):
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533.

Fahd Kalloubi Machine Learning Techniques 82

 Suggestions for Pleasure Reading

Fahd Kalloubi Machine Learning Techniques 83

 The Origins of Deep
Learning
Representation learning with advanced architectures with
many layers & algorithmic improvements for better
computational efficiency and convergence

1. Artificial neurons
2. Multilayer neural networks
3. Deep learning
4. Current research trends

Fahd Kalloubi Machine Learning Techniques 84

 Convolutional Neural Networks
Shortly after followed a breakthrough in image recognition using
some clever enhancements to

a) make training more efficient

b) extract local features (and better capture feature dependency)

By

• Weight sharing
• Pooling

LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W.,
& Jackel, L. D. (1989). Backpropagation applied to handwritten zip code
recognition. Neural computation, 1(4), 541-551.

Fahd Kalloubi Machine Learning Techniques 85

 Recurrent Neural Networks
Recurrent Neural Networks and Backpropagation through time
created in the 1980's based on Rumelhart's work

New problem: Vanishing and exploding gradients

Schmidhuber, Jürgen (1993). Habilitation thesis: System modeling and optimization.

Page 150 ff demonstrates credit assignment across the equivalent of 1,200 layers in an
unfolded RNN.

Solution: LSTMs (still popular and commonly used)

Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780.

Fahd Kalloubi Machine Learning Techniques 86

 About the term “Deep Learning”

Representation learning is a set of methods that allows a machine to be fed

with raw data and to automatically discover the representations needed for
detection or classification. Deep learning methods are representation-
learning methods with multiple levels of representation [...]

-- LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep

learning. Nature, 521(7553), 436.

Fahd Kalloubi Machine Learning Techniques 87

 Neural Networks and Deep Learning – A timeline
• 2nd "AI Winter" in the late 1990's and 2000's
• Probably due to popularity of Support Vector Machines and Random
Forests
• Also, neural networks were still expensive to train, until GPUs came into
play
Oh, K. S., & Jung, K. (2004). GPU implementation of neural networks. Pattern Recognition, 37(6), 1311-1314.

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural
networks. In Advances in neural information processing systems (pp. 1097-1105).

(data from ~2017)

Image source: [Link]
Fahd Kalloubi Machine Learning Techniques 88
 When did Deep Learning Become Really Popular?

AlexNet achieved 15.4%

error on top-5 in 2012
2nd best was not even
close: 26.2% (nowadays
~3% error on ImageNet)

Fahd Kalloubi Machine Learning Techniques 89

 Evolution of Benchmark Datasets for Computer Vision
MNIST (1998)
• 60,000 examples, 10 classes
• features: 28x28x1
• [Link]

CIFAR-10/CIFAR-100 (2009)
• 60,000 examples, 10 or 100 classes
• features: 32x32x3,
• [Link]

ImageNet (~2010)
• ~14 million images
• features: full resolution
• [Link]

Fahd Kalloubi Machine Learning Techniques 90

 Neural Networks and Deep Learning – A timeline
• Many enhancements where developed to make neural networks perform
better and solve new problems

Rectified Linear Units BatchNorm

Nair, V., & Hinton, G. E. (2010). Rectified linear units Ioffe, S., & Szegedy, C. (2015, June). Batch Normalization:
improve restricted boltzmann machines. In Proceedings of Accelerating Deep Network Training by Reducing Internal
the 27th international conference on machine learning Covariate Shift. In International Conference on Machine
(ICML-10) (pp. 807-814). Learning (pp. 448-456).

Dropout GANs
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-
Salakhutdinov, R. (2014). Dropout: a simple way to prevent Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative
neural networks from overfitting. The Journal of Machine adversarial nets. In Advances in neural information processing
Learning Research, 15(1), 1929-1958. systems (pp. 2672-2680).

(All covered in this course)

& many more
Fahd Kalloubi Machine Learning Techniques 91
 Training neural networks
with multiple layers

1. Artificial neurons
2. Multilayer neural networks
3. Deep learning
4. Current research trends

Fahd Kalloubi Machine Learning Techniques 92

 Self-supervised learning
Self-supervised learning and computer vision by Jeremy Howard;
https:// [Link]/2020/01/13/self_supervised/

contrastive learning gained popularity in self-supervised

representation learning
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020). Unsupervised learning of
visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09882.

Instance discrimination compares features from different transformations of the

same images to each other

Fahd Kalloubi Machine Learning Techniques 93

 Graph Neural Networks
A gentle introduction to graph neural networks:
(

[Link]

Hot Subfield in ML
Fahd Kalloubi Machine Learning Techniques 94
 Graph Neural Networks
Learning Node embedding
• Goal is to encode nodes so that similarity in the embedding space (e.g., dot
product) approximates similarity in the graph

• Function ENC() encodes nodes into embedding space

• ENC() is learned from the graph

• These embeddings are task independent

• They are not trained for a specific task but can be used for any task.
Fahd Kalloubi Machine Learning Techniques 95
 Graph Neural Networks
Digging deeper into GML and GNNs

Stanford CS224W: Machine

Learning with Graphs

• Excellent lecture by Jure

Leskovec
• Slide sets
• [Link]

• Lecture videos
• [Link]
PLKxIpqhjhPgdQy7imNkDn

• Student projects
• [Link]

Fahd Kalloubi Machine Learning Techniques 96

 Large-scale language models

[Link]

Model sizes of language models from 2018–2020 (Credit: State of AI Report 2020)

Fahd Kalloubi Machine Learning Techniques 97

 [Link]

"Transformer is data-hungry in nature

e.g., a large- scale dataset like ImageNet
[14 million images] is not enough to train
vision transformer from scratch so [10]
proposes to ..."

Fahd Kalloubi Machine Learning Techniques 98

 Next Lecture:
Linear Algebra and Vector Calculus

Fahd Kalloubi Machine Learning Techniques 99