Basics of Machine Learning
Jun Zhu
dcszj@[Link]
[Link]
Sate Key Lab of Intelligent Tech. & Systems,
Tsinghua University
What is Machine Learning?
Machine learning, a branch of artificial intelligence, is a scientific
discipline concerned with the design and development of
algorithms that take as input empirical data, and yield patterns or
predictions thought to be features of the underlying mechanism
that generated the data
Data
23453981100010202000
Learning algorithm
23235537845120123568
54596312785294520126
Knowledge
2
(Statistical) Machine Learning in AI
post-Pearl
since 1990s
Great growth in machine learning
AI
1970s & 1980s: expert systems
1960s & 1970s: success on NLP systems
pre-Pearl
1950s & 1960s: symbolic reasoning,
representation, search
AI
pre-computer age
thoughts on symbolic reasoning
[Judea Pearl, Turing Award 2011]
• For "innovations that enabled remarkable advances in the partnership between
humans and machines that is the foundation of Artificial Intelligence (AI)”
• “His work serves as the standard method for handling uncertainty in computer systems,
with applications from medical diagnosis, homeland security and genetic counseling to
natural language understanding and mapping gene expression data.”
• “Modern applications of AI, such as robotics, self-driving cars, speech recognition, and
machine translation deal with uncertainty. Pearl has been instrumental in supplying the
rationale and much valuable technology that allow these applications to flourish.” 3
“Machine learning will become a calculus”
--- Tom Mitchell
4
Machine learning in Action
• Document classification
Sports
News
Politics
…
5
Spam Filter
Dear Jun:
2015 ASE Data Science conference will be held at Stanford University, August
18-20, 2015. You can find more details via the conference website
([Link] We would like to invite
Spam /
you to our program committee. Could you please confirm your availability by not Spam
March 6, 2015.
Thanks,
Justin
6
Regression
• Stock market prediction
7
Computer Vision
• Image Classification, Face recognition, Scene understanding, Action/behavior
recognition, Image tagging and search, Optical character recognition (OCR)
8
Speech Recognition
• A classic problem in AI, very difficult!
– “Let’s talk about how to wreck a nice beach”
– small vocabulary is – easy
– challenges: large vocabulary, noise, accent, semantics
9
Control
• Cars navigating on their own
– DARPA urban challenge
– Tsinghua Mobile Robot V (THMR-V):
10
Control (cont’d)
• How the Google car works
11
AlphaGO
• March, 2016: AlaphaGO beats Sedol Lee at 4:1
12
Science
• Decoding thoughts from brain activity
Tool
Animal
…
[Mitchell et al, Science 2008]
[Kay et al., Nature, 2008]
13
Science (cont’d)
• Bayesian models of inductive learning and reasoning [Tenenbaum
et al., Science 2011]
– Challenge:
• How can people generalize well from sparse, noisy, and ambiguous data?
– Hypothesis:
• If the mind goes beyond the data given, some more abstract background
knowledge must generate and delimit the possible hypotheses
– Bayesian models make structured abstract knowledge and statistical
inference cooperate
– Examples
• Word learning [Xu & Tenenbaum, Psychol. Rev. 2007]
• Causal relation learning [Griffiths & Tenenbaum, 2005]
• Human feature learning [Austerweil & Griffiths, NIPS 2009]
• …
J. Tenenbaum et al., “How to grow a mind: Statistics, Structure, and Abstraction”. Science 331, 1279 (2011)
14
Science (cont’d)
• AlphaFold:
– an ML program developed by DeepMind which
performs predictions of protein structure
15
[Improved protein structure prediction using potentials from deep learning, Nature 2020]
Science (cont’d)
AlphaFold & AlphaFold2 solve the 50yrs grand challenge, in 2021
[Improved protein structure prediction using potentials from deep learning, Nature 2020]
16 16
[Highly accurate protein structure prediction with AlphaFold, Nature, 2021]
GPT for NLP
by OpenAI
(build on GPT3.5,
not clear on data)
2022.1, InstructGPT
2022.11, ChatGPT
2021.12, CodeX by OpenAI
by OpenAI (build on GPT3.5,
2020.5, GPT3
(6B params, not clear on data)
by OpenAI
2019.2, GPT2 800M lines of
(175B params, code)
2017, by OpenAI
~700GB text)
Transformer 2018.6, GPT1 (1.5B params,
by Google
by OpenAI 40GB text)
(117M params,
8M webpages)
17
AIGC for Visual Data
• UniDiffuser (ICML 2023) for cross-modal generation
Paper：[Link] GitHub：[Link]
UniDiffuser (Ours) Stable Diffusion DALL·E 2
参数量 9.5亿 8.9亿 35亿
训练数据量 50亿 50亿 6.5亿
跨模态互生成 Image-2-Text，Text-2-Image Text-2-Image Text-2-Image
水墨画 高清摄影
18
More others …
• Many more
– Natural language processing
– Speech recognition
– Computer vision
– Robotics
– Computational biology
– Social network analysis
– Sensor networks
– Health care
– Protest ??
– …
19
Machine learning in Action
• Machine learning for protest?
CMU ML students and post-docs at G-20 Pittsburgh Summit 2009
20
Machine Learning – practice
decoding brain signal face recognition
document classification
robot control stock market prediction
21
Machine Learning – theory
Other theories for
• semi-supervised learning
• reinforcement skill learning
• active learning
•…
… also relating to
• # mistakes during training
• asymptotic performance
• convergence rate
• bias, variance tradeoff
•…
[Leslie G. Valiant, 1984; Turing Award, 2010]
“For transformative contributions to the theory of computation, including the theory of
probably approximately correct (PAC) learning, the complexity of enumeration and of
algebraic computation, and the theory of parallel and distributed computing.” 22
23
Growth of Machine Learning in CS
• Machine learning already the preferred approach to
– Speech recognition, natural language process
– Computer vision
– Medical outcomes analysis
ML apps.
– Robot control
– …
All software apps.
• This ML niche is growing (why?)
24
Growth of Machine Learning in CS
• Machine learning already the preferred approach to
– Speech recognition, natural language process
– Computer vision Huge amount of data …
• Web: estimated Google index 45 billion pages
– Medical outcomes analysis
• Transaction data: 5-50 TB/day
ML apps.
– Robot control • Satellite image feeds: ~1TB/day/satellite
– … • Biological data: 1-10TB/day/sequencer
• TV: 2TB/day/channel;
All software apps.
• YouTube 4TB/day uploaded
• This ML niche is growing • Photos: 1.5 billion photos/week uploaded
– Improved machine learning algorithms
– Increased data capture, networking, new sensors
– Software too complex to write by hand
– Demand for self-customization to user, environment
25
ML still has a long way to go …
• Adversarial attack for deep neural networks
26
ML still has a long way to go …
• Unlocked 19 types of the mainstream smart phones within 15
minutes with one “adversarial glass”!
27
ML still has a long way to go …
• Adversarial attack for deep neural networks
– L4 Auto-Driving systems, attack success rate > 99%
28
ML still has a long way to go …
• Attack to large-scale pre-trained foundation models
Clean data
Output：a man wearing a suit is standing next to a car
Clear Data
Adversarial data
Output：a woman wearing a uniform walks next to a
horse carriage
Adversarial Data
Meta Segment Anything Model (SAM) MiniGPT-4 Opensourced LLM
29
ML still has a long way to go …
• Attack GPT-4 via API calls
30
ML still has a long way to go …
31
ML still has a long way to go …
• Facebook AI announced a challenge to detect fake videos
and dedicate more than $10 million to fund this industry-
wide effort
32
ML still has a long way to go …
• Over-confidence in OOD setting
Classification Question Answering by GPT-3
Human: How many eyes does a giraffe have?
GPT-3: A giraffe has two eyes.
Human: How many eyes does my foot have?
GPT-3: Your foot has two eyes.
Human: How many eyes does the sun have?
GPT-3: The sun has one eye.
Over-confident mistakes will propagate to the downstream decision-making! 33
ML still has a long way to go …
• Hard to interpret
Bird
#147 Neuron
Most active in VGG-
16 pool5
• 1. normal images
• 2. adversarial images
[Dong et al., Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples, arXiv 2017]
Basic principles of ML are NOT out-of-dated!
35
Machine Learning Tasks
• Broad categories
– Supervised learning
• Classification, Regression
– Unsupervised learning
• Density estimation, Clustering, Dimensionality reduction
– Semi-supervised learning
– Reinforcement learning
– Active learning
– Transfer learning
– Many more …
36
Supervised Learning
• Task: learn a predictive function
Feature space Label space
“Sports”
“News”
Words in documents “Politics”
…
Share price
Market information “$ 20.50”
up to time t
• “Experience” or training data:
37
Supervised Learning – classification
Feature space Label space
“Sports”
“News”
Words in documents “Politics”
…
“Tool”
“Animal”
…
Stimulus response
Discrete Labels
38
Supervised Learning – regression
Feature space Label space
Share price
Market information “$ 20.50”
up to time t
(session, location, time …) Temperature
“42o F”
Continuous Labels
39
How to learn a classifier?
C2
C1 ?
K-NN: a Non-parametric approach
Distance metric matters!
40
How to learn a classifier?
Parametric (model-based) approaches:
g(x) = 0; w here g(x) = w > x + w 0
C2
C1
a good decision boundary
½
¤ C 1 if g(x) > 0
y =
C 2 if g(x) < 0
41
How to learn a classifier?
C2
C1
Many good decision boundaries
which one should we choose?
42
How to learn a classifier?
How about non-linearity?
43
How to learn a classifier?
• 2D mapping is insufficient
44
How to learn a classifier?
How about non-linearity?
The higher dimension, the better?
45
How to learn a classifier?
• Curse of dimensionality
– A high dimensional space is always almost empty
d dimensional space
46
How to learn a classifier?
• Curse of dimensionality
– A high dimensional space is always almost empty
when one wants to learn pattern from data in high dimensions no matter how much
data you have it always seems less! 47
How to learn a classifier?
• Curse of dimensionality
– A high dimensional space is always almost empty
– … in high dimensions no matter how much data you have it always seems
less!
• The blessing of dimensionality
– … real data highly concentrate on low-dimensional, sparse, or degenerate
structures in the high-dimensional space.
• But no free lunch: Gross errors and irrelevant measurements are
now ubiquitous in massive cheap data.
48
How to learn a classifier?
• The blessing of dimensionality
– … real data highly concentrate on low-dimensional, sparse, or
degenerate structures in the high-dimensional space.
Images of the same face under varying illumination lie approximately on a low (nine)-
dimensional subspace, known as the harmonic plane [Basri & Jacobs, PAMI, 2003].
49
How to learn a classifier?
• Support vector machines (SVM) – basics
– SVM is among the most popular/successful classifiers
– It provides a principled way to learn a robust classifier (i.e., a decision
boundary)
support vectors
• SVM C2
– chooses the one with maximum margin C1
principle
– has sound theoretical guarantee
– extends to nonlinear decision boundary by
using kernel trick
– learning problem efficiently solved using
convex optimization techniques
margin
50
How to learn a classifier?
• Naïve Bayes classifier – basics
– an representative method from the very important family of probabilistic
graphical models and Bayesian methods
A joint distribution： p(x;y) = p(y)p(xjy) Y
Inference using Bayes rule： prior likelihood
p(yjx) =
p(x;y)
p(x)
=
p(y)p(xjy)
p(x)
X
evidence
Prediction rule： y ¤ = arg m ax p(yjx)
y2 Y
– fundamental building blocks for Bayesian networks
– nice illustrative example of Bayesian methods
51
How to learn a classifier?
• Many other classifiers
– K-nearest neighbors
– Decision trees
– Logistic regression
– Boosting
– Random forests
– Mixture of experts
– Deep neural networks
– Bayesian classifiers
– …
Advice #1:
All models are wrong, but some are useful. – G.E.P. Box
52
Are complicated models preferred?
• A simple curve fitting task
53
Are complicated models preferred?
• Order = 1
54
Are complicated models preferred?
• Order = 2
55
Are complicated models preferred?
• Order = 3
56
Are complicated models preferred?
• Order = 9?
57
Are complicated models preferred?
Advice #2: use ML & sophisticated models when necessary
• Issues with model selection!!
58
Unsupervised Learning
• Task: learn an explanatory function
• Aka “Learning without a teacher”
Feature space
Word distribution
(probability of a word)
Words in documents
• No training/test split
59
Unsupervised Learning – density estimation
Feature space Density function
geographical information of a location
60
Unsupervised Learning – clustering
[Link]
Feature space Cluster assignment function
Attributes (e.g., pixels & text) of images
61
Unsupervised Learning – dimensionality reduction
Images have thousands or
millions of pixels
Can we give each image a
coordinate, such that similar
images are near each other ?
Feature space Coordinate function in 2D space
pixels of images
62
Deep Generative Models
• Learn a generative model
63
Deep Generative Models
Generative modeling
with diffusion models
(ICLR’2022; ICML’22)
ICLR’22 Outstanding
paper award
Midjourney V5 – Creates Photorealistic Images
Prompt:「Portait of an older man sat in a • Prompt：「A pair of young Chinese
coffee shop, shot through a window. 」 lovers, wearing jackets and jeans,
sitting on the roof, the background is
Beijing in the 1990s, and the opposite
building can be seen --v 5 --s 250 --q 2」
DALLE3
• DALLE3
[Link]
Gen-2 – Text to Video
Prompt：
「The late afternoon sun
peeking through the window
of a New York City loft.」
[Link]
ProlificDreamer – Text to 3D
[Wang, Lu, et al., ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation,
NeurIPS 2023; Project page: [Link] ]
Summary: what is machine learning
• Machine Learning seeks to develop theories and computer systems for
dealing with
• complex, real world data, based on the system's own experience with data, and
(hopefully) under a unified model or mathematical framework, that
have nice properties.
69
Summary: what is machine learning
• Machine Learning seeks to develop theories and computer systems for
– representing;
– classifying, clustering, recognizing, organizing;
– reasoning under uncertainty;
– predicting;
– and reacting to
– …
• complex, real world data, based on the system's own experience with data, and
(hopefully) under a unified model or mathematical framework, that
have nice properties.
70
Summary: what is machine learning
• Machine Learning seeks to develop theories and computer systems for
– representing;
– classifying, clustering, recognizing, organizing;
– reasoning under uncertainty;
– predicting;
– and reacting to
– …
• complex, real world data, based on the system's own experience with data, and
(hopefully) under a unified model or mathematical framework, that
– can be formally characterized and analyzed;
– can take into account human prior knowledge;
– can generalize and adapt across data and domains;
– can operate automatically and autonomously;
– and can be interpreted and perceived by human.
• ML covers algorithms, theory and very exciting applications
• It’s going to be fun and challenging ☺
71
Resources for Further Learning
• Top-tier Conferences:
– International Conference on Machine Learning (ICML)
– Advances in Neural Information Processing Systems (NeurIPS)
– International Conference on Learning Representations (ICLR)
– Uncertainty in Artificial Intelligence (UAI)
– International Joint Conference on Artificial Intelligence (IJCAI)
– AAAI Annual Conference (AAAI)
– Artificial Intelligence and Statistics (AISTATS)
• Top-tier Journals:
– Journal of Machine Learning Research (JMLR)
– Machine Learning (MLJ)
– IEEE Trans. on Pattern Recognition and Machine Intelligence (PAMI)
– Artificial Intelligence
– Journal of Artificial Intelligence Research (JAIR)
– Neural Computation
72
Resources for Further Learning
• Text books:
– Pattern Recognition and Machine Learning
– Deep Learning
• Public lectures:
– CMU :
• [Link]
– Stanford:
• [Link]
• [Link]
– UPenn:
• [Link]
73
Thanks!
Jun Zhu
dcszj@[Link]
[Link]
74