Deep Learning
Module 1 and module 2 – part 4
Parametric Learning
• A learning model that summarizes data with a
set of parameters of fixed size (independent of
the number of training examples) is called a
parametric model.
The algorithms involve two steps:
– Select a form for the function.
– Learn the coefficients for the function from the
training data.
Parametric Learning - example
• An easy to understand functional form for the mapping
function is a line, as is used in linear regression:
b0 + b1*x1 + b2*x2 = 0
• Where b0, b1 and b2 are the coefficients of the line
that control the intercept and slope, and x1 and x2 are
two input variables.
• Assuming the functional form of a line greatly
simplifies the learning process.
• Now, all we need to do is estimate the coefficients of
the line equation and we have a predictive model for
the problem.
Parametric Learning Algorithms
• Logistic Regression
• Linear Discriminant Analysis
• Perceptron
• Naive Bayes
Parametric Learning
Benefits
• Simpler: These methods are easier to
understand and interpret results.
• Speed: Parametric models are very fast to
learn from data.
• Less Data: They do not require as much
training data and can work well even if the fit
to the data is not perfect.
Parametric Learning
Limitations
• Constrained: By choosing a functional form
these methods are highly constrained to the
specified form.
• Limited Complexity: The methods are more
suited to simpler problems.
• Poor Fit: In practice the methods are unlikely
to match the underlying mapping function.
Non Parametric Learning
• Algorithms that do not make strong
assumptions about the form of the mapping
function are called nonparametric machine
learning algorithms.
• By not making assumptions, they are free to
learn any functional form from the training
data.
Non-Parametric Learning Algorithms
• k-Nearest Neighbors
• Decision Trees
• Support Vector Machines
Non-Parametric Learning Algorithms
Benefits
• Flexibility: Capable of fitting a large number of
functional forms.
• Power: No assumptions (or weak
assumptions) about the underlying function.
• Performance: Can result in higher
performance models for prediction.
Non-Parametric Learning Algorithms
Limitations
• More data: Require a lot more training data to
estimate the mapping function.
• Slower: A lot slower to train as they often
have far more parameters to train.
• Overfitting: More of a risk to overfit the
training data and it is harder to explain why
specific predictions are made.
Deep Neural Networks can be parametric as well
as non-parametric
Restricted Boltzmann Machine (RBM)
Model probability and are great at feature extraction.
Feedforward networks in which data is fed through them
in one direction with two biases rather than one bias as
in traditional backpropagation feed-forward networks.
Used in deep learning for the following:
Feature extraction
Dimensionality reduction
The “restricted” part of the name “Restricted Boltzmann Machines”
- connections between nodes of the same layer are prohibited
(e.g., there are no visible-visible or hidden-hidden connections
along which signal passes).
Geoff Hinton—” A network of symmetrically connected, neuron-
like units that make stochastic decisions about whether to be on
or off.”
Restricted Boltzmann Machine (RBM)
Also, a type of autoencoder
Used for pretraining layers in larger networks such as
Deep Belief Networks.
Network layout
5 main parts of a basic RBM:
Visible units
Hidden units
Weights
Visible bias units
Hidden bias units
A standard RBM has a visible layer and a hidden layer
Every visible unit is connected to every hidden unit,
Each layer of an RBM can be imagined as a row of
nodes.
The nodes of the visible and hidden layers are
connected by connections with associated weights
Restricted Boltzmann Machine (RBM)
Network layout
Visible and
hidden layers-
Connections and
weights-
Biases-
Training-
Reconstruction
Restricted Boltzmann Machine (RBM)
Reconstruc
tion in
RBMs
RBM
RBM
RBM
Restricted Boltzmann Machine (RBM)
Other uses of RBMs
Dimensionality reduction
Classification
Regression
Collaborative filtering
Topic modelling
Imbalance data problem
Unstructured data
Noisy label problems