DEEP LEARNING

23DS5PCDLG
• Deep Generative Models: • Applications of Deep
Learning:
• Boltzmann Machines
• Large-Scale Deep Learning
• Restricted Boltzmann Machines
• Computer Vision
• Deep Belief Networks • Speech Recognition
• Deep Boltzmann Machines • Natural Language Processing
• Boltzmann Machines for Real- • Other Applications
Valued Data
Deep Generative Models
• These models represent probability distributions over multiple
variables.
Some allow explicit evaluation of the probability distribution
function.
Others support operations like sampling, which implicitly require
knowledge of the distribution.
Some models are structured as probabilistic graphical models with
factors.
Others cannot easily be described as factors but still represent
probability distributions.
Boltzmann Machines
• Boltzmann machines were invented in 1985 by Geoffrey Hinton and
Terrence Sejnowski.
• They are fully connected ANNs.
• Boltzmann machines use stochastic neurons instead of deterministic
step functions.
• Neurons output 1 with a certain probability, otherwise 0; based on
the Boltzmann distribution.
• The probability of a neuron outputting 1 is given by Equation:
Probability that the ith neuron will output 1
• Neurons in Boltzmann machines are separated into two groups:
visible units and hidden units.
• All neurons work in the same stochastic way, but the visible units are
the ones that receive the inputs and from which outputs are read.
• A Boltzmann machine doesn't stabilize into a fixed configuration; it
keeps switching between many configurations.
• After running long enough, the probability of a configuration depends
on connection weights and bias terms, not the initial state.
• This state is called thermal equilibrium, where the configuration keeps
changing.
• We can simulate various probability distributions by setting
parameters and allowing thermal equilibrium.
• This makes the Boltzmann machine a generative model.
• Training a Boltzmann machine aims to approximate the training set’s
probability distribution.
• For example, it can generate random binary triplets with the same
probability distribution as the training data.
• A generative model can repair incomplete or noisy images and be
used for classification by encoding the class as visible neurons.
• When given a new image, the network automatically identifies the
class by activating the corresponding visible neuron.
• While Boltzmann machines lack efficient training methods, restricted
Boltzmann machines (RBMs) have more efficient training algorithms.
Restricted Boltzmann
Machines(RBM)
• An RBM is a Boltzmann machine in which there are no connections
between visible units or between hidden units, only between visible
and hidden units.
• For example, Figure represents an RBM with three visible units and
four hidden units.
• Contrastive Divergence is an efficient training algorithm for Boltzmann
machines.
• For each training instance x, the algorithm starts by feeding it to the
network by setting the state of the visible units to x1, x2, ⋯, xn.
• Then compute the state of the hidden units by applying the stochastic
equation (Slide 5).
• This gives a hidden vector h (where hi is equal to the state of the ith
unit).
• Next compute the state of the visible units, by applying the same
stochastic equation. This gives you a vector xʹ.
• Then once again compute the state of the hidden units, which gives a
vector hʹ.
• Now update each connection weight by applying the rule in Equation,
where η is the learning rate.
• Contrastive Divergence is highly efficient as it doesn’t require thermal
equilibrium.
• It simply goes forward, backward, and forward again to update
weights.
• This efficiency made it a key component in the success of Deep
Learning with stacked RBMs.
Deep Belief Networks
• Several layers of RBMs can be stacked; the hidden units of the first-
level RBM serve as the visible units for the second-layer RBM, and so
on.
• Such an RBM stack is called a deep belief net (DBN).
• It was possible to train DBNs one layer at a time using Contrastive
Divergence, starting with the lower layers and then gradually moving
up to the top layers.
• Unsupervised Learning: DBNs, like RBMs, model the probability
distribution of inputs without supervision.
• Hierarchical Structure:
• Lower layers: Learn low-level features.
• Higher layers: Learn high-level features.
• Supervised & Semisupervised Learning:
• DBNs can also be trained in a supervised way by adding visible units for labels.
• Semisupervised learning: DBNs excel in semisupervised learning (partly
labeled data).
A deep belief network configured for
semisupervised learning
• Training a DBN: Step-by-Step
• RBM 1 (Unsupervised):
• Trained without supervision to learn low-level features of the input data.
• RBM 2 (Unsupervised):
• Trained using RBM 1's hidden units as inputs to learn higher-level features.
• RBM 3 (Supervised):
• Trained using RBM 2's hidden units as inputs, along with extra visible units for
labels.
• Learns to associate high-level features with training labels.
• Final Classification (DBN):
• After training, when a new instance is fed into RBM 1, the signal
propagates up through RBM 2 and RBM 3, eventually reaching the
label units, where the correct label should be predicted.
• Benefits of Semisupervised Learning with DBNs
• Less Labeled Data Needed:
• Unsupervised RBMs learn useful features, requiring only a
small amount of labeled data for training.
• Human Analogy (Baby Learning):
• Like a baby associating words with objects, DBNs can learn
to associate labels with features using a few examples.
• Reverse Operation of DBNs (Generative Capability)
• Generative Power:
• Activating a label unit in DBNs can generate a new
instance.
• Example: Activating the "chair" label can generate a new
image resembling a chair.
• Image-Caption Generation Example:
• Training:
• Two unsupervised DBNs are trained:
• One on images to learn visual features.
• One on captions to learn text features.
• Associating Features:
• An RBM is stacked on top of both DBNs, associating image features with
caption features.
• Generating Captions:
• Feed an image (e.g., a car) into the image DBN.
• The caption DBN generates a caption for the image.
• The most frequent caption generated (from random variations) will describe
the image.
Deep Boltzmann Machines
• Overview:
1. A DBM is a type of probabilistic graphical model that is deep and
undirected.
2. It consists of multiple layers of stochastic binary units.
3. DBMs learn to represent complex, high-dimensional distributions of input
data.
• Architecture:
• Multiple Layers of RBMs:
• DBMs are stacked layers of Restricted Boltzmann Machines (RBMs).
• RBMs: Unsupervised models that learn to capture the distribution of their
input.
• Learning Process:
• Unsupervised Training:
• Each layer learns to capture low-level to high-level features from the data.
• Energy-based Model: The learning process minimizes the energy function to
make the model better at reproducing the data distribution.
• Advantages of DBMs:
Deep Representation:
• Captures complex patterns by stacking layers of RBMs.
Generative Modeling:
• Can generate new data by sampling from learned distributions.
Boltzmann Machines for Real-Valued
Data
• Standard Boltzmann Machine (BM):
• Originally designed for binary data (units with values of 0
or 1).
• Challenges with Real-Valued Data:
• Real-valued data: Requires handling continuous input values,
not just binary ones.
• Solution: Use a Gaussian-Boltzmann Machine (GBM) or
Continuous-Boltzmann Machine.
Gaussian-Boltzmann Machine (GBM)
• Real-Valued Units: The visible units are modeled using Gaussian
distributions.
• Energy Function:
• The energy function is adapted to handle continuous values.
• Probabilistic Output: The network’s output is a probability distribution for
real-valued inputs.
• Training for Real-Valued Data:
• Similar to standard BMs, but with an adaptation to minimize errors for
continuous data.
• Learning Process: Uses contrastive divergence or persistent
contrastive divergence for efficient training.
• Advantages:
Can Model Continuous Data: Unlike standard BMs, can handle real-
valued inputs like images, time-series, and more.
Generative Power: Generates data samples with continuous features.
---XXX---