DEEP LEARNING

23DS5PCDLG
• Deep Generative Models: • Applications of Deep Learning:
• Boltzmann Machines • Large-Scale Deep Learning
• Computer Vision
• Restricted Boltzmann Machines
• Speech Recognition
• Deep Belief Networks
• Natural Language Processing
• Deep Boltzmann Machines • Other Applications
• Boltzmann Machines for
Real-Valued Data
Large-Scale Deep Learning
• Deep Learning Philosophy: Based on connectionism—large networks of
neurons/features exhibit intelligent behavior.
• Individual neurons/features are not intelligent; their collective action
creates intelligence.
• Advancement Since 1980s:
• Dramatic growth in network sizes.
• Exponential increase in complexity and accuracy of tasks solved.
• Current Scale: Neural networks are still only as large as insect nervous
systems.
• The size of neural networks is of paramount importance, deep learning
requires high performance hardware and software infrastructure.
Fast CPU Implementations
• Traditional Training: Neural networks were initially trained on
single-machine CPUs, which is now insufficient.
• Modern Approach: GPU computing or networked CPUs are commonly
used for heavy workloads.
• CPU Optimization:
Efficient CPU code can improve performance significantly.
Example: In 2011, fixed-point arithmetic sped up neural networks by
3x compared to floating-point (Vanhoucke et al.).
Performance depends on CPU models; sometimes floating-point is
faster.
• Key Strategies:
Specialize numerical computation routines for the hardware.
Optimize data structures to reduce cache misses.
Use vector instructions for better efficiency.
• Impact: Ignoring optimization limits model size and reduces accuracy.
GPU Implementations
• GPUs in Neural Networks:
GPUs, originally developed for graphics, are now widely used in neural networks.
Video gaming demands for high-performance graphics hardware drove GPU
development.
• Why GPUs Work Well for Neural Networks:
Perform many operations in parallel efficiently.
Handle tasks like matrix multiplications and pixel color computations with minimal
branching.
High parallelism and memory bandwidth are ideal for neural network workloads.
• GPU Characteristics:
Optimized for independent, parallel computations.
High memory bandwidth for processing large data buffers.
Lower clock speed and branching capability compared to CPUs.
• Neural Networks and GPU Performance:
Neural networks require large buffers (parameters, activations,
gradients) updated each training step.
High memory bandwidth in GPUs overcomes CPU limitations, as
memory bandwidth often limits speed.
• Why GPUs Excel for Neural Networks:
Minimal branching and simple control align well with GPU
architecture.
Independent neuron processing enables effective parallelism.
• Challenges of GPU Programming:
Writing efficient GPU code is complex and differs from CPU
programming.
Key Differences:
• GPUs lack caching for writable memory; recomputation can be faster than
memory reads.
• Multi-threaded coordination is critical, with faster memory operations
through coalesced reads/writes.
• Threads in a warp must execute the same instruction; branching within warps
slows performance.
• Optimization Tips for GPU Code:
Align memory accesses across threads for coalescing(combine).
Avoid branching within warps for consistent instruction execution.
Tailor memory patterns to specific GPU model capabilities.
• Simplifying GPU Programming:
Avoid writing new GPU code for testing models/algorithms.
Use software libraries with high-performance operations (e.g.,
convolution, matrix multiplication).
• Benefits:
✔ Streamlines development and testing.
✔ Supports multiple hardware types (CPU, GPU) seamlessly.
Large-Scale Distributed Implementations
• Need for Distributed Computing:
Single machines often lack sufficient computational resources for
training and inference.
Workloads can be distributed across multiple machines.
• Types of Parallelism:
Data Parallelism: Each machine processes separate input examples
(simple for inference).
Model Parallelism: Multiple machines process different parts of the
model (works for inference and training).
• Challenges in Data Parallel Training:
Larger minibatches improve performance but with diminishing returns.
Gradient descent is inherently sequential, making parallelization harder.
• Solution: Asynchronous Stochastic Gradient Descent (ASGD):
Multiple processors compute gradients in parallel and update
parameters without locks.
Increased step frequency compensates for reduced individual step
efficiency.
Parameters managed by a parameter server for multi-machine setups.
• Adoption:
ASGD is widely used in industry for large-scale training.
Academic research explores low-cost distributed setups.
 Model Compression
for Efficient Inference
• Model Compression reduces the time and memory cost of running inference
without sacrificing performance.
• This is crucial in applications where inference needs to be efficient, such as in mobile
devices or resource-constrained environments.
• Commercial Application Example:
• Train models on powerful computer clusters (e.g., for speech recognition), then deploy them
on mobile phones.
• The end users often have more resource limitations than the developers.
• When to Apply Model Compression:
• When the original model is large primarily due to the need to prevent overfitting.
• Ensemble models with multiple independently trained models can be costly to evaluate; model
compression can help reduce this cost.
• Overfitting and Generalization:
• In some cases, large models (e.g., those using techniques like dropout) may generalize better
than smaller models, making them candidates for model compression.
• Training a Smaller Model through Model Compression
• Challenge with Large Models:
• Large models learn the function f(x) but use more parameters than necessary, mainly
due to the limited number of training examples.
• Solution: Generate More Training Data:
• Once the large model has learned f(x), we can generate an infinite number of new
training examples by applying f(x) to randomly sampled points x.
• Training the Smaller Model:
• The smaller model is then trained to match f(x) on these new examples.
• To efficiently use the capacity of the smaller model, the new points should come from a
distribution similar to the actual test inputs the model will see later.
• Sampling New Points:
• New points can be sampled by:
• Corrupting original training examples.
• Using a generative model trained on the original training set to generate new points.
• Alternative Approach:
• Train the smaller model on the original training points, but have it mimic other aspects of
the large model, such as its posterior distribution over incorrect classes.
Dynamic Structure
• Dynamic Structure in Data Processing Systems:
Systems can dynamically determine which subset of neural networks
should process a given input.
This approach allows for faster data processing by running only the
necessary networks based on the input.
• Conditional Computation:
• Neural networks can exhibit dynamic structure internally, deciding
which subset of features (or hidden units) to compute based on input
information.
• This is known as conditional computation.
• The goal is to compute only relevant features, improving efficiency.
• Benefit of Dynamic Structure:
• Many components of the network may only be relevant for a small
set of inputs, allowing the system to run faster by computing only
what's needed.
• Dynamic structure is a fundamental computer science concept,
widely applied in software engineering to improve system efficiency.
• Simple Applications in Neural Networks:
• Basic forms of dynamic structure involve determining which subset of
neural networks or models should be applied to a specific input.
 Cascade Classifiers for Accelerating Inference
• Cascade of Classifiers is a strategy used to accelerate inference, especially in
rare object detection tasks.
• The goal is to use low-cost computations for rejecting non-relevant inputs
while reserving high-capacity classifiers for confirming the presence of the
rare object.
• How Cascade Classifiers Work:
First Classifiers: Have low capacity and are trained for high recall (ensuring no false
negatives, i.e., missing the rare object).
Final Classifier: Has high precision and is used to confirm the object’s presence with
high confidence.
At test time, classifiers are applied sequentially, and inputs are abandoned early if
any classifier rejects them.
• Benefits:
• High confidence in detecting rare objects using high-capacity models without
the full cost of inference for every example.
• Reduces computational cost while maintaining accuracy.
• Two Ways to Achieve High Capacity:
• Later Classifiers with High Capacity: The final classifiers in the cascade are
high-capacity, making the overall system powerful.
• Multiple Low-Capacity Models: Many small models are combined to create a
high-capacity system overall.
• Example Use Cases:
• Viola and Jones (2001): Used a cascade of boosted decision trees for a fast and
robust face detection system in handheld digital cameras.
• Google Address Number Transcription: A two-step cascade that first locates
address numbers in Street View imagery and then transcribes them using a
different model.
Dynamic Structure in Decision Trees and
Neural Networks
• Decision Trees as Dynamic Structure:
• Each node in a decision tree dynamically determines which subtree to
evaluate for each input.
• This is an example of dynamic structure in machine learning models.
• Neural Networks and Decision Trees:
• A decision tree with neural networks at each node can be used to make
splitting decisions, combining dynamic structure with deep learning.
• Mixture of Experts (MoE):
• Gater Networks: A neural network (the gater) selects which expert network
to use for a given input.
• Soft MoE: The gater assigns probabilities to each expert using softmax, and
the final output is a weighted combination of expert outputs.
• This method does not reduce computation cost but provides a soft selection of
experts.
• Hard MoE: The gater selects one expert per example, accelerating training
and inference time by using only one expert.
• Works well when the gating decisions are few and non-combinatorial.
• Challenges with Combinatorial Gaters:
• Hard selection of units or parameters in large networks requires computing
outputs for all possible gating configurations, which can be computationally
expensive.
• Solutions:
• Gradient estimators for gating probabilities.
• Reinforcement learning techniques for conditional dropout on hidden unit blocks,
reducing computation cost without affecting model performance.
Dynamic Structure: Switch and Attention
Mechanism
• Dynamic Routing with Switches:
• Switch: A hidden unit can receive input from different units depending on
the context, enabling a form of dynamic routing.
• This can be interpreted as an attention mechanism, where the model
dynamically chooses which features to focus on.
• Challenges with Hard Switches:
• Hard switches (selecting a single input) have not been effective for
large-scale applications due to computational inefficiencies.
Modern methods use a weighted average over many possible inputs, which
sacrifices some of the computational benefits of true dynamic structure.
 Challenges of Dynamically Structured
1. Systems
Decreased Parallelism:
Dynamically structured systems reduce parallelism as different inputs follow different code
branches.
Operations are not easily expressed as matrix multiplications or batch convolutions, limiting
optimization on parallel hardware.
2. Specialized Sub-Routines:
More specialized sub-routines are needed, such as convolving each example with different
kernels or multiplying rows by different weight sets.
These specialized operations are difficult to implement efficiently.
3. Inefficiencies in CPU and GPU Implementations:
CPU: Slow due to lack of cache coherence.
GPU: Slow due to non-coalesced memory transactions and need to serialize warps when
members take different branches.
4. Mitigating Performance Issues:
Partitioning examples into groups that follow the same branch can improve performance in
offline settings.
Real-time settings: Partitioning can cause load-balancing issues, such as uneven distribution of
work in cascades or decision trees.
Specialized Hardware Implementations of Deep
Networks
• Types of Specialized Hardware:
ASICs (Application-Specific Integrated Circuits): Tailored for specific tasks.
Digital Hardware: Based on binary representations of numbers.
Analog Hardware: Utilizes physical continuous values like voltages or
currents.
Hybrid Implementations: Combine both digital and analog components.
FPGA (Field Programmable Gate Arrays): Flexible hardware, where circuits
can be programmed post-manufacture, allowing for adaptability.
Advancements: FPGAs offer increased flexibility, enabling custom
configurations after chip production.
Precision and Specialized Hardware for Deep Learning
• Floating Point Precision:
• General-purpose CPUs and GPUs typically use 32 or 64 bits for
floating point representation.
• Reduced Precision can be used without significantly compromising
performance.
• Motivations for Reduced Precision:
• Industrial Demand: need for faster hardware.
• Hardware Limitations: The progress of single CPU or GPU cores has slowed,
with improvements now largely driven by parallelization across cores.
• specialized hardware can now push performance boundaries.
• Current Trends:
• Low-Power Devices: New hardware designs are focused on low-power
devices like phones, aiming to support real-world deep learning applications
(e.g., speech recognition, computer vision, and natural language processing).
Low-Precision Implementations for Neural Networks
• Backpropagation & Low Precision: 8-16 bits of precision suffice for
training and using deep neural networks.
• Training vs. Inference: More precision is needed during training, less
so during inference.
• Dynamic Fixed-Point Representation:
• allocates a range across multiple numbers (e.g., all weights in one layer).
• This reduces bit requirements and improves efficiency in hardware.
• Benefits of Fixed-Point over Floating-Point:
• Reduces hardware surface area, power consumption, and computation time.
• Beneficial for the demanding multiplication operations in deep learning
models.
 Computer Vision
•Why Computer Vision?
•Easy for humans, challenging for computers.
•Key focus: Object Recognition and OCR.
•Applications
Human-like Abilities:
Recognize faces, objects, text.
New Abilities:
Detect sound from visual vibrations (Davis et al., 2014).
•Core Tasks
•Object detection: Identify or locate objects.
•Pixel labeling: Classify every pixel.
•Sequence transcription: Extract symbols.
•Generative Models-Image Synthesis:
•Create or restore images.
•Fix defects or remove objects.
Preprocessing
• Minimal Preprocessing
Standardize pixel values: [0, 1] or [-1, 1].
Avoid mixing ranges (e.g., [0, 1] vs. [0, 255]).
• Image Formatting
Resize or crop to standard dimensions.
Some models handle variable input sizes dynamically.
• Dataset Augmentation
For training: Add variations like crops, flips, etc.
For testing: Use ensemble-like voting from multiple versions.
• Canonical Preprocessing
Reduce irrelevant input variability.
Simplifies tasks, enabling smaller, generalizable models.
Example: Subtract pixel mean which involves computing the average
pixel value across the training set and subtracting it from each image
to center the data and improve model convergence.
Contrast Normalization
•Contrast refers to the magnitude of the difference between
the bright and the dark pixels in an image.
•In the context of deep learning, contrast refers to the
standard deviation of the pixels in an image or region of an
image.
•Contrast normalization is a preprocessing technique that
adjusts the contrast of an image by normalizing its pixel
intensity values, enhancing visual consistency and aiding
model performance.
Aspect Global Contrast Normalization (GCN) Local Contrast Normalization (LCN)
Scope Local regions or neighborhoods of
Entire image
the image
Normalization Method Centers pixel intensities and scales globally Centers and scales based on local
mean and variance
Purpose Normalize overall brightness and contrast Enhance fine-grained details and
reduce local variations
Effect Uniform contrast adjustment across the Localized contrast enhancement in
whole image different image parts
Dataset Augmentation
• Data Augmentation improves Classifier Generalization
• Key Idea
Add modified copies of training examples.
Use transformations that keep class unchanged.
• Why It Works in Object Recognition
Class Invariance: Robust to transformations.
Easy Transformations: Geometric operations like flips, rotations, and translations.
• Advanced Techniques
Color Perturbations: involve randomly adjusting image colors, such as brightness,
contrast, saturation, or hue, to enhance data diversity and robustness.
Nonlinear Geometric Distortions: apply complex transformations like warping or
bending to images, simulating realistic variations and improving model
generalization.
• Benefits
Expands dataset size.
Improves model robustness and performance.
Speech Recognition
• The task of speech recognition is to map an acoustic signal containing a spoken
natural language utterance into the corresponding sequence of words intended
by the speaker.
• Deep learning systems learn features from raw input.
• Let y = (y1 , y2 , . . . , yN ) denote the target output sequence (usually a sequence
of words or characters).
• The automatic speech recognition (ASR) task consists of creating a function f∗ASR
that computes the most probable linguistic sequence y given the acoustic
sequence X:

• where P∗ is the true conditional distribution relating the inputs X to the targets y.
Speech Recognition (1980s–2012)
• GMM-HMM Systems dominated ASR.
• GMMs: Model relationship between acoustic features and phonemes.
• HMMs: Model the sequence of phonemes.
• Process:
• HMM generates phoneme sequences.
• GMM transforms symbols into audio segments.
• Neural Networks in Early ASR
• Neural networks were used in ASR since the late 1980s.
• Performance matched GMM-HMM systems.
• Example: Robinson and Fallside (1991) achieved 26% phoneme error rate on
the TIMIT corpus.
 Deep Learning for Speech Recognition (Post-2009)
• Neural Networks Replace GMMs: Deep models and larger datasets
significantly improved accuracy by replacing GMMs for phoneme
association.
• Unsupervised Learning (RBMs):
• Restricted Boltzmann Machines (RBMs) used for unsupervised
pretraining of deep networks.
• Key Results
TIMIT Corpus: Phoneme error rate reduced from 26% to 20.7% .
Speaker-Adapted Features: Further reduced error rates.
Expanding to Large-Vocabulary Recognition:From phoneme recognition to
recognizing word sequences.
Evolution of Deep Networks
Shifted from RBMs to Rectified Linear Units (ReLU) and dropout.
Collaborations with Industry: Major companies adopted deep learning for
ASR, leading to breakthroughs in mobile phone products.
Advancements in Deep Learning for Speech Recognition
• Eliminating Unsupervised Pretraining: Large labeled datasets and improved
training methods made unsupervised pretraining unnecessary, leading to faster
progress.
• Performance Gains: Around 30% improvement in word error rates, surpassing the
decade-long stagnation of GMM-HMM systems.
• Industry Shift: Within two years, deep neural networks became integral to speech
recognition products.
• Innovations in Deep Learning
1. Convolutional Networks:
1. Weights replicated across time and frequency, improving on time-delay networks.
2. Spectrogram treated as a 2D image (time × frequency).
2. End-to-End Deep Learning
1. Deep learning models for ASR move away from HMMs.
2. Deep LSTM RNNs achieved 17.7% phoneme error rate on TIMIT.
3. Alignment Learning: Systems learn to align acoustic and phonetic information.
• Current Focus: Ongoing research into improving architectures and algorithms.
Natural Language Processing
• NLP enables computers to process human languages (e.g., English,
French).
• Applications: Machine translation, language modeling.
• Challenges
Human language is ambiguous and complex.
Language models predict word, character, or byte sequences.
• Neural Network Techniques
Generic Models: Works well for many NLP tasks.
Domain-Specific Strategies: Needed for large-scale applications.
• Sequential Data
NLP often treats language as sequences of words, not characters.
Word-based models deal with large, sparse spaces.
• Efficiency Strategies
Focus on computational and statistical efficiency.
Use specialized techniques for high-dimensional spaces.
Topics
1. n-grams
2. Neural Language Models
3. High-Dimensional Outputs
a. Use of a Short List
b. Hierarchical Softmax
c. Importance Sampling
d. Noise-Contrastive Estimation and Ranking Loss
4. Combining Neural Language Models with n-grams
5. Neural Machine Translation
Using an Attention Mechanism and Aligning Pieces of Data
n-grams
• A language model defines a probability distribution over sequences of
tokens in a natural language.
• Depending on how the model is designed, a token may be a word, a
character, or even a byte.
• Tokens are always discrete entities.
• The earliest successful language models were based on models of
fixed-length sequences of tokens called n-grams.
• An n-gram is a sequence of n tokens.
• Training n-gram models:
• The maximum likelihood estimate can be computed by counting how many times
each possible n-gram occurs in the training set.
• Unigram (1-gram):
• A single word or item.
• Example: "The cat sat" → Unigrams: "The", "cat", "sat“
• Bigram (2-gram):
• A pair of consecutive words.
• Example: "The cat sat" → Bigrams: "The cat", "cat sat“
• Trigram (3-gram):
• A triplet of consecutive words.
• Example: "The cat sat" → Trigrams: "The cat sat“
• Higher-order N-grams:
• Sequences with more than three items, like 4-grams, 5-grams, etc.
• Applications:
Language Modeling: Predicting the next word or evaluating sentence
likelihood.
Text Generation: Creating sentences by selecting the most likely next
word based on previous words.
Speech Recognition: Matching audio sequences to probable word
sequences.
• Limitations:
Sparsity: For large vocabularies, higher-order n-grams require
massive amounts of data.
Context Limitation: Limited by the Markov assumption, it ignores
long-range dependencies.
 • Combining N-grams with Deep Learning:
1. Text Classification with CNNs:
Convert text into n-grams (e.g., bigrams, trigrams).
Use these n-grams as features in a CNN.
The model learns to identify important n-gram patterns (e.g., sentiment phrases)
for classification.
2. RNN-based Language Model:
Use n-grams to create a fixed-size context window for input.
Feed the n-grams into an RNN to predict the next word in the sequence.
• While deep learning models such as RNNs, CNNs, and Transformers have
significantly advanced NLP tasks, n-grams still play an important role in
certain hybrid or preprocessing techniques.
• By leveraging both n-gram features and deep learning models, systems can
achieve better performance in tasks like text classification, language
modeling, and machine translation.
Neural Language Models
• Goal: Overcome the curse of dimensionality in modelling natural language by
using distributed word representations.
• Key Advantage:
Unlike n-gram models, NLMs recognize word similarities without losing
distinctness.
Share statistical strength between similar words and contexts.
• Word Representations
Words like "dog" and "cat" may share many attributes in their representations.
This allows sentences containing "dog" and "cat" to inform each other's
predictions.
• Generalization
• NLMs can generalize to a vast number of semantically related sentences,
countering the curse of dimensionality.
• Word Embeddings: These learned representations enable this sharing and
generalization.
• Word Embeddings :
• Word representations in a lower-dimensional space (compared to the
original one-hot vector space).
• Original Space: Each word is a one-hot vector, with all words equidistant
from each other.
• Embedding Space: Words with similar contexts or features are closer
together.
• Result: Words with similar meanings tend to be neighbors in the
embedding space.
• Example: Semantic similarity is reflected in the proximity of word
representations.
• Embeddings in Neural Networks
• In convolutional networks, a hidden layer provides an "image
embedding.“
• NLP Focus: More significant in NLP as language doesn't naturally exist
in a real-valued vector space. Hidden layers dramatically change data
representation.
• Beyond Neural Networks
• Graphical Models: Distributed representations can also be used in
graphical models with multiple latent variables to improve NLP
models.
High-Dimensional Outputs
• In deep learning, high-dimensional outputs refer to tasks where the
model needs to make predictions over a large set of possible outputs,
such as in language modeling, machine translation, or image
captioning.
• Dealing with these large output spaces efficiently requires techniques
that reduce computational costs while maintaining or improving
performance.
• Below are some common methods used to handle high-dimensional outputs:
a. Use of a Short List
b. Hierarchical Softmax
c. Importance Sampling
d. Noise-Contrastive Estimation and Ranking Loss
Use of a Short List (Top-k Approximation)
• Instead of computing the full probability distribution over all possible
outputs (which can be computationally expensive), a short list of the most
likely candidates (top-k) is considered.
• How It Works:
After calculating the logits for all possible outputs, the model only looks at
the top-k outputs based on these scores.
This reduces the number of computations required for the softmax
operation, which would involve a summation over all classes.
• Application:
• used in language models or image captioning
• where predicting the top-k most probable words or phrases can yield useful results
without needing to consider every possible candidate.
Hierarchical Softmax
• Instead of calculating the full softmax over all classes, hierarchical softmax
reduces the complexity by organizing the classes into a binary tree
structure.
• How It Works:
Each output class is assigned to a leaf in the tree, and the model computes
a binary decision at each internal node of the tree, ultimately leading to a
leaf.
The softmax is then computed along this hierarchical path instead of over
all classes. This reduces the computational complexity from O(C) (where C
is the number of classes) to O(log C).
• Application:
Hierarchical softmax is often used in tasks such as word embeddings (e.g.,
Word2Vec) and
language modeling, where the output space (vocabulary size) is large.
Importance Sampling
• Importance sampling is a technique used to approximate high-dimensional
distributions by sampling from a proposal distribution that is easier to
sample from, rather than directly sampling from the true distribution.
• How It Works:
A model can use a simpler distribution (like a uniform distribution or a prior
distribution) to sample a subset of potential outputs and then reweight
those samples based on how likely they are under the true distribution.
This way, the computation for the softmax or similar loss functions is
reduced by focusing on more likely outputs.
• Application:
used in variational inference and reinforcement learning where the action
space or output space is very large, but the model can focus on more
important or likely actions.
Noise-Contrastive Estimation (NCE)
• Noise-Contrastive Estimation (NCE) is a method used to approximate the
softmax function in a high-dimensional output space by framing it as a
binary classification problem.
• How It Works:
NCE approximates the true distribution by training the model to distinguish
between real data and noise samples.
Instead of computing the full softmax distribution over all possible outputs,
the model is trained to classify whether a given output is from the true
distribution or a noise distribution.
This reduces the complexity.
• Application: used in language models and word embeddings (e.g., in
Word2Vec), where training on a large vocabulary requires efficient
approximations.
Ranking Loss
• Ranking loss is a loss function designed to optimize the relative ordering of
outputs rather than predicting exact probabilities for each output.
• This is useful in tasks where the model needs to rank items rather than
predict specific labels.
• How It Works:
The model is trained to output relative scores for items, and the loss
function penalizes the model if the order of predictions is incorrect.
Instead of predicting the exact probability for each class, the model only
cares about whether a higher-ranked item is assigned a higher score than a
lower-ranked one.
• Application: used in information retrieval, recommender systems, and
machine translation.
• For example, in document ranking, the model is trained to rank relevant
documents higher than irrelevant ones, rather than predicting the exact
probability of each document.
Summary of the methods
1. Use of a Short List: Focuses on the top-k most likely outputs, reducing
computational cost.
2. Hierarchical Softmax: Reduces complexity by structuring output classes
in a binary tree, leading to faster calculations.
3. Importance Sampling: Reduces high-dimensional computations by
sampling from a simpler distribution and adjusting for importance.
4. Noise-Contrastive Estimation (NCE): Approximates softmax by
distinguishing real data from noise, significantly reducing computational
cost.
5. Ranking Loss: Optimizes the relative ordering of outputs, useful in
ranking tasks where exact probabilities are less important.
Combining Neural Language Models with n-grams
• Advantages of N-gram Models
High Model Capacity: Store many frequency counts of tuples.
Low Computation: Process examples with minimal computation (lookup
tuples).
Efficiency: Using hash tables or trees keeps computation almost independent
of capacity.
• Neural Networks vs. N-grams
Neural Networks: Doubling parameters typically doubles computation time.
Exceptions: Embedding layers and tiled convolutional networks can increase
capacity without doubling computation.
Matrix Multiplication Layers: Computation grows with the number of
parameters.
Combining N-gram and Neural Models
• Ensemble Approach: Combine a neural language model with an n-gram
model to increase capacity.
• Ensemble Learning: Reduces test error if models make independent
mistakes, with methods like uniform weighting or validation-set-based
weights.
• Extending the Ensemble
• Multiple Models: Mikolov et al. (2011a) expanded the ensemble to include
a large array of models.
• Neural Network + Maximum Entropy: Mikolov et al. (2011b) trained both
models jointly.
• Extra inputs represent n-grams and are sparse, increasing model capacity with
minimal additional computation.
Neural Machine Translation
• Machine Translation
• Task: Translate a sentence from one language to another while preserving
meaning.
• Components:
Proposal Component: Suggests multiple translations, including
ungrammatical ones like "apple red" instead of "red apple.“
Language Model: Evaluates and scores the proposed translations (e.g., "red
apple" is better than "apple red").
• Neural Networks in Machine Translation
Early Use: Neural networks were first used to upgrade the language model.
N-gram Models: Earlier models used n-grams and maximum entropy
language models to predict the next word.
• Encoder-Decoder Architecture
• Purpose: Maps between a surface
representation (e.g., words or
images) and a semantic
representation. • Applications
• How It Works: Machine Translation: Translates
between languages using
Encoder: Converts input (e.g., French
sentences) into hidden encoder-decoder.
representations. Image Captioning: Generates
Decoder: Converts hidden captions for images using similar
representations into output in architecture.
another modality (e.g., English
sentences).
RNNs for Flexible Translation
• MLP Limitation: MLP requires fixed-length input sequences.
• RNN Advantage: RNNs handle variable-length inputs and outputs, making
translation more flexible.
• Encoder-Decoder Framework
1. Encoder: Reads the input sequence and produces a summary (context C).
• Can be an RNN or convolutional network.
2. Decoder: Uses the context C to generate the target language sentence.
• Typically an RNN.
• Representation Learning
Goal: Learn representations where sentences with the same meaning have
similar representations, regardless of source or target language.
Early models: Used a combination of convolutions and RNNs.
Later models: Improved with RNNs for translation generation.
 Using an Attention Mechanism and Aligning Pieces of Data
• An attention-based system has three components:
• 1. A process that “ reads” raw data (such as source words in a source sentence), and
converts them into distributed representations, with one feature vector associated with
each word position.
• 2. A list of feature vectors storing the output of the reader. This can be understood as a
“memory” containing a sequence of facts, which can be retrieved later, not necessarily in
the same order, without having to visit all of them.
• 3. A process that “exploits” the content of the memory to sequentially perform a task, at
each time step having the ability put attention on the content of one memory element
(or a few, with a different weight).
• The third component generates the translated sentence.
• When words in a sentence written in one language are aligned with corresponding words
in a translated sentence in another language, it becomes possible to relate the
corresponding word embeddings.
Other Applications
1. Recommender Systems
•Exploration Versus Exploitation
2. Knowledge Representation, Reasoning and Question
Answering
3. Knowledge, Relations and Question Answering
Recommender Systems
• Main Application Areas
Online Advertising
Item Recommendations (e.g., product sales, services)
• Goal
Predict user-item associations.
Estimate probability of user actions or expected gain from recommendations.
• Economic Impact
Online Advertising: Major internet revenue source.
E-commerce: Key for companies like Amazon and eBay.
• Other Recommendation Examples
Social Media: News feed posts.
Entertainment: Movie, music recommendations.
Other Services: Jokes, expert advice, game matchmaking, dating.
• The association problem is handled like a supervised learning problem:
• given some information about the item and about the user, predict the
proxy of interest
• user clicks on ad, user enters a rating, user clicks on a “like” button, user
buys product, user spends some amount of money on the product, user
spends time visiting a page for the product, etc.
• This ends up being either
• a regression problem (predicting some conditional expected value) or
• a probabilistic classification problem (predicting the conditional probability
of some discrete event).
Early Recommender Systems: Collaborative
Filtering
• Minimal Inputs
User ID and Item ID used to predict preferences.
• Generalization via Similarity
Users with similar preferences are grouped.
Example: If two users like items A, B, and C, they are likely to share similar tastes.
• Collaborative Filtering
Non-Parametric: Nearest-neighbor methods based on preference similarity.
Parametric: Learning distributed representations (embeddings) for users and items.
• Bilinear Prediction
Uses the dot product of user and item embeddings to predict ratings.
Often includes constants for user or item bias.
 Let ˆR be the matrix containing predictions,
A a matrix with user embeddings in its rows and
B a matrix with item embeddings in its columns.
Let b and c be vectors that contain respectively a kind of bias
for each user (representing how grumpy or positive that user is
in general) and for each item (representing its general
popularity).
• The bilinear prediction is thus obtained as follows:
• Singular Value Decomposition (SVD) decomposes a user-item matrix
into user and item embeddings, capturing latent factors that explain
user-item interactions.
• The user embeddings represent users in a lower-dimensional space,
while the item embeddings represent items in a similar way.
• The dot product of these embeddings provides predicted ratings or
interactions, revealing hidden patterns in the data.
Collaborative Filtering and Cold-Start
Problem
• Neural Networks in Collaborative Filtering
RBM used for collaborative filtering.
Key component in the Netflix competition.
• Cold-Start Problem
New users/items lack rating history, making it hard to evaluate similarities.
Requires additional information to address the issue.
• Content-Based Recommender Systems
Uses extra user/item features (e.g., profiles, item characteristics).
Deep learning maps features to embeddings.
• Deep Learning for Rich Content
Convolutional Networks extract features from rich content like music.
Embeddings for items (e.g., songs) are used for recommendation prediction.
Exploration Versus Exploitation
Recommendations and Reinforcement Learning
• Contextual Bandits in Recommendation
Recommendation problems are often modeled as contextual bandits.
Bias arises because we only see responses to recommended items, not to
others.
• Bias in Data Collection
No information about users who don't receive a recommendation.
We don't know the outcome of recommending other items.
• Comparison to Supervised Learning
Similar to training a classifier where only feedback on the predicted class is
received.
More data is needed to learn correct decisions.
• Reinforcement Learning in Bandits
Reinforcement learning involves a sequence of actions and rewards.
In bandit problems, only one action is taken, and the associated
reward is observed.
• Contextual Bandits
Decisions are made based on context (e.g., user identity).
The mapping from context to action is called a policy.
Exploration vs. Exploitation in Reinforcement Learning
• Exploitation
• Take actions based on the current best policy to maximize rewards.
• Exploration
• Take actions to gather more training data, even at the cost of uncertain
rewards.
• Tradeoff
• Exploitation: Maximize known rewards (e.g., action 𝑎 gives reward 1).
• Exploration: Try new actions (e.g., action 𝑎ʹ) to potentially gain better
rewards, despite risk.
• Exploration Strategies
• Random actions to cover the action space.
• Model-based approaches to choose actions based on expected rewards and
uncertainty.
Evaluation in Reinforcement Learning
• Time Scale Influence
Short time: Prefer exploitation.
Long time: Start with exploration for better future planning, then shift to
exploitation.
• Supervised Learning
No exploration-exploitation tradeoff.
Supervision always provides the correct output label for each input.
• Policy Evaluation Challenge
Reinforcement learning involves a feedback loop between the learner and
environment.
Evaluating policies is hard because the policy determines the inputs seen.
 Knowledge Representation, Reasoning and
Question Answering
• Distributed representations can be trained to capture the relations between two entities.
• In mathematics, a binary relation is a set of ordered pairs of objects.
• Pairs that are in the set are said to have the relation while those who are not in the set do
not. S = {(1, 2), (1, 3), (2, 3)}; (1, 2) ∈ S
• In the context of AI, we think of a relation as a sentence in a syntactically simple and
highly structured language.
• The relation plays the role of a verb, while two arguments to the relation play the role of
its subject and object. These sentences take the form of a triplet of tokens.
• (subject, verb, object) with values
• (entityi, relationj, entityk).
• For example, we could define the has_fur attribute, and apply it to entities like dog.
Many applications require representing relations and reasoning about them.
How should we best do this within the context of neural networks?
• Training Data for ML Models
• Models need training data to infer entity relations.
• Relations can be extracted from:
• Unstructured data: Natural language text.
• Structured data: Relational databases explicitly define relations.
• Knowledge Bases (KBs)
• KBs store common-sense or expert knowledge for AI systems.
• Examples: Freebase, WordNet, Wikibase, GeneOntology.
• Learning Representations
• Entities and relations in KBs are represented as triplets.
• Training involves maximizing objectives based on triplet distributions.
Define a model family to train
• Models extend neural language models to include entities and relations.
• Neural models learn:
• Word embeddings: Distributed vector representations.
• Word interactions: Predict next words in a sequence.
• Extending to Entities and Relations
• Learn embedding vectors for entities and relations.
• Use data from:
Knowledge bases.
Natural language sentences.
Multiple relational databases.
• Evaluating Link Prediction Models
• Challenge: Only positive examples (true facts) are available.
• If a proposed fact is missing from the dataset, it could be an error or a new
discovery.
• Evaluation Metrics
• Test how the model ranks held-out true facts vs. unlikely facts.
• Generate negative examples by corrupting true facts (e.g., replace one
entity randomly).
• Precision@10%
• Measures how often true facts rank in the top 10% of all corrupted
versions.
 • Applications of Knowledge Bases
1. Word-Sense Disambiguation
1. Identifies the correct meaning of a word in context.
2. Question Answering Systems
1. Combine relations, reasoning, and natural language understanding.
2. Goal: Process input, store facts, and enable retrieval and reasoning.
• Explicit Memory Mechanisms
• Best for storing and retrieving declarative facts.
• Memory Networks:
• Proposed for toy QA tasks (Weston et al., 2014).
• Kumar et al. (2015) used GRUs to read input into memory and generate answers.
---XXX---