Representation Learning

Representation Learning is a process that simplifies raw data into understandable patterns for machine
learning. It enhances interpretability, uncovers hidden features, and aids in transfer learning.

Representation learning in deep learning is the automated process of discovering optimal data features,
or embeddings, directly from raw input using multi-layered neural networks

Representation learning refers to a set of techniques that allow a system to discover different
representations of the data. If done correctly, the new representation of the data can be more
meaningful than the original raw representation and it may allow an ML algorithm to learn patterns not
found in the raw data.
Why Representations Matter

● Data representation affects task difficulty (daily life, programming, ML)

● Example: 210 ÷ 6 easy in Arabic numerals, hard in Roman (CCX ÷ VI)
● Algorithms: O(n) insert in lists vs. O(log n) in red-black trees
● Goal: Simplify learning, e.g., linear separability in hidden layers
What is a Good Representation?

A good representation has three characteristics: Information, compactness, and generalization.

● Information: The representation encodes important features of the data into a compressed form.
● Compactness:
○ Low Dimensionality: Learned embedding representations from raw data should be much
smaller than the original input. This allows for efficient storage and retrieval, and also discards
noise from the data, allowing the model to focus on relevant features and converge faster.
○ Preserves Essential Information: Despite being lower-dimensional, the representation retains
important features. This balance between dimensionality reduction and information preservation
is essential.
● Generalization (Transfer Learning): The aim is to learn versatile representations for transfer
learning, starting with a pre-trained model.
Sharing Across Tasks
● Transfer knowledge: Unsupervised aids supervised on same data
● Multi-modal: Text + images; low-data tasks; multiple domains
● Share internal layers across supervised/unsupervised tasks
● Balances preserving info vs. nice properties (e.g., independence)

Unsupervised and Semi-Supervised Benefits

Learning balances preserving input info vs. gaining "nice" properties (e.g., independence)—a core
trade-off.
It's ideal for semi-supervised learning: tons of unlabeled data, little labeled. Supervised-only on labels
overfits badly.
Solution: Learn good representations from unlabeled data, then apply to supervised task—like
humans/animals thriving on few examples (hypothesis: via unsupervised learning).wikipedia+1
Greedy LayerWise Unsupervised Pre Training Algorithm

An unsupervised pre-training algorithm is a machine learning technique that trains models on vast
amounts of unlabeled data to learn meaningful data representations, patterns, and structures before
fine-tuning on smaller labeled datasets.

Why "Greedy," "Layer-Wise," and "Unsupervised"?

● Greedy: Like a greedy algorithm in optimization—handles each piece independently for quick progress,
without perfect global planning.
● Layer-wise: Focuses sequentially on layers, introducing upper ones only after lower ones are set.
● Unsupervised: Each step uses no labels, just learns latent features.
● Pretraining: It's phase 1 only—followed by supervised fine-tuning of the whole stack.

In supervised contexts, it serves as:

● Parameter initialization (better starting points avoid poor local minima).

● Regularizer (often cuts test error without raising training error).
Core Mechanism: Greedy Layer-Wise Pretraining

The process uses a single-layer unsupervised algorithm

● Start with raw input data.

● Train the first layer unsupervised to output a new, hopefully simpler representation.
● Feed that output as input to train the second layer, and so on—greedily, one layer at a time.
● Lower layers stay fixed once trained; no joint optimization yet.

Write Greedy layer wise unsupervised pretraining protocol algorithm.(6M)

Algorithm Greedy layer-wise unsupervised pretraining protocol .Given the following: Unsupervised feature
learning algorithm L, which takes a training set of examples and returns an encoder or feature function f. The
raw input data is X, with one row per example,and f (1)(X) is the output of the first stage encoder on X. In the
case where fine-tuning is performed, we use a learner T , which takes an initial function f, input examples X
(and in the supervised fine-tuning case, associated targets Y), and returns a tuned [Link] number of stages
is m.

f ← Identity function
˜X = X
for k = 1, . . . , m do
f(k)= L(˜X)
f ← f(k) ◦ f
˜X ← f(k)(˜X)
end for
if fine-tuning then
f ← T (f, X, Y )
end if
Return f
Standard Two-Phase Protocol

Most applications follow this:

1. Pretraining phase: Unsupervised layer-by-layer.

2. Supervised phase: Either:
○ Train a simple classifier on top of frozen pretraining features
○ Fine-tune the entire network end-to-end.

● Phase 1: Unsupervised pretraining layer-by-layer

● Phase 2: Supervised – classifier on top (frozen) or full fine-tune

Benefits: Better init (avoids poor minima), regularizer (cuts test error)
Applications: Deep autoencoders, belief networks
key success conditions

1. Good separation of objectives

● Phase I should focus on representation quality , not direct task performance.
● Phase II should focus on task‑specific adaptation

2. High‑quality Phase‑I representations

● The Phase‑I representation must be sufficiently expressive and semantically meaningful for the target
task.

3. Proper phase‑transition scheduling

● too early → representations are under‑trained;

● too late → representations may overfit the Phase‑I objective

[Link] on downstream tasks

● The protocol is only successful if the two‑phase representation helps downstream performance
Key Success Conditions

● Poor initial representations: Most effective when raw inputs lack structure, e.g., one-hot word vectors
(distance 2 apart, ignoring similarity). Learned embeddings encode semantic proximity, making NLP a
stronghold. Less needed for images, where pixel vectors already offer crude similarity.
● Data imbalance: Thrives with few labeled examples (handfuls per class) and vast unlabeled data—won 2011
transfer contests (Mesnil et al., 2011; Goodfellow et al., 2011). Paine et al. (2014) confirmed via controlled
tests.
● Complex functions: Unlike weight decay (biasing toward simplicity), it guides toward input-regularity features,
ideal for intricate mappings shaped by data distribution.
Two Interacting Mechanisms

1. Regularizing Initialization: Places parameters in a "good" region—small, consistent function space unreachable
otherwise Training converges reliably here: lower gradient noise from minibatches, better-conditioned Hessians.
Reduces solution variance, curbing severe overfitting. Modern view: Not about local minima (rarely hit), but
escaping flat/noisy zones.
2. Shared Useful Features: Unsupervised captures input structure (e.g., wheel counts in car images) transferable
to supervised goals—if aligned (e.g., linear separability for linear classifiers). Mathematical details elusive;
task/model-specific.
Practical Drawbacks and Alternatives

Two-phase setup (pretrain → fine-tune) lacks flexibility:

● No "regularization dial"—full strength or none.

● Many hyperparameters per phase; supervised feedback delayed, hard to tune (Larochelle et al., 2009 suggests
validation-based).
● Freezing extractor + top classifier avoids fine-tune mysteries.

Better options:

● Single-stage: Joint supervised+unsupervised objectives (e.g., discriminative RBMs; Larochelle & Bengio, 2008;
ladder nets; Rasmus et al., 2015). Single coefficient tunes unsupervised influence predictably.
● Other paradigms: Virtual adversarial training concurrent autoencoder+supervised.
Transfer learning and Domain Adaptation
The learner must perform two or more different tasks, we assume that many of the factors that explain the variations
in P1 are relevant to the variations that need to be captured for learning P2.

Transfer learning and domain adaptation refer to the situation where what has been learned in one setting (e.g.,
distribution P1) is exploited to improve generalization in another setting (say, distribution P2).
Two extreme form of transfer learning:

(1) One-shot learning

One-shot learning is a machine learning paradigm where a model learns to recognize, classify, or generate new concepts, objects, or patterns
after exposure to just a single labeled example per class or [Link] representation learns to cleanly separate the underlying classes during
the first stage. During the transfer learning stage, only one labeled example is needed to infer the label of many possible test examples that all
cluster around the same point in representation space.

(2) Zero-shot learning

Zero - Shot Learning (ZSL) is a branch of machine learning that allows models to recognize and classify instances from classes they haven't
encountered during the training phase. Instead of relying on labeled examples of every category, Zero-Shot Learning makes use of semantic
attributes or relationships between seen and unseen classes.

Three random variable

● traditional input x
● traditional output y
● additional random variable describing the task T - p(y | x, T)
Key Requirements for Zero-Shot Learning
1. Semantic Representations
ZSL relies heavily on semantic representations that describe the relationships between classes. These could be:

● Attributes: For example, animals might be described by their attributes (e.g., "has wings," "is a mammal").
● Word Embeddings: embeddings can capture semantic relationships between classes in a continuous vector space.

2. Shared Feature Space

Both seen and unseen classes are mapped into a shared feature or semantic space, where the similarities between
classes can be measured. This allows the model to leverage known features to infer unseen ones.

3. Generalization Mechanism
The ability of the model to transfer knowledge from seen classes to unseen classes is essential. This transfer happens
through shared attributes, relations, or embeddings that link the two groups together.
The workflow of ZSL typically involves two steps:

● Training: The model is trained on seen classes and their corresponding semantic representations.
● Prediction: During testing, the model uses semantic information to classify unseen classes based on the
relationship it learned from seen classes.
Challenges in Zero-Shot Learning

1. Bias Toward Seen Classes: ZSL models can sometimes be biased toward seen classes since the majority of the training
data is derived from them. This can lead to poor performance on unseen classes.
2. Domain Shift: The feature space of unseen classes might not align perfectly with that of seen classes, causing a
domain shift. This results in inaccurate mappings and incorrect predictions.
3. Semantic Representation Quality: The quality of the semantic representation (e.g., word embeddings or attributes)
greatly influences the performance of ZSL models. Poor representations can mislead the model, resulting in low
accuracy.
4. Scalability: Scaling Zero-Shot Learning to handle hundreds or thousands of unseen classes can be difficult, especially
when the semantic space is not large enough to differentiate between many categories.
Multimodal Machine Learning refers to the use of multiple data types such as text, images, audio and video or modalities to
build models that can process and integrate them into a unified understanding. The main objective is to enhance decision-making
by integrating various data sources, each offering unique insights that contribute to improving the overall performance of AI
systems.

● Learn fx:x→hx, fy:y→hy

● Model joint distribution over paired (hx,hy)

Core Concepts of Multimodal Machine Learning

1. Modalities: Modalities are the various types of data that can be processed by machine learning models. These can include:

● Text: Written language such as articles, posts or spoken words transcribed to text.
● Images: Visual data captured through cameras which can be used in tasks such as object detection or facial recognition.
● Audio: Sound data like speech or environmental sounds which can be used for speech recognition or emotion
detection.
● Video: A combination of images and audio that can be used for tasks like video classification or action recognition.
● Sensor Data: Information from IoT devices or sensors, such as temperature readings or motion sensors.
Semi-Supervised Disentangling of Causal Factors
● Question: What makes one representation better than another
● One hypothesis: the one in which the features within the representation corresponds to the underlying
causes of the observed data, with separate features or directions in the feature space corresponding to
different causes, so that the representation disentangles the causes from one another.

This hypothesis motivates approaches in which we first seek a good representation for p(x).
Situation 1: unsupervised learning fails to help supervised
learning
● P(x): uniformly distributed
● Goal: Learn E(y | x)

Situation 2: unsupervised learning succeeds to help supervised

learning
Shared semantics of the output
If y is closely associated with one of the causal factors of x, then p(x) and p(y|x) will be strongly tied and unsupervised
representation learning tries to disentangle the underlying factors of variation is likely to be useful as a semi-supervised
learning strategy.

● Problem: Most of observations are formed by an extremely large number of underlying causes.
● Brute Force Solution: An unsupervised learner learns a representation that captures all the reasonably salient
generative factors and disentangles them from each other, thus making it easy to predict y from h, regardless of
which hi
● is associate with y. Not feasible. It is not possible to capture all or most of the factors of variation that influence an
observation.
● Two Main Strategies used
○ Use a supervised learning signal at the same time as the unsupervised learning signal so that the
model will choose to capture most relevant factors of variation.
○ Use much larger representation if using purely unsupervised learning.
An emerging strategy for unsupervised learning is to modify the definition of which underlying causes are
most salient. Historically, autoencoder and generative models have been trained to optimize a fixed criterion.
These fixed criteria determine which causes are considered salient. See figure below how MSE failed to
learn to encode a small ping pong ball:
Other definitions of salient are possible. e.g. Recognizable Pattern. One way to implement such a definition of
salience is to use Generatibe Adverserial Network:

● A feedforward classifier: attempts to recognize all samples from the generative model as being fake
and samples from training set as real
● Generative model: provide “fake samples”

The network will learn what is salient.

If the true generative process has x as an effect and y as a cause, then modeling p(x|y) is robust to changes in
p(y). If the cause-effect relationship is reserved, that would not be true. Very often, when we consider changes in
distribution due to different domains, temporal nonstationary or changes the nature of task, the causal
mechanisms remain invarianct. So, better generalization and robustness to all kinds of change can be expected
via learning a generative model that attempts to recover the causal factors h and p(x|h).
Transfer Learning and Domain Adaptation

Transfer learning and domain adaptation refer to the situation where what has been learned in one setting (e.g., distribution P1)
is exploited to improve generalization in another setting (say, distribution P2).

Zero-data learning/zero-shot learning

Zero - Shot Learning (ZSL) is a branch of machine learning that allows models to recognize and classify instances from classes
they haven't encountered during the training phase. Instead of relying on labeled examples of every category, Zero-Shot
Learning makes use of semantic attributes or relationships between seen and unseen classes.

The goal: estimate

p(y∣x,T) where T describes the task, with y=1 for yes and 0 for no.
Transfer Learning Framework

Transfer learning is a machine learning technique where a model trained on one task is
repurposed as the foundation for a second task. This approach is beneficial when the second
task is related to the first or when data for the second task is limited.
● Encoder Functions: Learn fx(x)→hx from unlabeled x Similarly, fy(y)→hy for y-space.
● Similarity Metrics: Distances in hx or hy outperform raw y space distances, as embeddings cluster meaningfully
● Mapping Between Spaces: Labeled (x,y) pairs train one- or two-way maps between hx and hy
Fusion of Modalities: One of the key challenges in MML is how to combine data from different
modalities. This process is known as fusion and it can be done in various ways:
● Early Fusion: It combines raw data from all modalities at the beginning of the model pipeline,
making it useful for tasks where modalities are closely related like combining audio and video
for emotion detection. For example mixing audio, video and text features at the start before
feeding them into the model.
● Late Fusion: In this processes each modality is independent and merges their results at the end
which is ideal for situations where each modality provides independent insights, such as in
self-driving cars where camera and radar data are processed separately.
● Hybrid Fusion: It combines both early and late fusion, capturing both low-level and high-level
features. It is often used when different data types require varying levels of integration like in
complex healthcare models combining text and image data for diagnosis.
Practical Implications and Examples
Example 1: Cat Recognition
Text corpus: "Cats: four legs, pointy ears, furry."
Test: Unseen cat image

Xtest Process: Embed image to hx, task T="cat?" to hT (word embedding). Map compares hx hT; proximity predicts y=1

Challenges and Tips:

● Embeddings must be low-dimensional, semantically rich (e.g., skip-gram models).
● Joint learning > separate: Aligns spaces end-to-end.
● Scales to vision-language models like CLIP today.
When Semi-Supervised Fails or Succeeds
Semi-supervised fails if p(x) is uniform, providing no info on p(y|x), as in f(x)=E[y|x] with no structure. It
succeeds in mixture models where components per y are separated; modeling p(x) reveals clusters, and one
label per class suffices for perfect p(y|x)
3.A Deep neural network is trained using greedy layer wise pretraining, where each layer is trained as an autoencoder using
sigmoid activation. The architecture consists of: Input Layer: 1 neuron, Hidden Layer 1: 1 neuron, Hidden Layer 2: 1
neuron, Reconstruction Output: 1 neuron, activation function is sigmoid and assume bias as 0. (10 M)

Given:
• Input: x=0.7
Weights:
• w1=1.5 (Input → Hidden Layer 1)
• w2=1.2 (Hidden Layer 1 → Hidden Layer 2)
• w2′=0.9 (Hidden Layer 2 → Reconstructed Hidden Layer 1)
• w1′=1.0 (Reconstructed Hidden Layer 1 → Reconstructed Input)

A. Compute the activation values for:

i. Hidden Layer 1
ii. Hidden Layer 2

B. Perform the reconstruction by computing:

i. Reconstructed Hidden Layer 1 from Hidden Layer 2
ii. Reconstructed Input from Reconstructed Hidden Layer 1
C. Calculate the Mean Squared Error (MSE) between the original input x=0.7 and the final reconstructed input x^.
i. Hidden Layer 1 (h₁):
z₁ = w₁ × x + b₁ = 1.5 × 0.7 + 0 = 1.05
h₁ = σ(z₁) = 1/(1 + e⁻¹⋅⁰⁵) = 0.7408

Hidden Layer 2 (h₂):

z₂ = w₂ × h₁ + b₂ = 1.2 × 0.7408 + 0 = 0.8889
h₂ = σ(z₂) = 1/(1 + e⁻⁰⋅⁸⁸⁸⁹) = 0.7087

Reconstruction Formulae (Part B)

i. Reconstructed Hidden Layer 1 (ĥ₁):
z₁' = w₂' × h₂ + b₂' = 0.9 × 0.7087 + 0 = 0.6378
ĥ₁ = σ(z₁') = 1/(1 + e⁻⁰⋅⁶³⁷⁸) = 0.6543

ii. Reconstructed Input (x̂):

zₓ' = w₁' × ĥ₁ + b₁' = 1.0 × 0.6543 + 0 = 0.6543
x̂ = σ(zₓ') = 1/(1 + e⁻⁰⋅⁶⁵⁴³) = 0.6580

MSE Formula (Part C)

Mean Squared Error:

MSE = (x - x̂)² = (0.7 - 0.6580)² = 0.042⁰² = 0.00177

 A deep neural network uses greedy layer-wise pretraining where each layer
trains as an autoencoder with sigmoid activation. Architecture: Input Layer (1
neuron) → Hidden Layer 1 (1 neuron) → Hidden Layer 2 (1 neuron) →
Reconstruction Output (1 neuron). All activations use sigmoid, bias ≠ 0.

Given:
● Input: x = 0.7
Weights: w1=1.5, w2=1.2, w2′=0.9, w1′=1.0
Biases: b1=0.3, b2=-0.2, b2′=0.1, b1′=-0.1
4.A Machine Learning engineer is working on sentiment classification for product reviews in a target domain (e.g.,
electronics). However, only 500 labelled reviews are available in the target domain. To improve performance, the
engineer decides to use Instance-Based Transfer Learning by borrowing data from a source domain (e.g., book
reviews), which contains 10,000 labelled reviews.

To address domain differences, instance re-weighting is applied:

• The classifier is trained on all 10,000 source domain reviews and 500 target domain reviews.
• Each source instance is given a weight of 0.3, and each target instance a weight of 1.0 during training.
• The accuracy of the model trained without weighting is 68%.
• The accuracy of the model trained with instance weighting improves to 75%.

For the given scenario, answer the following:

i. What is the purpose of instance re-weighting in this context?

ii. Compute the effective weighted sample size used in training.
iii. Calculate the relative improvement in accuracy due to re weighting.
iv. Name one challenge that can arise in instance-based transfer learning and explain briefly.
 1. Instance re-weighting adjusts the influence of source domain instances during training to better
align with the target domain, reducing negative effects from domain [Link] re-weighting in
instance-based transfer learning aims to mitigate domain differences by down-weighting source
instances that may not generalize well to the target domain, like electronics reviews differing from
book reviews. This makes the weighted source distribution closer to the target, improving adaptation
without discarding data.

2. Compute the effective weighted sample size used in training.

10000×0.3+500×1.0=3000+500=3500
1000x(0.3)2+500X(1.0)2 =

neff=(∑wi)2/∑wi2= (3500)2/

3. new accuracy−old accuracy/old accuracy×100%