MODULE 1 : SELF SUPERVISED

DEFINITION:
• Self-supervised learning is a machine learning process where the model trains itself to
learn one part of the input from another part of the input. It is also known as
predictive or pretext learning.
• In this process, the unsupervised problem is transformed into a supervised problem by
auto-generating the labels. To make use of the huge quantity of unlabeled data, it is
crucial to set the right learning objectives to get supervision from the data itself.
• The process of the self-supervised learning method is to identify any hidden part of
the input from any unhidden part of the input.
What is the difference between self-supervised and unsupervised learning?
• Self-supervised learning and unsupervised learning methods can be considered
complementary learning techniques as both do not need labeled datasets.
Unsupervised learning can be considered as the superset of self-supervised learning as
it does not have any feedback loops. On the contrary, self-supervised learning has a
lot of supervisory signals that act as feedback in the training process.
• An easier way to put it is that the ‘unsupervised’ learning technique focuses a lot on
the model and not on the data whereas the ‘self-supervised learning’ technique works
the other way around. However, unsupervised learning methods are good at
clustering, and dimensionality reduction, while self-supervised learning is a pretext
method for regression and classification tasks.
Limitations of Supervised Learning
Supervised learning requires explicit "input-output" pairs (labels), which presents three major
hurdles:
• High Costs & Time: Manual labeling is labor-intensive. For instance, labeling one
hour of speech can take 10+ hours of human work.
• Need for Domain Expertise: Fields like medical imaging (e.g., Radiology on NIH)
or law require highly paid specialists to annotate data, making it prohibitively
expensive.
• Human Bias: Annotations are subjective. Different humans may label the same
image or sentiment differently, introducing noise that limits model accuracy.
Self-Supervised Learning Workflow
• Unlabeled Data Input: Massive volumes of raw data (images, text, audio, or IoT
sensor logs) are fed into the system without manual annotations.
• Pretext Task (Self-Supervision): The model creates its own "labels" by solving
artificial puzzles. Common tasks include:
Masking: Predicting hidden parts of a sentence or image.
Rotation: Predicting the angle an image has been rotated.
Contrastive Learning: Determining if two views belong to the same object.
• Representation Learning: Through these tasks, the model develops a deep "feature
extraction layer" that understands the data's underlying patterns, such as shapes,
textures, or grammar.
• Fine-Tuning: The pre-trained model is adapted for a specific "downstream task"
using a very small labeled dataset.
• Downstream Task: The final model is deployed for real-world applications like
medical image classification, object detection, or speech recognition.
key characteristics of SSL
• Uses Unlabeled Data: The model learns directly from raw data without needing
humans to label it.
• Dynamic Label Generation: The model generates training labels by understanding
the data structure itself.
• Mix of Learning Methods: SSL is a middle ground between supervised learning
(with labels) and unsupervised learning (without labels).
• Learns Useful Features: By learning from the data itself, the model can understand
important patterns and details which helps it perform better on new data.
• Wide Applications: It is widely used in areas like image recognition, natural
language processing and speech recognition, where labeled data can be expensive or
limited.
• Helps Transfer Learning: SSL makes it easier to adapt models to new tasks by using
the knowledge gained from pre-training on unlabeled data.
 MODULE 1 : Pretext Tasks

Pretext tasks are recognized as the core driving mechanism behind self-supervised
and autonomous foundation models. They enable learning from the massive amounts of
unlabeled data available in the real world by generating their own supervision signals, a
process often referred to as synthetic supervision. Instead of relying on costly human
annotations, the model learns by solving carefully designed artificial tasks derived directly
from the raw data.
1. Definition and Core Concept
A Pretext Task (also called a proxy or auxiliary task) is an artificial learning objective
constructed such that the ground-truth labels are automatically obtained from the data
itself. This means the model supervises its own learning.
Key Characteristics
(a) Internal Learning Signal (Synthetic Supervision)
Unlike supervised learning, where labels are externally provided, pretext tasks create labels
internally.
For example:
 In NLP, a word in a sentence is masked, and the model is trained to predict the
missing word using surrounding context.
 In Computer Vision, an image may be rotated, and the model predicts the rotation
angle.
Here, the input data itself provides the target, eliminating the need for manual labeling.
(b) No Human Annotation Required
Because labels are generated automatically:
 The approach scales efficiently to millions or billions of samples
 Training cost and time are drastically reduced
 Learning becomes feasible for large-scale foundation models
This property makes pretext tasks ideal for learning from raw text, images, videos, and
sensor data.
2. Representation Learning & Downstream:
The goal of a pretext task is not to solve the artificial task itself. Instead, it serves as a tool
to force the model to learn useful and transferable internal representations of data.
(a) Representation Learning - Feature Discovery
• Definition: The process of automatically transforming raw, high-dimensional data
(pixels, audio waves, sensor logs) into a dense mathematical format (embeddings) that
captures the "essence" of the information. Through these tasks, the model develops a
deep "feature extraction layer" that understands the data's underlying patterns, such as
shapes, textures, or grammar.
To perform well on the pretext task, the model must capture:
 Visual structure (edges, shapes, depth, lighting)
 Linguistic structure (syntax, semantics, grammar)
 Temporal dependencies (motion, sequence flow)
These learned features form a rich, general-purpose embedding space that captures the true
structure of the data.
(b) Downstream - “Discard and Transfer” Strategy
Once pre-training is complete:
1. The task-specific head (used only for solving the pretext task) is discarded
2. The feature extractor / backbone network is retained
3. These learned representations are transferred to downstream tasks, such as:
o Object detection
o Medical image diagnosis
o Speech recognition
o Sentiment analysis
This approach significantly improves performance, especially when labeled data for the
downstream task is limited.

3. Common Types of Pretext Tasks

pretext tasks have become highly sophisticated across various modalities:
1. Context-Based (Spatial) Pretext Tasks
2. Transformation-Based Pretext Tasks
3. Reconstruction-Based Pretext Tasks
4. Temporal (Video-Based) Pretext Tasks
5. Contrastive Pretext Tasks
3.1 Context-Based (Spatial) Pretext Tasks
 The primary goal of context-based pretext tasks is to learn spatial structure
and object layout within an image.
 These tasks encourage the model to understand how different parts of an image
are arranged relative to each other, rather than focusing only on local textures.
By learning spatial context, the network develops an implicit understanding of:
 Object parts (head, limbs, wheels, etc.)
 Scene geometry
 Structural relationships in images

Types of Context-Based (Spatial) Pretext Tasks

 Relative Patch Position Prediction
 Jigsaw Puzzle Solving
3.1.1 Predicting relative position of image patches
 Input: 2 image patches, one is the anchor image patch while the other is the query
image patch.
 Given the 2 image patches, the network needs to predict the relative position of the
query image patch with respect to the anchor image patch.
 Thus, this problem can be modelled as an 8-way classification problem, since there
are 8 possible locations for a query image, given an anchor.
 And, the label for this task can be automatically generated by feeding the relative
position of query patch with respect to the anchor.
3.1.2. Jigsaw Puzzle Pretext Task
In the Jigsaw Puzzle task, an image is divided into a grid of patches (e.g., 3×3).
These patches are randomly shuffled, and the network must predict the correct
arrangement.
3.2 Transformation-Based:
Transformation-Based Pretext Tasks are a category of self-supervised learning tasks
in which a model is trained to predict a known transformation applied to the input
data.
Since the transformation is artificially applied, the labels are automatically available,
eliminating the need for manual annotation.

3.2.1 Predicting Rotation of Images

 Predicting rotations is one of the most popular pretext task which has a simple and
straightforward architecture and requires minimal sampling.
 We apply rotations of 0, 90, 180, 270 degrees to the image and send these rotated
images to the network to predict what sort of rotation was applied to the image and
the network simply performs a 4-way classification to predict the rotation.
 Predicting rotations does not make any semantic sense, we are just using this pretext
task as a proxy to learn some features and representations to be used in a downstream
task.
Figure 5: Rotations of Image
Why rotation helps or why it works?
It has been proven that it works empirically. The intuition behind it is that in order to predict
the rotations, model needs to understand the rough boundaries and representation of an
image. For example, it will have to segregate the sky from water or sand from the water or
will understand that trees grow upwards and so on.
3.2.2 Colourisation

Figure 6: Colourisation
In this pretext task, we predict the colours of a grey image. It can be formulated for any
image, we just remove the colour and feed this greyscale image to the network to predict its
colour. This task is useful in some respects like for colourising the old greyscale films. we
can apply this pretext task). The intuition behind this task is that the network needs to
understand some meaningful information like that the trees are green, the sky is blue and so
on.
3.3 Reconstruction-Based Pretext Tasks
Learn global and local context by reconstructing data
This means:
  The model is shown incomplete or damaged data
 It must rebuild the original input
 To do this, it must understand what should be there

Masked Language Modeling: Predicting hidden words in a text (e.g., BERT-style).

Context-Aware: Predicting missing pixels in an image based on the surrounding area.
3.3.1 Masked Language Modeling (MLM) Real-Time Applications

Masked Language Models (MLMs) are a type of machine learning model designed to
predict missing or "masked" words in a sentence. These models are trained on large
datasets of text where certain words are intentionally hidden during training. The goal
of the model is to guess the hidden word based on the surrounding context. This
approach helps the model learn the relationships between words and develop a deeper
understanding of language structure.
How Do Masked Language Models Work?
The process of training a masked language model involves two main steps:
1. Masking Words
During training, the model is presented with sentences where some words are
randomly replaced with a special token, such as "[MASK]." In the below example,
two words have been replaced with mask tokens while another word replaced by
different word token.

2. Predicting Missing Words

The model is then tasked with predicting the original word that was masked. It does
this by analyzing the surrounding words in the sentence. This process is repeated
millions of times across vast amounts of text data and allow the model to learn
patterns, grammar and semantic relationships in language.

Why Are Masked Language Models Important?

Masked language models become important for modern NLP for several reasons:
1. Bidirectional Understanding
Unlike earlier models that processed text in a single direction (either left-to-right or
right-to-left) MLMs are bidirectional . This means they analyze the entire context of
a word—both the words before it and the words after it. This bidirectional approach
allows the model to capture richer and more nuanced meanings.
 2. Contextual Word Representations
Words can have different meanings depending on the context in which they appear.
For example the word "bank" could refer to a financial institution or the side of a
river. MLMs excel at understanding these contextual differences because they rely on
the surrounding words to make predictions.
3. Versatility
Once trained, masked language models can be fine-tuned for a wide range of
downstream tasks, such as:
 Text Classification : Determining the sentiment of a review (positive, negative,
neutral).
 Named Entity Recognition : Identifying names, dates and locations in a document.
 Question Answering : Providing answers to questions based on a given passage of
text.
 Language Translation : Converting text from one language to another.

Applications of Masked Language Models

The versatility of masked language models makes them applicable to a wide range of
real-world scenarios. Some common applications include:
1. Search Engines : MLMs help improve search engine results by understanding the
intent behind user queries and providing more relevant answers.
1. Chatbots and Virtual Assistants : By understanding context and generating coherent
responses, MLMs power conversational AI systems like Siri, Alexa, and Google
Assistant.
1. Content Generation : MLMs can assist in writing articles, creating marketing copy,
or even generating creative stories.
1. Healthcare : In medical research, MLMs can analyze clinical notes, extract important
information, and assist in diagnosing diseases.
1. Education : MLMs can be used to create personalized learning experiences, such as
grading essays or providing feedback on grammar and style.
1. Customer Support : Many companies use MLMs to automate responses to customer
inquiries, improving efficiency and reducing response times.

3. 3.2 Context-Aware
What Does “Context-Aware” Mean in SSL?
A self-supervised model is context-aware if it:
 Learns representations by predicting missing, future, or related parts of data
 Preserves global and local dependencies
 Understands semantic consistency beyond simple similarity
Core idea:
“Use context as supervision instead of labels.”
Fill in the blanks
We hide a part of an image and predict the hidden part from the remaining surrounding part
of the image. This works because the network will learn the implicit structure of the data like
to represent that cars run on roads, buildings are composed of windows & doors and so on.

3.4 Temporal (Video-Based) Pretext Tasks

The objective of temporal pretext tasks is to enable the model to learn motion patterns
and temporal dependencies present in video data.
Instead of relying on static appearance, the model is forced to understand how visual content
evolves over time.

These tasks exploit the natural ordering of video frames as a free source of supervision.

Examples:
 Temporal Order Prediction (Shuffle & Learn)
 Future Frame Prediction
  Odd-One-Out in Video Clips
3.4.1 Future Frame Prediction
Videos are composed of sequences of frames and this notion is the idea behind self-
supervision, which can be leveraged for some pretext tasks like predicting the order of
frames, fill in the blanks and object tracking.
3.4.2 Shuffle & Learn

Figure 7: Interpolation
Given a bunch of frames, we extract three frames and if they are extracted in the right order
we label it as positive, else if they are shuffled, label it as negative. This now becomes a
binary classification problem to predict if the frames are in the right order or not. So, given a
start and end point, we check if the middle is a valid interpolation of the two.

Figure 8: Shuffle & Learn architecture

We can use a triplet Siamese network (Anchor (A), Positive (P) , Negative (N) )where the
three frames are independently fed forward and then we concatenate the generated features
and perform the binary classification to predict if the frames are shuffled or not.

Figure 9: Nearest Neighbours Representation

 Query (left):
A query video frame (e.g., a person doing a handstand or sports action).
 Nearest Neighbors (right):
The most similar frames retrieved using learned feature embeddings (fc7 features).
 Three different feature sources are compared:
1. ImageNet (supervised)
2. Shuffle & Learn (self-supervised)
3. Random (untrained)
The better the learned representation, the more semantically similar the retrieved frames
look to the query.
1. Query
 A single video frame representing an action (e.g., gymnastics, baseball pitch).
 This frame is encoded into a feature vector.
 Nearest neighbors are retrieved based on feature similarity.
2. ImageNet (Supervised Learning)
 Features extracted from a CNN pretrained on ImageNet (object classification).
 Observations:
o Retrieved frames often contain similar objects or scenes (people, sports
equipment).
o However, they may not capture motion or action semantics well.
 Limitation:
o ImageNet focuses on static object labels, not temporal relationships or
actions.
3. Shuffle & Learn (Self-Supervised Learning)
  Model trained using a self-supervised pretext task:
o Given shuffled video frames, predict the correct temporal order.
 This forces the network to learn:
o Motion cues
o Human pose
o Action dynamics
 Observations:
o Retrieved frames show similar actions or body configurations
o Strong semantic alignment with the query (e.g., gymnastics ↔ gymnastics)
4. Random (Untrained Network)
 Features from a randomly initialized network.
 Observations:
o Retrieved frames are visually and semantically unrelated.
 Meaning:
o No meaningful representation is learned.

3.4.3 Odd-One-Out in Video Clips (Temporal Pretext Task)

Correct video sequences follow natural temporal order, while incorrect ones look
unrealistic.
The model is trained to identify the one video clip that violates temporal consistency
among a set of otherwise correct clips.
MODULE 1 : CONSTRATIVE LEARNING
What is contrastive learning?
Contrastive Learning is a way of training a model using data without labels.
The model learns by comparing data points. It is taught that some data points are similar
and some are different, and it learns to tell them apart.
That is, as the name suggests, samples are contrasted against each other, and those
belonging to the same distribution are pushed towards each other in the embedding space.
In contrast, those belonging to different distributions are pulled against each other.

How Contrastive Learning works

Contrastive Learning mimics the way humans learn. For example, we might not know what
otters are or what grizzly bears are, but seeing the images (as shown below), we can at least
infer which pictures show the same animals.
The basic contrastive learning framework consists of selecting a data sample, called
“anchor,” a data point belonging to the same distribution as the anchor, called the “positive”
sample, and another data point belonging to a different distribution called the “negative”
sample.
The SSL model tries to minimize the distance between the anchor and positive samples, i.e.,
the samples belonging to the same distribution, in the latent space, and at the same time
maximize the distance between the anchor and the negative samples.

As shown in the example above, two images belonging to the same class lie close to each
other in the embedding space (“d+”), and those belonging to different classes lie at a greater
distance from each other (“d-”). Thus, a contrastive learning model (denotes by “theta” in the
example above) tries to minimize the distance “d+” and maximize the distance “d-.”
Two methods
1. Instance Discrimination Method
2. Image Subsampling/Patching Method
Instance Discrimination Method
In this class of Contrastive Learning, the entirety of images are made to undergo
transformations and used as positive samples to an anchor image. For example, if we select
an image of a dog as the anchor, we can mirror the image or convert it to grayscale to use
as the positive sample. The negative sample can be any other image in the dataset.
The image below shows the basic framework of the instance discrimination-based
contrastive learning technique. The distance function can be anything, from Euclidean
distance to cosine distances in the embedding space.
Some image augmentation methods popularly used for Instance Discrimination-based
Contrastive Learning is listed as follows:
1. Colour Jittering: Here, the brightness, contrast, and saturation of an RGB image are
changed randomly. This technique is helpful to ensure a model is not memorizing a
given object by the scene's colors. While output image colors can appear odd to
human interpretation, such augmentations help a model consider the edges and
shape of objects rather than only the colors.
2. Image Rotation: An image is rotated randomly within 0-90 degrees. Since rotating
an image doesn’t change the core information contained in it (i.e., a dog in an image
will still be a dog), models are trained to be rotation invariant for robust prediction.

3. Image Flipping: The image is flipped (mirrored) about its center, either vertically or
horizontally. This is an extension of the concept of image rotation-based
augmentation.

4. Image Noising: Random noise is added to the images pixel-wise. This technique
allows the model to learn how to separate the signal from the noise in the image and
makes it more robust to changes in the image during test time. For example,
randomly changing some pixels in the image to white or black is known as salt-and-
pepper noise (an example is shown below).

5. Random Affine: Affine is a geometric transformation that preserves lines and

parallelism, but not necessarily the distances and angles.
Image Subsampling/Patching Method
This class of Contrastive Learning methods breaks a single image into multiple patches of a
fixed dimension (say, 10x10 patch windows). There might be some degree of overlap
between the patches.
Now, suppose we take the image of a cat and use one of its patches as the anchor while
leveraging the rest as the positive samples. Patches from other images (say, one patch each
of a raccoon, an owl, and a giraffe) are used as negative samples.

Contrastive Learning: Objectives

A number of loss functions have been defined in the Contrastive Learning literature for
applications in different problems, each with its own set of functionalities. Let us discuss
some of these in this section.
Types of loss
1. Max margin Contrastive Loss
2. Triplet Loss
3. N-pair Loss
4. InfoNCE
5. NT-Xent Loss

1. Max margin Contrastive Loss

It is one of the oldest loss functions proposed in the Contrastive Learning literature .
The basic idea here is that the loss function maximizes the distance between samples if they
do not belong to the same distribution and instead minimizes the distance between them if
they belong to the same distribution. It is mathematically represented as follows
Let:
 → similar pair
 → dissimilar pair


Here, “s_i” and “s_j” are the two samples with corresponding labels “y_i” and “y_j” that need
to be compared, “theta” is the embedding network, and “epsilon” is a hyperparameter,
defining the lower bound distance between samples of different classes.
The labels for the samples are generated by whether the samples belong to the same
distribution. For example, if the two samples are cropped versions of the same image or the
augmented versions of the same sample, the labels will be the same.
2. Triplet Loss
The triplet loss is a lot similar to the contrastive loss, both of which try to minimize the
distance between similar distributions and maximize the distance between unlike
distributions. The primary difference in the triplet loss is that a positive and a negative
sample are simultaneously taken as input with the anchor sample to compute the loss.
Mathematically it is represented as follows:
Given three samples:
 Anchor
 Positive (same class as anchor)
 Negative (different class)
Let:
 be the embedding function

 = margin

For example, raccoons and ringtails look a lot similar (see the image below). Both have
striped, bushy tails and similar body fur colors. When sampling the negative samples against
raccoons, choosing ringtails will enable the model to differentiate between classes more
effectively, than say, when choosing an elephant as the negative sample.

3. N-pair Loss
The N-pair loss is an extension of the triplet loss function. Instead of sampling a single
negative sample, an “N” number of negative samples are sampled along with one anchor
and one positive sample. The mathematical representation for this loss is as follows:
N-Pair Loss Formula
Given:
 One anchor sample
 One positive sample (same class as anchor)
 negative samples
The N-Pair Loss is:

Meaning in Simple Words

 : similarity between anchor and positive
 : similarity between anchor and negative
 The loss:
o Minimizes distance between anchor and positive
o Maximizes distance between anchor and all negatives at once

4. InfoNCE
InfoNCE, where NCE stands for Noise-Contrastive Estimation, is another type of contrastive
loss function.
If “S = {s_1, s_2, …, s_N}” denotes the set of “N” random samples containing one positive
sample and “N-1” negative samples, the loss function can be mathematically represented as
follows:
Given:
 An anchor
 A positive sample
 negative samples
Let:
 be the embedding
 = similarity (usually cosine similarity)
 = temperature

[Link]-Xent Loss (SimCLR Form)

NT-Xent (Normalized Temperature-scaled Cross-Entropy) is the contrastive loss used
in SimCLR to learn representations without labels.
NT-Xent Formula (SimCLR)
For a positive pair :

Symbo Meanin
l g

Embedding of one augmented view

(anchor)
 Embedding of the other augmented view of the same image
(positive)

Any embedding in the

batch

sim(· Cosine
) similarity

Temperatur
e

Total number of
embeddings

Excludes self-
comparison

MODULE 1: SIMCLR
SimCLR: A Simple Framework for Contrastive Learning of Visual
Representations
SimCLR, developed by researchers at Google Brain, is a self-supervised learning framework
that learns visual representations without requiring labeled data. It is built upon contrastive
learning, where the model is trained to bring similar (positive) image pairs closer and push
dissimilar (negative) pairs apart in the feature space.
SimCLR a self-supervised learning framework that trains models to understand images
without labels by teaching them to recognize different "views" (augmented versions) of the
same image as similar, while treating views from different images as dissimilar, using a
contrastive loss function and large batch sizes for strong feature learning
Traditional deep learning relies heavily on labeled datasets, which are expensive and time-
consuming to create. Self-supervised learning, and specifically SimCLR, tackles this by:
 Eliminating the need for manual labels: Models learn from raw, unlabeled data.
 Learning robust visual features: These features can be used for a variety of
downstream tasks.
 Enabling strong performance with simple linear classifiers: After SSL
pretraining, even a basic classifier on top of the learned features can achieve
competitive results.

SimCLR Training Workflow

A detailed explanation of the SimCLR’s training workflow can be found in the next section
(SimCLR’s Algorithm) of this article.
 Augment input images into x1 and x2 (refer to the image below).
 Encode both views using the ResNet encoder to get learned representations h1​ and h2​.
 Pass the encoded representations through the projection head to get projections z1 and
z2 in the feature space.
 Compute the NT-Xent loss over all pairs (positive and negative) and then maximize
the agreement between the projections in the feature space itself.
 Backpropagate gradients and update parameters using LARS Optimizer.
 SimCLRs Contrastive Learning Process

Core Architecture of SimCLR

SimCLR consists of four main components:
1.1. Data Augmentation
 A stochastic data augmentation module that transforms any given data example
randomly resulting in two correlated views of the same example,
denoted ~xi and ~xj, as positive pairs.
  Three simple augmentations are applied sequentially: random cropping followed
by resize back to the original size, random color distortions, and random
Gaussian blur.

Random Crop
 By randomly cropping images, the contrastive prediction tasks are sampled that
include global to local view (B→A) or adjacent view (D→C) prediction.

Illustrations of the studied data augmentation operators

 The above data augmentation operators are studied.
Linear evaluation (ImageNet top-1 accuracy) under individual or composition of data
augmentations, applied only to one branch
 No single transformation suffices to learn good representations.
The combination of random crop and color distortion is crucial to achieve a good
performance.
1.2. Base Encoder
Each augmented image is passed through a shared encoder network, typically a ResNet
(e.g., ResNet-50).

Where:
 is the ResNet encoder
 are high-level feature representations
Key points:
 The same encoder weights are used for both views
 The encoder learns semantic features such as shapes, textures, and object
structure
 These representations are still not directly optimized for contrastive loss
Purpose:
To extract meaningful visual features from augmented images.
Linear evaluation of models with varied depth and width
Unsupervised learning benefits more from bigger models than its supervised
counterpart.
1.3. Projection Head
The encoded features and are passed through a projection head, usually a small MLP
(Multi-Layer Perceptron).

Where:
 is the projection head (e.g., Linear → ReLU → Linear)
 are low-dimensional vectors in the contrastive feature space
Why projection head?
 Contrastive loss performs better in projection space than representation space
 The encoder output remains more general and transferable
 The projection head absorbs task-specific constraints
Purpose:
To map representations into a space where similarity comparison is more effective.
Linear evaluation of representations with different projection heads g() and various
dimensions of z = g(h). h has 2048 dimensional
It is found that it is beneficial to define the contrastive loss on zi’s rather than hi’s.

1.4. Contrastive Loss

 A minibatch of N examples is randomly sampled.

Linear evaluation models (ResNet-50) trained with different batch size and epochs
Large batch size is beneficial, longer training time is also beneficial.
 The contrastive prediction task is defined on pairs of augmented examples derived
from the minibatch, resulting in 2N data points.
 Given a positive pair, the other 2(N-1) augmented examples within a minibatch
as negative examples.
 The loss function for a positive pair of examples (i, j) is defined as:
  where sim(,) is cosine similarity, τ is the temperature parameter.
 The final loss is computed across all positive pairs, both (i, j) and (j, i), in a mini-
batch.
It is named as NT-Xent (the normalized temperature-scaled cross entropy loss).
Advantages
 Label-efficient: Needs fewer or no labels for pretraining
 Modular: Works with different architectures
 Simple to implement: No complex pretext tasks or auxiliary networks
 Scalable with larger batch sizes and stronger augmentations
Limitations
 Requires very large batch sizes for many negative examples (128+ images)
 Compute-intensive (especially with ResNet-50/101 backbones)
 Contrastive loss may not capture higher-order semantics (like object relationships)

MODULE 1 :MoCo(Momentum Contrast )

One of the main issues that the computer vision field is facing, is that databases need human
annotation. This means that collected data needs to be annotated and labeled which is very
expensive and time-consuming. In order to solve this problem, unsupervised learning
approaches were proposed. MoCo's is closing the gap between Unsupervised representation
learning and supervised representation learning in vision tasks.
MoCo uses contrastive learning technique by making the dynamic dictionary large and
consistence. Among the most successful of the recent self-supervised approaches to learning
visual representations, a subset of these termed ‘‘contrastive’’ learning methods have
achieved the most success.
The negative samples used for contrastive learning are obtained from a dynamic dictionary.
Initially, the dictionary is empty. As the model learns, the encoder representations of the input
data are stored in the dictionary. The dictionary is maintained using a queue-based
mechanism, where new representations replace the oldest ones. This way, the dictionary
captures a wide range of negative samples over time, providing diverse negative pairs for
contrastive learning.
By continuously updating the dictionary of negative samples and training the encoder using
the contrastive loss, MoCo encourages the model to capture useful and semantically
meaningful representations. The dynamic nature of the dictionary allows the model to adapt
to changing data distributions and learn robust representations.
The MoCo approach builds on the contrastive learning principle by introducing a momentum-
based update rule that improves the stability and convergence of the training process.
Specifically, the MoCo approach uses a memory bank to store a large number of negative
examples that are used to compute contrastive losses during training. The memory bank is
updated using a momentum- based update rule that averages the parameters of the current
model with those of a slowly-updated "queue" model. This update rule helps to stabilize the
training process by providing a more consistent source of negative examples.

MoCo’s architecture:
MoCo’s main components are the query and the keys, encoder and momentum encoder,
feature queue, one hot target, and the contrastive loss. As shown in the following
figure, MoCo aims to find the matching similarity between extracted features from the query
encoder and the keys’ momentum encoder. The keys are a set of samples stored in a
dictionary.
Figure 1: MoCo architecture with query and ky dictionary [1].
Dictionary as a Queue
Momentum contrast uses a dictionary queue encoded with keys, The queue is updated by
adding the representation of the current image to the queue and removing the oldest
representation. The dictionary acts as a "memory bank" that stores a history of feature
representations. The queue is updated using a momentum-based update rule, which allows the
model to maintain a smooth and stable representation of the feature space.
During training, the images are split into two groups: a query group and a key group. The
query group is used to compute a query feature representation, while the key group is used to
compute a set of key feature representations. The query feature representation is then
compared to the key feature representations stored in the dictionary using a contrastive loss
function.
Momentum Update
Momentum update is a key component in Momentum Contrastive learning, a technique
commonly used in self-supervised learning tasks such as image or video representation
learning. It helps improve the stability and convergence speed of the learning process by
introducing a momentum term during the update of the model's parameters.
In MoCo, the momentum update is used to update the model's parameters based on the
current gradient and the momentum term. The momentum update can be visualized as
follows:
Initialize the model's parameters and momentum parameters.
At each training iteration:
a). Compute the gradients of the loss function with respect to the
parameters using the current mini-batch of data.
b).Update the momentum parameters using the momentum update equation:
where Vt is the velocity term at time step t, α is the momentum
coefficient, and gt is the gradient at time step t.
Update the model's parameters using the momentum parameters:
represents the updated parameters at time step t+1.
The momentum update equation calculates the velocity term by combining the previous
velocity ø {t-1} and the current gradient gt. The momentum coefficient α determines the
contribution of the previous velocity compared to the current gradient. A higher α value gives
more weight to the previous velocity, resulting in a smoother and more stable update
trajectory.
The momentum update allows the model to accumulate information from previous gradients
and helps the optimization process by maintaining a consistent direction of updates. This can
help the model escape shallow local minima and converge faster to better representations.
Keep in mind that while the momentum update is an essential component of MoCo, the
specific implementation details and hyperparameters may vary depending on the exact
architecture and training setup.
 MODULE 1 : What is Meta Learning?
Meta-learning is learning to learn algorithms, which aim to create AI systems that can adapt
to new tasks and improve their performance over time, without the need for extensive
retraining.
Meta-learning algorithms typically involve training a model on a variety of different tasks, with
the goal of learning generalizable knowledge that can be transferred to new tasks. This is
different from traditional machine learning, where a model is typically trained on a single task
and then used for that task alone.
 Meta-learning, also called "learning to learn" algorithms, is a branch of machine
learning that focuses on teaching models to self-adapt and solve new problems with
little to no human intervention.
 It entails using a different machine learning algorithm that has already been trained to
act as a mentor and transfer knowledge. Through data analysis, meta-learning gains
insights from this mentor algorithm's output and improves the developing algorithm's
ability to solve problems effectively.
 To increase the flexibility of automatic learning, meta-learning makes use of
algorithmic metadata. It comprehends how algorithms adjust to a variety of problems,
improving the functionality of current algorithms and possibly even learning the
algorithm itself.
  Meta-learning optimizes learning by using algorithmic metadata, including
performance measures and data-derived patterns, to strategically learn, select, alter,
or combine algorithms for specific problems.
The process of learning to learn or the meta-training process can be crudely summed up in
the following diagram:

Working of Meta Learning

Training models to quickly adapt to new tasks with minimal data is the focus of a machine
learning paradigm known as "meta-learning," or "learning to learn." In order to help models
quickly adapt to new, untested tasks using a limited amount of task-specific data, meta-
learning aims to enable models to generalize learning experiences across different tasks.
Two primary phases are involved in the typical meta-learning workflow:
 Meta - Learning
o Tasks: Exposure to a range of tasks, each with its own set of parameters or
characteristics, is part of the meta-training phase.
o Model Training: Many tasks are used to train a base model, also known as a
learner. The purpose of this model is to represent shared knowledge or
common patterns among various tasks.
o Adaption: With few examples, the model is trained to quickly adjust its
parameters to new tasks.
 Meta - Testing(Adaption)
o New Task: The model is given a brand-new task during the meta-testing
stage that it was not exposed to during training.
 o Few Shots: With only a small amount of data, the model is modified for the
new task (few-shot learning). In order to make this adaptation, the model's
parameters are frequently updated using the examples from the new task.
o Generalization: Meta-learning efficacy is evaluated by looking at how well
the model quickly generalizes to the new task.
Why we need Meta-Learning
Meta-Learning can enable the machine to learn more efficiently and effectively from limited
data and it can adapt to any changes in the problem quickly. Here are some examples of
meta-learning processes:
 Few-shot Learning: It is a type of learning algorithm or technique, which can learn in
very few steps of training and on limited examples.
 Transfer Learning: It is a technique in which knowledge is transferred from one task
to another if there are some similarities between both tasks. In this case, another
model can be developed with very limited data and few-step training using the
knowledge of another pre-trained model.
Learning the meta-parameters
Throughout the whole training process, backpropagation is used in meta-learning to back-
propagate the meta-loss gradient, all the way back to the original model weights. It is highly
computational, uses second derivatives, and is made easier by frameworks such as
Tensorflow and PyTorch. By contrasting model predictions with ground truth labels, the meta-
loss—a measure of the meta-learner's efficacy—is obtained. Parameters are updated during
training by meta-optimizers such as SGD, RMSProp, and Adam.
Three main steps subsumed in meta-learning are as follows:
1. Inclusion of a learning sub-model.
1. A dynamic inductive bias: Altering the inductive bias of a learning algorithm to
match the given problem. This is done by altering key aspects of the learning
algorithm, such as the hypothesis representation, heuristic formulae, or parameters.
Many different approaches exist.
1. Extracting useful knowledge and experience from the metadata of the
model: Metadata consists of knowledge about previous learning episodes and is
used to efficiently develop an effective hypothesis for a new task. This is also a form
of Inductive transfer.
Meta-Learning Approaches
There are several approaches to Meta-Learning, some common approaches are as follows:
1. Metric-based meta-learning: This approach basically aims to find a metric space. It
is similar to the nearest neighbor algorithm which measures the similarity or distance
to learn the given examples. The goal is to learn a function that converts input
examples into a metric space with labels that are similar for nearby points and
dissimilar for far-off points. The success of metric-based meta-learning models
depends on the selection of the kernel function, which determines the weight of each
labeled example in predicting the label of a new example.
 Applications of metric-based meta-learning include few-shot classification, where the
goal is to classify new classes with very few examples.
1. Optimization-based Meta-Learning: This approach focuses on optimizing
algorithms in such a way that they can quickly solve the new task in very less
examples. In the neural network to better accomplish a task Usually, multiple neural
networks are used. One neural net is responsible for the optimization (different
techniques can be used) of hyperparameters of another neural net to improve its
performance.
Few-shot learning in reinforcement learning is an example of an optimization-based
meta-learning application where the objective is to learn a policy that can handle new
issues with a small number of examples.
1. Model-Agnostic Meta-Learning (MAML): It is an optimization-based meta-learning
framework that enables a model to quickly adapt to new tasks with only a few
examples by learning generalizable features that can be used in different tasks. In
MAML, the model is trained on a set of meta-training tasks, which are similar to the
target tasks but have a different distribution of data. The model learns a set of
generalizable parameters that can be quickly adapted to new tasks with only a few
examples by performing a few gradient descent steps.
1. Model-based Meta-Learning: Model-based Meta-Learning is a well-known meta-
learning algorithm that learns how to initialize the model parameters correctly so that
it can quickly adapt to new tasks with few examples. It updates its parameters rapidly
with a few training steps and quickly adapts to new tasks by learning a set of
common parameters. It could be a neural network with a certain architecture that is
designed for fast updates, or it could be a more general optimization algorithm that
can quickly adapt to new tasks. The parameters of a model are trained such that
even a few iterations of applying gradient descent with relatively few data samples
from a new task (new domain) can lead to good generalization on that task.
Model-based meta-learning has shown impressive results in various domains,
including few-shot learning, robotics, and natural language processing.
 Memory-Augmented Neural Networks: Memory-augmented neural
networks, such as Neural Turing Machines (NTMs) and Differentiable Neural
Computers (DNCs), utilize external memory for improved meta-learning,
enabling complex reasoning and tasks like machine translation and image
captioning.
 Meta Networks: Meta Networks is a model-based meta-learning. The key
idea behind Meta Networks is to use a meta-learner to generate the weights
of a task-specific network, which is then used to solve a new task. The task-
specific network is designed to take input from the meta-learner and produce
output that is specific to the new task. In other words, the architecture of the
task-specific network is learned on-the-fly by the meta-learner during the
meta-training phase, which enables rapid adaptation to new tasks with only a
few examples.
 Bayesian Meta-Learning: Bayesian Meta-Learning or Bayesian optimization
is a family of meta-Learning algorithms that uses the bayesian method for
optimizing a black-box function that is expensive to evaluate, by constructing
a probabilistic model of the function, which is then iteratively updated as new
data is acquired.
Comparison of Various Meta-Learning Techniques

Approach Description Application

Learns a metric space

Metric-based meta-
where nearby points have Few-shot classification.
learning
similar labels.

Optimizes algorithms to
Optimization-based meta- Few-shot learning in
quickly solve new tasks
learning reinforcement learning.
with limited data.

Framework for quickly Various machine-learning

Model-Agnostic Meta-
adapting to new tasks with tasks.
Learning (MAML)
limited data.

Gradient-based meta-
learning algorithm that
Reptile Few-shot learning.
updates model parameters
through iterations.

Learning to learn by
Meta-learning approach
gradient descent by Few-shot learning and
that optimizes meta-
gradient descent (L2L- transfer learning.
optimization algorithms.
GD2)

Advantages of Meta-learning
1. Meta-Learning offers more speed: Meta-learning approaches can produce learning
architectures that perform better and faster than hand-crafted models.
1. Better generalization: Meta-learning models can frequently generalize to new tasks
more effectively by learning to learn, even when the new tasks are very different from
the ones they were trained on.
1. Scaling: Meta-learning can automate the process of choosing and fine-tuning
algorithms, thereby increasing the potential to scale AI applications.
1. Fewer data required: These approaches assist in the development of more general
systems, which can transfer knowledge from one context to another. This reduces
the amount of data you need in solving problems in the new context.
1. Improved performance: Meta-learning can help improve the performance of
machine learning models by allowing them to adapt to different datasets and learning
environments. By leveraging prior knowledge and experience, meta-learning models
can quickly adapt to new situations and make better decisions.
 1. Fewer hyperparameters: Meta-learning can help reduce the number of
hyperparameters that need to be tuned manually. By learning to optimize these
parameters automatically, meta-learning models can improve their performance and
reduce the need for manual tuning.
Meta-learning Optimization
During the training process of a machine learning algorithm, hyperparameters determine
which parameters should be used. These variables have a direct impact on how successfully
a model trains. Optimizing hyperparameters may be done in several ways.
1. Grid Search: The Grid Search technique makes use of manually set
hyperparameters. All suitable combinations of hyperparameter values (within a given
range) are tested during a grid search. After that, the model selects the best
hyperparameter value. But because the process takes so long and is so ineffective,
this approach is seen as conventional. Grid Search may be found in the Sklearn
library.
1. Random Search: The optimal solution for the created model is found using
the random search approach, which uses random combinations of the
hyperparameters. Even though it has characteristics similar to grid search, it has
been shown to produce superior results overall. The disadvantage of random search
is that it produces a high level of volatility while computing. Random Search may be
found in the Sklearn library. Random Search is superior to Grid Search.
Applications of Meta-learning
Meta-learning algorithms are already in use in various applications, some of which are:
1. Online learning tasks in reinforcement learning
1. Sequence modeling in Natural language processing
1. Image classification tasks in Computer vision
1. Few-shot learning: Meta-learning can be used to train models that can quickly
adapt to new tasks with limited data. This is particularly useful in scenarios where the
cost of collecting large amounts of data is prohibitively high, such as in medical
diagnosis or autonomous driving.
1. Model selection: Meta-learning can help automate the process of model selection
by learning to choose the best model for a given task based on past experience. This
can save time and resources while also improving the accuracy and robustness of
the resulting model.
1. Hyperparameter optimization: Meta-learning can be used to automatically tune
hyperparameters for machine-learning models. By learning from past experience,
meta-learning models can quickly find the best hyperparameters for a given task,
leading to better performance and faster training times.
1. Transfer learning: Meta-learning can be used to facilitate transfer learning, where
knowledge learned in one domain is transferred to another domain. This can be
especially useful in scenarios where data is scarce or where the target domain is
vastly different from the source domain.
 1. Recommender systems: Meta-learning can be used to build better recommender
systems by learning to recommend the most relevant items based on past user
behavior. This can improve the accuracy and relevance of recommendations, leading
to better user engagement and satisfaction.
Conclusion: Although Meta-Learning approaches are currently computationally expensive,
they are an exciting frontier for AI Research and can be a big step forward in our quest to
achieve Artificial General Intelligence, as computers would have the ability to not only make
accurate classifications and estimates but would able to improve their parameters (and
hyperparameters) to get better at multiple tasks in multiple problem contexts.
What is Zero-Shot Learning?
Zero-shot learning (ZSL) is a machine learning technique where a model can classify data it
has never seen before. Instead of relying on training data for each possible category, ZSL
uses semantic information about the categories to make predictions. This approach mimics
how humans can identify new objects by sensing their characteristics or details.
Key Characteristics of Zero-Shot Learning
 No Training Data for New Classes: The model is not exposed to any examples of
the new classes during training.
 Utilizes Prior Knowledge: The model relies on external knowledge or semantic
descriptions to understand and classify the new classes.
 Zero-shot learning (ZSL) is a machine learning technique where a model can classify
data it has never seen before.
For example, if a machine has learned about animals like cats, lions, and horses by
understanding features like “small and fluffy,” “big wild cat,” or “long face,” it can use
this knowledge to identify a new animal, like a tiger. Even if it has never seen a tiger
before, it can use a description like “a lion-like animal with dark stripes” to identify it
correctly. This makes it easier for machines to learn and adapt without needing lots of
examples.
What is One-Shot Learning?
One-shot learning is a machine learning approach where a model is trained to recognize or
classify new categories using only a single example. This technique is designed to handle
situations where it is impractical to gather multiple examples of every possible category. OSL
focuses on learning to generalize from a very limited amount of data, mimicking the human
ability to recognize new objects or concepts from just one observation.
Key Characteristics of One-Shot Learning
 Single Example per Class: The model is provided with exactly one example for
each new class during the training or evaluation phase. It must learn to generalize
and make accurate predictions based on this single instance.
 Similarity-Based Methods: OSL often employs similarity metrics or distance-based
approaches to compare new instances with the single example and determine their
class. Techniques such as Siamese Networks or Prototypical Networks are
commonly used.

What is Few-Shot Learning?

Few-shot learning (FSL) takes a slightly different approach. This enables a model to learn
new tasks with only a few examples. This technique is particularly useful in situations where
obtaining large amounts of labelled data is challenging. FSL takes advantage of prior
knowledge and adapts it to new tasks quickly and efficiently.
 Few-shot learning is a method where systems learn to recognize new objects using
just a small number of examples. For example, if you show a model a few pictures of
a penguin, pelican, and puffin (this small group is called the "support set"), it learns
what these birds look like.
 Later, if you show the model a new picture, like a penguin, it compares this new
picture with the ones in its support set and picks the closest match. When gathering a
large amount of data is difficult, this method is beneficial because the system can still
learn and adapt with only a few examples.
Key Characteristics of Few-Shot Learning
 Limited Training Examples: The model is trained with a few examples of the new
classes or tasks, which allows it to adapt to new scenarios with minimal data.
 Meta-Learning: Often involves meta-learning techniques where the model learns to
learn from few examples, adapting quickly to new tasks.
 Applications: Useful in domains where acquiring large amounts of labeled data is
expensive or time-consuming, such as image recognition, natural language
processing, and medical diagnostics.

Similarities Between Zero-Shot Learning, One-Shot and Few-Shot Learning

Zero-Shot Learning (ZSL), One-Shot Learning, and Few-Shot Learning (FSL) share
several similarities.
Here are the key similarities:
1. Handling Limited Data: All three approaches address the challenge of limited data
availability, aiming to enable effective learning and prediction even when the amount
of training data for new classes is minimal.
1. Focus on Generalization: All three methods emphasize the ability to generalize
from limited examples or information, striving to make accurate predictions about
new or unseen instances based on minimal prior knowledge.
1. Adaptation to New Tasks or Classes: Each approach is focused on adapting the
model to handle new classes or tasks, whether through leveraging external
 knowledge, learning from a single example, or generalizing from a small set of
examples.
Differences Between Zero-Shot, One-Shot, and Few-Shot Learning
1. Number of Training Examples
 Zero-Shot Learning (ZSL): The model encounters no examples of new classes
during training. Instead, it relies on auxiliary information like semantic descriptions or
attributes to make predictions about these unseen classes.
 One-Shot Learning (OSL): The model is provided with exactly one example per new
class during the classification phase. It must generalize and recognize instances of
these classes based on this single example.
 Few-Shot Learning (FSL): The model learns from a small number of examples
(typically a few to several dozen) for each new class. It uses these few examples to
adapt and generalize to new tasks.
2. Approach to Learning
 ZSL: Utilizes semantic representations or attributes to infer the properties of unseen
classes. The model uses descriptions or embeddings to relate new classes to known
ones.
 OSL: Employs similarity-based methods, such as Siamese Networks or Prototypical
Networks, to classify new examples by comparing them to the single example
provided.
 FSL: Applies meta-learning techniques, such as Model-Agnostic Meta-Learning
(MAML), to quickly adapt to new classes with minimal examples. The model learns
how to learn from few examples effectively.
3. Training Data Requirements
 ZSL: Does not require training examples for new classes; relies on indirect
information for prediction.
 OSL: Requires exactly one example for each new class during the evaluation phase.
 FSL: Needs a small but sufficient number of examples for each new class to train the
model.
4. Applications
 ZSL: Useful when acquiring examples for new classes is impractical, such as in
image classification with unseen categories or NLP tasks with new concepts.
 OSL: Ideal for scenarios where only one example is available, such as facial
recognition with limited photos or rare disease detection with minimal data.
 FSL: Effective when a few examples can be collected, such as in object detection
with limited instances or personalized recommendations based on limited user data.

5. Challenges
  ZSL: Depends heavily on the quality of semantic information and may struggle with
very dissimilar classes.
 OSL: Performance relies on the representativeness of the single example; poor
examples can lead to inaccurate predictions.
 FSL: Risks overfitting to the small dataset; performance is sensitive to the quality and
diversity of the examples provided.
6. Methodologies
 ZSL: Techniques include semantic embedding models and attribute-based methods.
The model uses high-level descriptions to make inferences.
 OSL: Methods involve similarity-based networks and metric learning, focusing on
comparing new examples to the single provided example.
 FSL: Uses meta-learning approaches, such as MAML, to enable rapid adaptation to
new classes with few examples.
Summary of Differences between Zero-Shot Learning (ZSL), One-Shot Learning (OSL),
and Few-Shot Learning (FSL)

Zero-Shot Learning One-Shot Few-Shot

Aspect (ZSL) Learning (OSL) Learning (FSL)

Few examples
Number of Exactly one
No training examples (typically a few to
Training example per new
for new classes several dozen) per
Examples class
new class

Uses semantic Uses similarity

Approach to Uses meta-learning
descriptions or metrics or metric
Learning techniques
attributes learning

Requires one Requires a small

Relies on indirect
Training Data example for each number of
information for
Requirements new class during examples for each
prediction
evaluation new class

Useful when
Ideal for scenarios Effective when a
examples for new
Applications with only one few examples can
classes are
example available be collected
impractical

Challenges Depends on quality of Performance relies Risk of overfitting;

semantic information; on the quality of the sensitive to quality
struggles with single example and diversity of few
 dissimilar classes examples

Semantic embedding Similarity-based Meta-learning

Methodologies models, attribute- networks, metric approaches like
based methods learning MAML

Conclusion
Zero-Shot Learning (ZSL), One-Shot Learning (OSL), and Few-Shot Learning (FSL) each
offer unique approaches to handling limited training data in machine learning. ZSL excels in
classifying new categories without prior examples by leveraging semantic information, while
OSL focuses on recognizing new classes from a single example using similarity-based
methods. FSL, on the other hand, adapts to new tasks with a small number of examples
through meta-learning techniques. Despite their differences, these methods collectively
enhance the versatility and efficiency of AI models, making them invaluable in scenarios
where large datasets are challenging to obtain.

MODULE 1 :Prototypical Networks

The Prototypical Network, or ProtoNet for short, is a metric-based meta-learning algorithm
that operates similarly to the nearest neighbor classification. Metric-based meta-learning
methods classify a new example based on some distance function between and all
elements in the support set. ProtoNets implements this idea with the concept of prototypes in
a learned feature space. First, ProtoNet uses an embedding function to encode each input
in the support set into a -dimensional feature vector. Next, for each class , we collect the
feature vectors of all examples with label and average their feature vectors. Formally, we
can define this as:
Embedding Network

Support and Query Sets

Prototype computation

where is the part of the support set for which , and represents the prototype of class . The
prototype calculation is visualized below for a 2-dimensional feature space and 3 classes
(Figure credit - Snell et al.). The colored dots represent encoded support elements with the
color-corresponding class labels, and the black dots next to the class label are the averaged
prototypes.
Distance-Based Classification
For a query sample :
1. Compute embedding:

2. Compute distance to each prototype:

3. Convert distances to probabilities using softmax:

4. Assign class with minimum distance

Based on these prototypes, we want to classify a new example. Remember that since we
want to learn the encoding function , this classification must be differentiable, and hence,
we need to define a probability distribution across classes. For this, we will make use of the
distance function : the closer a new example is to a prototype , the higher the
probability for belonging to class . Formally, we can simply use a softmax over the
distances of to all class prototypes:

Note that the negative sign is necessary since we want to increase the probability for close-
by vectors and have a low probability for distant vectors. We train the network based on
the cross-entropy error of the training query set examples. Thereby, the gradient flows
through both the prototypes and the query set encodings . For the distance function
, we can choose any function as long as it is differentiable concerning both of its inputs.
The most common function, which we also use here, is the squared euclidean distance, but
there have been several works on different distance functions as well.
MODULE 1 : Graph Machine Learning
What is Graph Machine Learning?
 Graph machine learning is a subfield of machine learning that focuses on
using graph-structured data to perform predictive and analytical tasks.
 In graphs, data is represented as nodes (vertices) and edges (links), capturing
complex relationships and interactions between entities.
 Unlike traditional data, which is often tabular, graph data is inherently
interconnected, making it suitable for modeling social networks, biological
networks, communication networks, and more.

Example 1 :
Example 2:

Example 3:
 Graph machine learning leverages algorithms designed to exploit these
connections, such as Graph Neural Networks (GNNs), to learn from the
structure and attributes of graphs. These algorithms can perform tasks like
node classification, link prediction, and graph clustering.

Node-level → predict something about each node

Edge-level → predict relationships between nodes

 Graph-level → predict something about the entire graph

Node-level task
Node-level tasks are concerned with predicting the identity or role of each node
within a graph. A classic example of a node-level prediction problem is Zach’s karate
[Link] dataset represents one social network of people who are members of a
karate club.
After a disagreement, the club split into two groups. One group supported Mr. Hi
(the instructor) and the other supported John H (the administrator).
 Each node represents a karate club member
 Each edge shows social interactions between members outside karate
Because of the conflict, every member chose one [Link] task is to predict which
group a person will [Link] this dataset, a person is more likely to support the
leader they are socially closer [Link], the distance between a member and each
leader in the network helps determine their loyalty.
Input: graph with unlabled nodes
Allegiance to John A Allegiance to Mr. Hi→Mr. HiJohn A
Output: graph node labels
On the left we have the initial conditions of the problem, on the right we have a
possible solution, where each node has been classified based on the alliance. The
dataset can be used in other graph problems like unsupervised learning.
Edge-level task
Another type of prediction problem in graphs is edge prediction. Edge
prediction means finding whether a connection exists between two
nodes. For example, in image understanding, deep learning models not only
detect objects in an image, but also try to understand how those objects are
related (such as “person riding a bike” or “cup on a table”).

In this case:
 Nodes represent objects in the image
 Edges represent relationships between objects

The task is to predict whether an edge should exist between two objects
and what type of relationship it is. To find unknown connections, we can
first assume that every object is connected to every other object, and then
remove weak or unimportant connections, leaving only meaningful
relationships.
In (b), above, the original image (a) has been segmented into five entities:
each of the fighters,
Graph-level task
In a graph-level task, our goal is to predict the property of an entire graph. For
example, for a molecule represented as a graph, we might want to predict what the
molecule smells like, or whether it will bind to a receptor implicated in a disease.
Input: graphs
Output: labels for each graph, (e.g., "does the graph contain two rings?")
This is analogous to image classification problems with MNIST and CIFAR, where
we want to associate a label to an entire image. With text, a similar problem is
sentiment analysis where we want to identify the mood or emotion of an entire
sentence at once.
  By learning from the topology and features of graphs, these models can
uncover hidden patterns, predict future connections, and identify influential
nodes within the network. The ability to handle and analyze graph data opens
up new possibilities for solving complex problems in various domains, from
recommending friends on social media to predicting molecular properties in
drug discovery.
How does graph machine learning work?
Graph machine learning works by leveraging the unique structure and relationships
inherent in graph data to perform various predictive and analytical tasks. Here’s a
breakdown of the process:
 Graph Representation: Data is represented as a graph, consisting of nodes
(representing entities) and edges (representing relationships). Each node and
edge can have associated features, providing additional context.
 Feature Extraction: Features from nodes, edges, and the overall graph
structure are extracted. This can include node attributes (e.g., user profiles in
social networks) and structural features (e.g., node degree, centrality
measures).
 Graph Neural Networks (GNNs): GNNs are the primary models used in graph
machine learning. They extend traditional neural networks to operate on
graph data by aggregating information from a node’s neighbors. This process,
called message passing or neighborhood aggregation, allows the model to
learn representations that capture both local and global graph structures.
 Propagation and Aggregation: During training, each node’s feature vector is
updated by combining its own features with aggregated features from its
neighbors. This iterative process enables nodes to gather information from
multi-hop neighborhoods, enriching their representations.
 Learning and Optimization: The GNN is trained using standard machine
learning techniques, such as backpropagation and gradient descent. The
objective is to minimize a loss function tailored to the specific task, like node
classification, link prediction, or graph classification.
 Prediction and Inference: Once trained, the model can be used to make
predictions. For example, in node classification, the model predicts the label
of each node based on its learned representation. In link prediction, the model
estimates the likelihood of new or missing edges. Graph machine learning
models excel in capturing the complex dependencies and patterns within
graph-structured data, making them powerful tools for applications ranging
from social network analysis to drug discovery.

Challenges in Graph Learning (Scalability, Sparsity and Over-Smoothing)

Traditional deep learning models (CNNs, RNNs, Transformers):
 Expect fixed-size, rectangular tensors
 Examples:
o Images → 2D grids (H × W × C)
o Text → sequences of tokens
o Tables → rows × columns
Graphs don’t fit this mold because:
 They are irregular
 Size varies (number of nodes and edges changes)
 No natural ordering of nodes
So the first challenge is:
How do we represent graphs so neural networks can process them?

What information exists in a graph?

A graph can contain four different types of information:

Component Meaning Example

Nodes Entities People, atoms, users

Edges Relationships Friendships, bonds

Global Graph label,

Whole-graph info
context environment

Who is connected to
Connectivity Graph structure
whom

The first three are easy to store.

The fourth (connectivity) is the hard one.
Representing node, edge, and global features (easy part)
🔹 Node features
We assign each node an index and store features in a matrix:

Example:
 Each row → one node
 Each column → one feature (age, embedding, type)
Even though varies across graphs: Neural networks can handle this easily
🔹 Edge features
Similarly:
Each row stores attributes of one edge (weight, type, distance).
🔹 Global features
One vector per graph:

Example:
 Molecule energy
 Scene category
So far, no problem.

The real problem: representing connectivity

Option 1: Adjacency matrix
An adjacency matrix where:
Problems with adjacency matrices
(a) Sparsity
 Real graphs may have millions of nodes
 Adjacency matrix size =
 Most entries are zeros → extremely sparse

💥 Huge waste of memory and computation

(b) Permutation sensitivity

Graphs do not have a fixed node order.
Reordering nodes:
 Changes the adjacency matrix
 Does not change the graph
But:
A neural network will see different matrices as different inputs, This breaks
permutation invariance
c. Scalabilty
Real-world graphs are huge:
 Social networks → millions/billions of users
 Web graphs → billions of pages
 Knowledge graphs → millions of entities
 Road networks → millions of intersections
If a method:
 Works only for small graphs
 Runs out of memory or becomes too slow for big graphs
It is not scalable.
d. Oversmoothing
Over-smoothing happens when node representations become too similar to each
other as we stack many GNN layers.
After enough layers:
 Nodes lose their individual identity
  Features of different nodes become nearly identical
 The model can no longer distinguish nodes → poor accuracy
What are Graph Neural Networks?
Graph Neural Networks (GNNs) are deep learning models designed to work with
graph-structured data, where information is represented as nodes and edges. Unlike
traditional neural networks that handle fixed-size inputs, GNNs capture relationships,
dependencies and interactions between entities.
 They operate on graphs made of nodes and edges.
 Information is passed between connected nodes (neighbors) through
message-passing steps.
 Useful for tasks like social network analysis, molecule prediction and
recommendation systems.
 They learn both node-level and graph-level patterns.

This image shows how a GNN processes a graph node features pass through
stacked graph convolution layers with regularization, gradually refining
representations until the model outputs predictions such as the probability of links
between nodes.
GNN layers, followed by a linear model with a sigmoid activation for classification.
The design space for our GNN has many levers that can customize the model:
1. The number of GNN layers, also called the depth.
2. The dimensionality of each attribute when updated. The update function is a
1-layer MLP with a relu activation function and a layer norm for normalization
of activations.
3. The aggregation function used in pooling: max, mean or sum.
4. The graph attributes that get updated, or styles of message passing: nodes,
edges and global representation. We control these via boolean toggles (on or
 off). A baseline model would be a graph-independent GNN (all message-
passing off) which aggregates all data at the end into a single global attribute.

GNN Predictions by Pooling Information

Message passing works in three steps:
1. For each node in the graph, gather all the neighboring node embeddings (or
messages), which is the g function described above.
2. Aggregate all messages via an aggregate function (like sum).
3. All pooled messages are passed through an update function, usually a
learned neural network.

 Node Prediction
 Edge Prediction
 Graph Prediction
Node :
We will consider the case of binary classification, but this framework can easily be
extended to the multi-class or regression case. If the task is to make binary
predictions on nodes, and the graph already contains node information, the
approach is straightforward — for each node embedding, apply a linear classifier.

However, it is not always so simple. For instance, you might have information in the
graph stored in edges, but no information in nodes, but still need to make predictions
on nodes. We need a way to collect information from edges and give them to nodes
for prediction. We can do this by pooling. Pooling proceeds in two steps:
1. For each item to be pooled, gather each of their embeddings and concatenate
them into a matrix.
2. The gathered embeddings are then aggregated, usually via a sum operation.
For a more in-depth discussion on aggregation operations go to the Comparing
aggregation operations section.
We represent the pooling operation by the letter ρ, and denote that we are gathering
information from edges to nodes as pEn→Vn.
Edge :
If we only have node-level features, and are trying to predict binary edge-level
information, the model looks like this.
Graph:
If we only have node-level features, and need to predict a binary global property, we
need to gather all available node information together and aggregate them. This is
similar to Global Average Pooling layers in CNNs. The same can be done for edges.

Message passing works in three steps:

1. For each node in the graph, gather all the neighboring node embeddings (or
messages), which is the g function described above.
2. Aggregate all messages via an aggregate function (like sum).
3. All pooled messages are passed through an update function, usually a
learned neural network.
[Link]
[Link]
architectural-insights-and-applications/

MODULE 1 :Graph Convolutional Networks (GCNs)

Graph Convolutional Networks (GCNs) have emerged as a powerful class of deep learning
models designed to handle graph-structured data. Unlike traditional Convolutional
Neural Networks (CNNs) that operate on grid-like data structures such as images, GCNs are
tailored to work with non-Euclidean data, making them suitable for a wide range of
applications including social networks, molecular structures, and recommendation systems.

What Are Graph Convolutional Networks?

Graph Convolutional Networks (GCNs) are a type of neural network designed to work
directly with graphs. A graph consists of nodes (vertices) and edges (connections between
nodes). In a GCN, each node represents an entity, and the edges represent the relationships
between these entities. The primary goal of GCNs is to learn node embeddings, which are
vector representations of nodes that capture the graph's structural and feature information.
Architecture of GCNs
GCNs typically consist of multiple layers, each responsible for refining node embeddings by
aggregating information from neighbors at increasing distances. The layers are:

1. Input Layer: The input layer initializes the node features, usually from raw data or
pre-trained embeddings.
1. Hidden Layers: Hidden layers perform the graph convolution operations,
progressively aggregating and transforming node features.
 A. Graph Convolutional Layers: These layers perform the convolution
operation on the graph. Each layer updates the feature representation of a
node by aggregating the features of its neighbors.
 B. Activation Functions: Non-linear functions such as ReLU are applied to
the output of each convolutional layer to introduce non-linearity into the
model.
 C. Pooling Layers: These layers reduce the dimensionality of the graph by
merging nodes, which helps in capturing hierarchical structures.
1. Output Layer: The output layer produces the final node embeddings or predictions,
depending on the task (e.g., node classification, link prediction).
1. Fully Connected Layers: These layers are used at the end of the network to
perform tasks such as classification or regression.
Types of Graph Convolutional Networks (GCNs)
GCNs can be broadly categorized into two types: Spectral-based and Spatial-based GCNs.
1. Spectral-based GCNs
Spectral-based GCNs are defined in the spectral domain using the graph Laplacian and
Fourier transform. The convolution operation is performed by multiplying the graph signal
with a filter in the spectral domain. This approach leverages the eigenvalues and
eigenvectors of the graph Laplacian to perform convolution.
Key Models:
 ChebNet: Uses Chebyshev polynomials to approximate the graph convolution
operation, allowing for efficient computation on large graphs.
 GCN (Kipf & Welling): Simplifies the spectral convolution by using a first-order
approximation, making it computationally efficient and scalable.
2. Spatial-based GCNs
Spatial-based GCNs perform convolution directly in the spatial domain by aggregating
features from neighboring nodes. This approach is more intuitive and easier to implement
compared to spectral-based methods. Key Models:
 GraphSAGE: Aggregates features from a fixed-size set of neighbors using mean,
LSTM, or pooling functions.
 GAT (Graph Attention Network): Introduces an attention mechanism to assign
different weights to the neighbors of a node based on their importance.

How Graph Convolutional Networks (GCNs) Work?

1. Graph Representation
 Nodes and Edges: Represent entities (nodes) and relationships (edges) between
them using a graph ,where V is the set of nodes and E is the set of edges.
 Node Features: Each node has associated features, which could be initial
attributes (e.g., text, images) or learned embeddings.
A graph G is represented by:
 A set of nodes
 A set of edges
 An adjacency matrix , where indicates the presence (and sometimes the weight) of
an edge between node and node .
2. Convolution Operation on Graphs
In GCNs, the convolution operation is adapted to work on graphs. The key idea is to
aggregate information from a node's neighbors to update its representation.
This process is analogous to the convolution operation in CNNs, which aggregates
information from neighboring pixels.
3. Mathematical Formulation
The core operation in a GCN layer can be described by the following equation:

where:
 is the matrix of node features at layer
 is the normalized adjacency matrix
 is the trainable weight matrix at layer
 σ is an activation function, such as ReLU
4. Normalization
Normalization of the adjacency matrix is crucial to ensure numerical stability and improve
model performance. A common normalization technique is:

where ? is the degree matrix.

5. Training and Learning
 Backpropagation: GCNs are trained using gradient-based optimization methods
(e.g., stochastic gradient descent) to minimize a loss function, typically tailored to the
specific task (classification, regression, etc.).
 End-to-End Learning: The entire network, including convolutional layers and
subsequent fully connected layers, is trained jointly to optimize performance on the
task.
6. Output
 Node Embeddings: After several layers of graph convolution, the final node
representations are used for downstream tasks like node classification or link
prediction.
 Graph Embeddings: For graph-level tasks, additional aggregation or pooling over
node embeddings can yield a single representation for the entire graph.
Training Graph Convolutional Networks (GCNs)
 Loss Functions: Training GCNs involves optimizing a loss function appropriate for
the specific task. Common loss functions include cross-entropy loss for classification
tasks and mean squared error for regression tasks
 Optimization: GCNs are trained using gradient-based optimization techniques such
as stochastic gradient descent (SGD) or Adam. The gradients are computed through
backpropagation, taking into account the graph structure.
 Regularization: To prevent overfitting, regularization techniques such as dropout and
weight decay are applied. Dropout involves randomly setting a fraction of the node
features to zero during training, while weight decay adds a penalty to the loss
function based on the magnitude of the weights.

MODULE 1 :GRAPH ATTENTION NETWORKS (GAT)

Definition
A Graph Attention Network (GAT) is a type of neural network that uses attention
mechanisms to process information in graph-structured data. GATs enable nodes in a graph
to weigh the importance of their neighbors’ features during aggregation, improving
representation learning.
Why It Matters: Enterprises handle complex data with inherent relationships, such as social
networks, communication systems, or supply chains. GATs help uncover insights by enabling
more nuanced information flow in tasks like node classification, link prediction, and
clustering. Their ability to dynamically focus on relevant neighboring nodes enhances model
interpretability and prediction accuracy. This can lead to more informed business decisions
and stronger models for fraud detection, recommendation systems, and infrastructure
monitoring. However, GATs can introduce computational overhead and require careful
resource management for large-scale deployments.

Key Characteristics:
GATs use a self-attention mechanism that learns to assign different importance to each
neighbor in a graph, making them more adaptable to varied graph structures. They operate
directly on graph data without needing to define explicit neighborhood rules, which lends
flexibility across domains. GATs are often more robust to noisy or incomplete data compared
to traditional graph neural networks. Hyperparameters include the number of attention
heads, aggregation functions, and hidden layer sizes. Their performance depends on graph
size and network complexity, and training may be sensitive to data order and initializations.
Working Principle of Graph Attention Networks (GAT)
A Graph Attention Network (GAT) processes data represented as a graph, where entities are
nodes and relationships are edges. The input consists of node feature matrices and an
adjacency matrix describing connections between nodes. Each node’s features are
initialized with contextual information, such as attributes or embeddings relevant to the
[Link] core mechanism involves attention layers. For each node, the network
computes attention coefficients for its neighbors based on their features and connectivity.
These coefficients determine the extent to which a node aggregates information from each
connected node, allowing the model to focus more on relevant neighbors. The process is
typically repeated across multiple layers, with key parameters including the number of
attention heads, activation functions, and layer depth. Weights are learned during training
using backpropagation to optimize a loss function appropriate for the task, such as node
classification or link [Link] attention-based aggregation, the output is a set of
updated node representations that capture both the original features and the influence of
connected nodes. These outputs can be used directly for downstream tasks or passed to
additional layers or modules depending on the system design and operational requirements.
Constraints such as fixed graph schemas or edge sparsity are addressed during
preprocessing and model configuration to ensure efficient computation and compliance with
enterprise data policy.
Steps:
1. Linear Transformation
Each node in the graph has an input feature vector .
To project these features into a higher-level representation space, a shared learnable
weight matrix is applied to all nodes:

This step ensures that all node features are in the same feature space before computing
attention and allows the model to learn task-specific representations.
2. Attention Mechanism
For every node , attention scores are computed with each of its neighboring nodes .
The transformed features of nodes and are concatenated and passed through a learnable
attention function:

This score reflects how important node is to node .

Unlike GCNs, this importance is learned dynamically rather than being fixed.

3. Normalization (Softmax)

Since node may have many neighbors, the raw attention scores are normalized using the
Softmax function:

This ensures:
 Attention weights are positive
 The sum of attention coefficients for a node equals 1
 More important neighbors receive higher weights
4. Aggregation
Each node updates its representation by aggregating information from its neighbors using
the normalized attention coefficients:

Here, neighbors with higher attention weights contribute more to the updated node
[Link] activation function (such as ELU or ReLU) introduces non-linearity.
5. Multi-Head Attention
Instead of using a single attention mechanism, GAT employs multiple attention heads.
Each head learns independent attention coefficients:
Benefits of Multi-Head Attention:
 Improves model stability
 Captures different types of relationships
 Reduces variance in attention learning
In hidden layers, outputs are concatenated; in the final layer, they are typically averaged.