Semi-Supervised
Learning
Introduction
Today’s Machine Learning algorithms can
be broadly classified into three categories,
Supervised Learning, Unsupervised Learnin
g
and Reinforcement Learning.
The primary two categories of Machine
Learning problems are Supervised and
Unsupervised Learning.
The basic difference between the two is
that Supervised Learning datasets have an
output label associated with each tuple
while Unsupervised Learning datasets do
Continue..
The most basic disadvantage of
any Supervised Learning algorithm is
that the dataset has to be hand-labeled
either by a Machine Learning Engineer or a
Data Scientist. This is a very costly process,
especially when dealing with large volumes
of data. The most basic disadvantage of
any Unsupervised Learning is that
it’s application spectrum is limited.
To counter these disadvantages, the
concept of Semi-Supervised
Learning was introduced.
Semi –Supervised Learning
Semi-supervised learning is an approach
to machine learning that combines a small
amount of labeled data with a large amount
of unlabeled data during training.
Semi-supervised learning falls
between unsupervised learning (with no
labeled training data) and supervised
learning (with only labeled training data).
Unlabeled data, when used in conjunction
with a small amount of labeled data, can
produce considerable improvement in
learning accuracy.
Continue…
Intuitively, one may imagine the three types of learning algorithms as
Supervised learning where a student is under the supervision of a
teacher at both home and school, Unsupervised learning where a
student has to figure out a concept himself and Semi-Supervised
learning where a teacher teaches a few concepts in class and gives
questions as homework which are based on similar concepts.
Continue..
A Semi-Supervised algorithm assumes the following
about the data –
Continuity Assumption: The algorithm assumes
that the points which are closer to each other are
more likely to have the same output label.
Cluster Assumption: The data can be divided
into discrete clusters and points in the same cluster
are more likely to share an output label.
Manifold Assumption: The data lie approximately
on a manifold of much lower dimension than the
input space. This assumption allows the use of
distances and densities which are defined on a
manifold.
Practical Applications
Speech Analysis: Since labeling of audio files is a
very intensive task, Semi-Supervised learning is a
very natural approach to solve this problem.
Internet Content Classification: Labeling each
webpage is an impractical and unfeasible process
and thus uses Semi-Supervised learning algorithms.
Even the Google search algorithm uses a variant of
Semi-Supervised learning to rank the relevance of a
webpage for a given query.
Protein Sequence Classification: Since DNA
strands are typically very large in size, the rise of
Semi-Supervised learning has been imminent in this
field.
SSL Algorithms
1. Self-Training
2. Graph-based methods
3. Co-Training
4. Semi-supervised SVM
Self-training
Graph Based Methods
Graph-based methods for semi-supervised learning
use a graph representation of the data, with a
node for each labeled and unlabeled example.
Co-training
Co-training Algorithm
Co-training (Blum & Mitchell, 1998) (Mitchell, 1999) assumes that
(i) features can be split into two sets;
(ii) each sub-feature set is sufficient to train a good classifier.
Initially two separate classifiers are trained with the labeled
data, on the two sub-feature sets respectively.
Each classifier then classifies the unlabeled data, and ‘teaches’
the other classifier with the few unlabeled examples (and the
predicted labels) they feel most confident.
Each classifier is retrained with the additional training examples
given by the other classifier, and the process repeats.
Co-training first learns a separate classifier for each view using
any labeled examples. The most confident predictions of each
classifier on the unlabeled data are then used to iteratively
construct additional labeled training data.
Co-training Algorithm