Instance Based Learning/Memory based Learning/Lazy Lear

* Instance based learning includes nearest neighbor

and locally weighted regression methods that
assume instances can be represented as points in a
Euclidean space. It also includes case-based
reasoning methods that use more complex,
symbolic representations for instances.
* Instance-based methods are sometimes referred to a
as "lazy" learning methods because they delay
processing until a new instance must be classified.
A key advantage of this kind of delayed, or lazy,
lcarning is that instcad of estimating the target Vst v | enrming Memmrs b | armins 403
function once for the entire instance space, these 1
methods can estimate it locally and differently for
cach new instance to be classified.
Instance Based Learning/Memory based Learning/Lazy Learnin l

* Instance-based learning methods such as ncarest

neighbor and locally weighted regression are
conceptually Supervised straightforward
approaches to approximating real-valued or
discrete valued target functions. Learning in these
algorithms consists of simply storing the presented
training data. When a new query instance is
encountered, a sct of similar related instances is
retrieved from memory and used to classify the
new query instance.
 k-Nearest Neighbour Learning '
* The most basic instance-based method is the k-Nearest
Neighbor algorithm. This algorithm assumes all instances
correspond to points in the n-dimensional space. The
ncarest neighbors of an instance are defined in terms ofthe
standard Euclidean distance.
* The K-Nearest Neighbors (K-NN) algorithm is a simple,
yet powerful supervised learning method often used for
classification and regression tasks. Here's an overview of
the K-NN leamning process, key steps, and considerations
1. Overview of K-NN Learning
* K-NN is a non-parametric and instance-based learning
algorithm. It doesn’t make any assumptions about the da*~" P
underlying distribution.
* In K-NN, the algorithm classifics a new data point basec
the majority class (in classification) or the average value
regression) of its closest neighbors.
* The "K" in K-NN is a parameter that represents the numr
ofnearest neighbors to consider in making a decision.
 k-Nearest Neighbour Learningcona...
* K-Nearest Neighbour is one ofthe simplest Machine Learning
algorithms based on Supervised Learning technique.
* KNN algorithm assumes the similarity between the new
casc/data and available cases and put the new case into the
category that is most like the available categories.
* KNN algorithm can be used for Regression as well as for
Classification but mostly it is used for the Classification
problems.
* KNN is a non-parametric algorithm, which means it docs not
make any assumption on underlying data.
* It is also called a lazy learner algorithm because it does =~* iN B
learn from the training set immediately instead it stores
dataset and at the time of classification, it performs an act
on the dataset.
* KNN algorithm at the training phase just stores the dataset
when it gets new data, then it classifies that data into a categ
that is much similar to the new data.
 k-Nearest Neighbour Learningcon...)
* Example: Suppose, we have
an im of a creature that
looks similar to cat and dog,
but we want to know cither it KNN Classifier
is a cat or dog. So for this
identification, we can use the
KNN algorithm, as it works
on a similarity measure. Qur
KNN model will find the
similar features of the new
data set to the cats and dogs
images and based on the
most similar features it will Input value
put it in cither cat or dog
category.
 Why do we need a K-NN Algorithm?

Suppose there are two

categories, i.c., Category A
and Catcgory B, and we
have a new data point x1, so
this data point will lic in
which of these categories. To
solve this type of problem,
we need a K-NN algorithm.
With the help of K-NN, we
can easily identify the
category or class of a
particular dataset. Consider
the below diagram:
 k-Nearest Neighbour Learning Algorithm Steps
1. Load the data
2. Initialize K to your chosen number of neighbors
3. For cach example in the data
3.1 Calculate the distance between the query example and the
current example from the data, Euclidean Distance between two
points= V(x2-x1)2+(y2-y1)2
3.2 Add the distance and the index of the example to an
ordered collection
4. Sort the ordered collection of distances and indices from
smallest to largest (in ascending order) by the distances
5. Pick the first K entrics from the sorted collection New

6. Gt the labels of the selected K entrics

7. If regression, return the mean of the K labels
8. If classification, return the mode of the K labels
 Pros and Cons of KNN
Pros
* Instead of estimating for the entire instance set, local
approximations can be made to the target function.
* Itis very simple algorithm to understand and interpret.
* Itis very useful for nonlinear data because there is no assumption
about data in this algorithm.
* Itis a versatile algorithm as we can use it for classification as well
as regression,
¢ It has relatively high accuracy but there are much better
supervised lcaming models than KNN.

Cons
* Itis computationally a bit expensive algorithm because it stores all
the training data.
* High memory storage required as compared to other supervised
leaming algorithms.
* Prediction is slow in case of big N.
* It is very sensitive to the scale of data as well as irrclevant
features,
 Applications of KNN
The following are some of the arcas in which KNN can be applied
successfully —
eBanking System- KNN can be used in banking system to predict
whether an individual is fit for loan approval? Does that individual have
the characteristics similar to the defaulters one?

ePolitics- With the help of KNN algorithms, we can classify a potential

voter into various classes like “Will Vote™, “Will not Vote™, “Will Vote to
Party *Congress’, “Will Vote to Party *BJP",
Other arcas in which KNN algorithm can be used arc Speech
Recognition, Handwriting Detection, Image Recognition and Video
Recognition.
 LOCALLY WEIGHTED REGRESSION e:'
* Locally weighted regression is instance based learning algorithm.

* The phrase "locally weighted regression" is called

’
- local because the function is approximated based only on data near the query
point,

— welghted because the contribution of each training examole is weighted by its

distance from the query point, and

— regression because this is the term used widelyinth.

community for the problem of approximating real-va
Algorithm Steps
1. Select the point where you want to make a prediction (the query point z,).

2. Compute weights for each training point with respect to 7, based on the distance from z,.

3. Fitaweighted linear regression mode! using these weights.

4. Predict the output for 7, using the local model.

Advantages
o Flexibility: Allows the model to capture non-linear relationships in data.

o Non-parametric Does not assume a specific global form for the data relationship, making it
robust to various data structures. S

Disadvantages
¢ Computationally Intensive: LWR requires fitting a new regression for every prediction, whi
can be slow for large datasets.
4
Choice of Bandwidth (7). Setting the banuwidth is crucial, as it affects the modef's ability1
 Radial Basis Function Networks

Radial Basis Function Networks (RBFNs) arc closcly

related to instance-based learning, where predictions are
made based on the similarity between new input data and
instances in the training set. In RBFNSs, this similarity-
based approach is reflected in the radial basis function
used in the hidden layer, which measures how close a
new input is to a set of predefined centers,

How RBFNs Work as Instance-Based Learners

[Link] as Instances: In RBFNs, centers of radial
basis functions serve as “prototypes™ or representative
instances in the feature space. These can be selected
using clustering techniques like k-means, cffectively
summarizing the training data with a fixed number of
instances.
How RBFNs Work as Instance-Based Learnerscon...) .

2, Distance and Similarity: Each radial basis

function computes a response based on the distance
between a new input and a center. Typically, a
Gaussian function is used, where closer inputs yield
higher outputs , indicating greater similarity. This
mirrors the core idea of instance-based leaming,
where similarity to stored instances (or prototypes)
determines the output.

3. Localized Influence: Each hidden layer unit

(RBF) has a “localized™ influence, meaning that it ([T TTENIY e| vars
responds strongly only to inputs within a certain
range (controlled by the RBF's spread parameter).
This local receptive field property resembles
instance-based leaming methods, like k-ncarest
neighbors (k-NN), which focus on nearby points,
Advantages of RBFNs in Instance-Based Learning Context
Efficient Representation: RBFNs reduce the need to
store all training examples by summarizing them into
a smaller set of centers, unlike traditional instance-
based methods that may store every example (c.g., k-
NN).
Flexibility in Approximation: With proper choice of
centers and widths, RBFNs can approximate complex
functions well by leveraging local information around
cach input point.
Differences from Traditional Instance-Based
Learning
Learning Phase: While instance-based methods like s amtages of KD NG bn Dastanes Based
| easmen © unte
k-NN don’t usually have a “leaming™ phase, RBFNs
involve training, especially in optimizing the output
layer weights.
Fixed Centers: RBFNs rely on a fixed sct of centers
and don’t dynamically adapt them for each prediction,
unlike k-NN, which recalculates neighbors on-the-fly.
LECTURE ON MAC!
 Case-based learning (CBL)
* Case-based learning is a machine learning method in
which a system lcams from solving previous cases to
the current task. In this method the system uses a case
base that contains several previously solved cases that
are like the current task.
* The CBL process consists of three phases: retrieval,
adaptation and evaluation. In the retrieval phase, the ¢
system scarches for similar cascs in the database. In
adaptation phase, the system modifics the solution of the °
previous case 1o fit the current task. In the evaluation
phase, the system evaluates the proposed solution and
compares it with the optimal solution.
* Casc-based learning is used in variety of applications
such as medical diagnostic problem solving, pattern
recognition, decision making, task planning, among
others.
 Case-based learning (CBL) D
¢ Instance-based learning (IBL) is a family of algorithms that
primarily rely on storing and using specific ‘instances of data for
prediction. It"does not construct a general model upfront; instead, it
defers learning until a query is made.
¢ Case-based learning (CBL) is a specific type of instance-based
learing that leverages previously encounteréd cases 1o solve new
problems.
Key Concepts of Case-Based Learning: ¢
1. Core ldea:
1. CBL operates on the principle of “solving a new problem by
adapting solutions from simitar past problems.”
2. Each case consists of a problem description and its
corresponding solution.
2. How it Differs from General 1BL:
1. Instance-Based Learning focuses on c«_unparir:]g the input
‘(qucry) to stored instances and makes predictions directly (e.g.,
-Nearest Neighbors).
2, Case-Based Learning involves more reasoning. It retrieves
cascs from the case library, adapts their solutions to fit the new
problem, and can update its case database with new
experiences.
 Process of Case-Based Learning
1. Case Storage:
1. Cases are stored in a structured case base. A case typically includes:
1. Problem: Description of the input conditions.
2, Solution: The solution or outcome corresponding to that problem.
3. Outcome: Results from applying the solution.
2, Retrieval:
1. Given a new problem, the system retrieves the most similar case(s)
from the case base.
2. Similarity measures (e.g., Euclidean distance, domain-specific
heuristics) are often used.
3. Reuse/Adaptation:
1. Once a similar case is retricved, its solution may need to be adapted to
the specifics of the new problem.
2. Adaptation can range from simple modifications to more complex
transformations. Proneve
ot s Hasedd
| easaing
4. Evaluation:
1. The proposed solution is tested, and feedback is collected to evaluat
its effecliveness in the new scenario,
5. Retention:
1. If the solution is deemed successful, the new case (problem an
solution) can be added to the case base for future reference.
 Example of Case-Based Learning D
Medical Diagnosis:
* A case-based leaning system could store cases of past
patients, including symptoms and diagnoses.
* When a new ent presents symptoms, the system
retrieves similar cases, adapts the diagnosis, and suggests a
treatment plan,
* I new symptoms are presented, it updates the case base.

Advantages:
1. Explainability: Solutions are based on actual past cases,
which can be casier to explain and justify,
2. Flexibility: Can adapt 10 new data incrementally.
3. Domain-Driven: Uscful in domains where expert knowledge
is encoded in historical cases (e.g., law, medicine),

Limitations
1. ifl;(orngc Requirements: Needs to store potentially large case

2. Efficiency: Retrieval and adaptation processes can be

computationally expensive,
3. Overfitting to Instances; Without abstraction, models may
fail to generalize cffectively.