ELECTRONICS AND COMMUNICATION ENGINEERING
8TH SEMESTER
1) Introduction: Basic definitions, types of learning, hypothesis space and inductive bias, evaluation, cross-validation. Linear
regression, Decision trees, over fitting
8 Hrs.
2) Instance based learning, Feature reduction, Collaborative filtering based recommendation. Probability and Bayes learning.
8 Hrs.
3) Logistic Regression, Support Vector Machine, Kernel function and Kernel SVM
8 Hrs.
4) Neural Network: Perceptron, multilayer network, back propagation, introduction to deep neural network
8 Hrs.
5) Computational learning theory, PAC learning model, Sample complexity, VC Dimension, Ensemble learning. Clustering: k-
means, adaptive hierarchical clustering, Gaussian mixture model.
8 Hrs.
INSTANCE BASED
LEARNING
Instance-based learning (IBL) is a type of machine learning approach that relies
on storing and using training instances to make predictions, rather than
creating a general model from the data
This category includes algorithms such as
(1) k-Nearest Neighbors (k-NN),
(2) Locally Weighted Learning (LWL),
(3) Case-Based Reasoning (CBR).
CHARACTERISTICS OF INSTANCE-BASED LEARNING
(1) Memory-Based: Instead of explicitly learning a function during training, these
models retain the training examples and use them to make predictions at query
time.
(2) Lazy Learning: Learning is delayed until a prediction is needed, making
instance-based learning computationally efficient during training but potentially
expensive during inference.
(3) Sensitive to Noise: Since all data points are stored, noisy or irrelevant data
can affect predictions significantly.
k-Nearest Neighbors (k-NN)
It classifies a new data point by finding the k most similar points from the training
dataset and using them to determine the label of the new instance.
Step : 1: Choose the Value of k:
• The user selects the number of nearest neighbors (k).
• A small k makes the model sensitive to noise, while a large k makes it more
general but may include irrelevant data points.
Step : 2: Calculate Distance:
The algorithm computes the distance between the query instance and each
stored instance. Common distance metrics include:
k-Nearest Neighbors (k-NN)
It classifies a new data point by finding the k most similar points from the training
dataset and using them to determine the label of the new instance.
Step : 3: Identify k Nearest Neighbors: The k instances closest to the query
instance are selected.
Step : 4: Make a Prediction:
• For Classification: A majority vote determines the class label.
• For Regression: The average of the k-nearest neighbors' values is used as the
predicted output.
ELECTRONICS AND COMMUNICATION ENGINEERING
8TH SEMESTER
1) Introduction: Basic definitions, types of learning, hypothesis space and inductive bias, evaluation, cross-validation. Linear
regression, Decision trees, over fitting
8 Hrs.
2) Instance based learning, Feature reduction, Collaborative filtering based recommendation. Probability and Bayes learning.
8 Hrs.
3) Logistic Regression, Support Vector Machine, Kernel function and Kernel SVM
8 Hrs.
4) Neural Network: Perceptron, multilayer network, back propagation, introduction to deep neural network
8 Hrs.
5) Computational learning theory, PAC learning model, Sample complexity, VC Dimension, Ensemble learning. Clustering: k-
means, adaptive hierarchical clustering, Gaussian mixture model.
8 Hrs.
FEATURE
REDUCTION
Feature reduction, also known as dimensionality reduction, is a crucial
preprocessing step in machine learning that involves reducing the number of input
variables (features) while retaining essential information.
Feature reduction techniques are broadly classified into Feature Selection and
Feature Extraction.
Scenario: Predicting Employee Attrition
A company wants to predict employee attrition using features like Age, Salary, Work
Hours, Employee ID, Performance Rating, and Job Satisfaction.
[Link] Selection:
1. Removing Employee ID (irrelevant feature)
2. Checking correlation: If Work Hours and Salary have high correlation, we
keep only one.
3. Using LASSO to shrink insignificant feature coefficients.
2. Feature Extraction:
1. Applying PCA to combine Job Satisfaction and Performance Rating into a
single component representing Overall Employee Well-being.
By reducing features, the model is more efficient and interpretable without losing
predictive power.
FEATURE SELECTION TECHNIQUES
Feature selection methods aim to identify the most important features from the
dataset while eliminating redundant or irrelevant ones. There are three main types:
(1) Filter Methods
These methods assess the statistical relevance of features without using any machine
learning models
• Correlation Coefficient: Measures the relationship between each feature and the
target variable.
• Chi-Square Test: Used for categorical data to check feature relevance.
• Variance Threshold: Removes features with low variance, assuming they contribute
less information.
FEATURE SELECTION TECHNIQUES
Example:
Suppose we have a dataset predicting whether a person has diabetes with features
Age, Height, Weight, and ID Number.
The ID Number is unique but not relevant to the prediction, so it should be removed.
Additionally, if Height and Weight are highly correlated, one may be dropped.
FEATURE SELECTION TECHNIQUES
(2) Wrapper Methods
Wrapper methods use machine learning models to evaluate the importance of features
by training different subsets and comparing their performance.
• Forward Selection: Starts with no features and adds them one by one based on
model improvement.
• Backward Elimination: Starts with all features and removes them one by one based
on performance degradation.
• Recursive Feature Elimination (RFE): Recursively removes features to identify the
most significant ones.
FEATURE SELECTION TECHNIQUES
Example:
Suppose we are training a model to predict house prices based on Location, Square
Footage, Number of Bedrooms, and Number of Bathrooms.
Using RFE, we may find that Number of Bedrooms and Number of Bathrooms
contribute similarly, so one can be removed without significantly affecting
performance.
FEATURE SELECTION TECHNIQUES
(3) Embedded Methods
These methods perform feature selection during model training by evaluating feature
importance.
• LASSO Regression (L1 Regularization): Shrinks coefficients of less important features
to zero.
• Decision Trees and Random Forest: Assign importance scores to features.
FEATURE SELECTION TECHNIQUES
Example:
A bank uses a decision tree model to predict whether a customer will default on a loan.
The model assigns high importance to Credit Score and Income, while Zip Code and
Marital Status receive lower scores and may be removed.
FEATURE EXTRACTION TECHNIQUES
Feature extraction involves transforming the original features into a new set of features
that retain essential information.
(1) Principal Component Analysis (PCA)
PCA reduces dimensionality by transforming features into a new set of uncorrelated
components called principal components.
FEATURE EXTRACTION TECHNIQUES
Scenario: Customer Segmentation in a Retail Business
A retail company wants to segment its customers based on their purchasing behavior to
offer personalized recommendations. The dataset contains the following features for each
customer:
[Link] Income (₹)
[Link] Score (1-100)
[Link]
[Link] of Store Visits per Month
[Link] Amount Spent (₹)
[Link] Loyalty Score (1-10)
With six features, the dataset is high-dimensional, making it difficult to visualize and
analyze patterns effectively. To simplify the dataset while retaining important information,
the company decides to use Principal Component Analysis (PCA).
FEATURE EXTRACTION TECHNIQUES
Step 1: Standardization of Data
Since PCA relies on variance, we first standardize the data so that all features have a mean
of 0 and standard deviation of 1.
This ensures that features with larger numerical scales (e.g., Total Amount Spent) do not
dominate the PCA process.
Step 2: Computing Covariance Matrix
We compute the covariance matrix, which helps understand the relationship between
different features. If two features are highly correlated, PCA can combine them into a
principal component, reducing redundancy.
For example, if Annual Income and Total Amount Spent have a strong correlation, PCA will
merge them into a new component.
FEATURE EXTRACTION TECHNIQUES
Step 3: Eigenvalues and Eigenvectors Calculation
We then compute eigenvalues and eigenvectors of the covariance matrix.
Eigenvalues indicate how much variance each principal component captures.
While eigenvectors define the direction of these components.
Step 4: Selecting Principal Components
After sorting the eigenvalues in descending order, we choose the top two principal
components that retain most of the variance (e.g., 85-90% of the total information).
• PC1 (Principal Component 1) = A weighted combination of Annual Income, Spending
Score, and Total Amount Spent
• PC2 (Principal Component 2) = A weighted combination of Age, Store Visits, and Loyalty
Score
FEATURE EXTRACTION TECHNIQUES
(2) Linear Discriminant Analysis (LDA)
LDA is similar to PCA but focuses on maximizing class separability rather than variance.
Example:
In an email classification task where emails are labeled as Spam or Not Spam, LDA can
extract key features that best distinguish these classes, such as word frequency
patterns.
FEATURE EXTRACTION TECHNIQUES
(3) Autoencoders
Autoencoders are neural networks used to compress and reconstruct data while
capturing its essential characteristics.
Example:
In facial recognition systems, an autoencoder can reduce a high-resolution image with
thousands of pixels into a compact representation while retaining key facial features for
identification.