Course Code : BCAM-503
Title of the Course : Machine Learning
Section-D
Notes
1. Introduction to Feature Selection
What is Feature Selection?
Feature selection is the process of choosing the most relevant and important features
(variables) from a dataset to improve model performance.
Diagram: Feature Selection
+--------------------------------------------------+
| ORIGINAL FEATURE SET |
+--------------------------------------------------+
| F1 | F2 | F3 | F4 | F5 | F6 | F7 | F8 | F9 |
+--------------------------------------------------+
|
| Select Only Useful Features
v
+-------------------------------+
| SELECTED FEATURE SUBSET |
+-------------------------------+
| F1 | F3 | F6 | F8 |
+-------------------------------+
Why Feature Selection?
Reduces overfitting
Increases model accuracy
Reduces computational cost
Removes redundant/noisy variables
Improves interpretability
Diagram: Impact of Feature Selection
Before FS: High Dimensional → Slow Training → Overfitting
After FS : Low Dimensional → Fast Training → Better Accuracy
How Feature Selection Works?
1. Evaluate each feature
2. Rank features
3. Select top-K features
4. Train model using selected subset
Where to Use Feature Selection?
Text mining
Image processing
Bioinformatics (gene expression)
IoT sensor data
High-dimensional datasets
2. Feature Selection vs Feature Extraction
Feature Selection
Selects a subset of the original features.
Original: F1 F2 F3 F4 F5
Selected: F1 F3 F5
Feature Extraction
Creates new features from existing ones (PCA, LDA).
Original: F1 F2 F3 F4
New: PC1, PC2 (transformed features)
Comparison Table
Feature Selection Feature Extraction
Keeps original features Creates new features
Simple to interpret Less interpretable
No transformation Uses mathematical transformation
Examples: Chi-square, MI Examples: PCA, LDA
3. FILTER-BASED FEATURE SELECTION
Filter methods use statistical tests independent of ML models.
Common Filter Methods
Chi-square test
ANOVA F-test
Mutual Information
Correlation coefficient
Variance threshold
Diagram: Filter Method
+------------------+
Dataset ------------>| Statistical Test |--> Rank Features --> Select Top Features
+------------------+
Advantages
Fast
Scalable
Independent of ML model
Limitations
Ignores interaction between features
4. WRAPPER METHODS
Wrapper methods evaluate subsets of features using a machine learning model.
Types
Forward selection
Backward elimination
Recursive Feature Elimination (RFE)
Diagram: Wrapper Approach
+---------------------+
Feature Subset 1 --------->| Train Model & Score |
+---------------------+
|
Feature Subset 2 --------->(Repeat Steps)
|
Best Feature Subset
Advantages
High accuracy
Considers feature interactions
Limitations
Computationally expensive
Slow on large datasets
5. Wrapper vs Filter Methods
Diagram: Comparison
FILTER:
Data → Statistical Test → Select Features
WRAPPER:
Data → Select Subset → Train Model → Evaluate → Repeat
Table
Filter Method Wrapper Method
Model-free Model-dependent
Fast Slow
Good for high dimensions Good for small datasets
Uses statistical scoring Uses prediction accuracy
6. MODEL-BASED SELECTION (REGULARIZATION)
Regularization adds a penalty to reduce unnecessary features.
L1 Regularization (LASSO) → Feature Elimination
Some coefficients become zero.
Feature Weights After LASSO:
F1=0.8, F2=0, F3=0.2, F4=0
Selected: F1, F3
L2 Regularization (Ridge) → Shrinks weights
No coefficients become zero.
Elastic Net
Combination of L1 + L2.
Diagram: L1 vs L2
L1: Forces some weights to zero → Feature Selection
L2: Shrinks weights only → No Feature Elimination
7. SEQUENCE LABELING
Assigning a label to each item in a sequence.
Examples
POS tagging
Named Entity Recognition
DNA sequence labeling
Speech recognition
Diagram: Sequence Labeling
Input : Ram eats mango
Labels: NNP VBZ NN
Or for DNA:
Input : A T G C T
Labels: 1 0 0 1 2
Models Used
HMM
CRF
RNN
LSTM
Transformers (BERT)
8. CLUSTERING
Unsupervised grouping of similar data points.
Diagram: K-Means Clustering
Cluster A : o o o
Cluster B : x x x
Applications
Customer segmentation
Image grouping
Document clustering
Anomaly detection
9. INTRODUCTION TO SEQUENCE LEARNING
Sequence Learning = Learning from ordered/temporal data.
Examples
Text sequences
Speech signals
Video frames
Sensor data
Diagram: RNN for Sequence Learning
+-------+ +-------+ +-------+
x1 -->| RNN |--> h1| RNN |--> h2| RNN |--> h3
+-------+ +-------+ +-------+
| | |
y1 y2 y3
Models
RNN
LSTM
GRU
Transformer