Introduction to Machine Learning Notes
1. Statistical ML Methods
Statistical ML Methods:
- Frequentist Approach: Parameters are treated as fixed. The parameter estimate is computed using
estimators such as mean, standard deviation, etc.
Example: If you roll a dice and want to estimate the probability of a 6, you compute the frequency
of occurrence from past data.
- Bayesian Approach: Treats parameters as random variables and computes a posterior probability
based on prior beliefs and observed data.
Formula: Posterior = Likelihood * Prior / Evidence.
Example: In a spam detection model, we update the belief about an email being spam based on
prior knowledge and current evidence.
2. Feature Extraction
Feature Extraction: Involves extracting useful attributes from data instances that help in
classification tasks.
- Example: For classifying fish (salmon vs sea bass), useful features could be length, texture,
lightness, etc.
Binary, Categorical, and Real-valued features can be used to create feature vectors.
(Figure 1: Feature extraction example for fish classification - where 'Length' and 'Lightness' are used
as key features)
3. K-Nearest Neighbor (KNN) Classifier
KNN is a lazy learning classifier that assigns a label based on the majority class of the 'k' nearest
points in the training data.
- Distance Metric: Euclidean distance is commonly used.
- Example: Classify a new data point by checking its k-nearest neighbors.
KNN works well in low-dimensional spaces but struggles in high-dimensional data (curse of
dimensionality).
4. Logistic Regression
Logistic regression estimates the probability of a binary outcome using the sigmoid function:
Sigmoid function: f(x) = 1 / (1 + exp(-x))
It's used for classification tasks where the output is binary, e.g., spam or not spam.