Introduction to Machine Learning: Lecture Notes &
Exercises
Concise lecture notes with algorithms, examples, and exercises — suitable for students and instructors.
Author: Ash Raf (Compiled)
Affiliation: Independent Educational Resource
Abstract
These lecture notes introduce core concepts in supervised and unsupervised machine learning. Topics
covered include linear models, classification, decision trees, model evaluation, regularization,
optimization via gradient descent, and an overview of unsupervised learning methods. The document
includes worked examples and exercises appropriate for undergraduate-level courses or self-study.
Contents
1. Supervised Learning: Regression and Classification
2. Optimization and Gradient Descent
3. Regularization and Model Selection
4. Decision Trees and Ensemble Methods (overview)
5. Unsupervised Learning: Clustering & Dimensionality Reduction
6. Evaluation Metrics and Cross-Validation
7. Exercises and Suggested Readings
1. Supervised Learning: Regression and Classification
Supervised learning maps inputs X to outputs y using labeled examples. Common tasks are regression
(continuous y) and classification (discrete y). Below are two foundational models.
1.1 Linear Regression (Ordinary Least Squares)
Model: y = Xw + ε. The ordinary least squares solution minimizes the sum of squared errors: J(w) =
||Xw - y||^2. Closed-form solution (if X^T X invertible): w = (X^T X)^{-1} X^T y. Gradient (for use in
gradient descent): ∇_w J = 2 X^T (Xw - y). Use cases: baseline regression, interpretable parameters.
1.2 Logistic Regression (Binary Classification)
Model: P(y=1|x) = σ(w^T x) where σ(z)=1/(1+e^{-z}). The loss is the negative log-likelihood
(cross-entropy): L(w) = -∑[ y_i log σ(w^T x_i) + (1-y_i) log(1-σ(w^T x_i)) ]. No closed-form; optimize with
gradient-based methods (gradient descent, LBFGS, etc.).
2. Optimization and Gradient Descent
Gradient descent iteratively updates parameters to reduce loss: w ← w - η ∇_w L(w), where η is the
learning rate. Variants: - Batch gradient descent: uses full dataset per update. - Stochastic gradient
descent (SGD): single example per update — faster for large datasets. - Mini-batch SGD: compromise
using small batches. Common tricks: learning rate schedules, momentum, adaptive optimizers (Adam,
RMSProp).
3. Regularization and Model Selection
Regularization reduces overfitting by penalizing model complexity. Ridge (L2) regularization adds
λ||w||^2 to the loss; Lasso (L1) adds λ||w||_1 and can produce sparse solutions. Model selection
techniques include cross-validation (k-fold CV), which estimates generalization performance and helps
choose hyperparameters.
4. Decision Trees and Ensemble Methods (Overview)
Decision trees partition feature space using axis-aligned splits. Trees are interpretable but prone to
overfitting. Ensembles such as Random Forests (bagging) and Gradient Boosted Trees (e.g., XGBoost,
LightGBM) combine many trees to improve accuracy and robustness.
5. Unsupervised Learning: Clustering & Dimensionality Reduction
Clustering groups similar observations: K-Means partitions data into k clusters by minimizing
within-cluster variance. Dimensionality reduction: PCA (Principal Component Analysis) finds orthogonal
directions of maximum variance — useful for visualization and noise reduction.
6. Evaluation Metrics and Cross-Validation
Regression metrics: Mean Squared Error (MSE), Root MSE (RMSE), R^2. Classification metrics:
Accuracy, Precision, Recall, F1-score, ROC AUC. For imbalanced datasets prefer precision/recall or
AUC. Cross-validation provides an unbiased estimate of model performance; common choice is 5- or
10-fold CV.
7. Exercises
• Exercise 1: Given dataset X (n x d) and target y, derive the normal equation for linear regression and
discuss conditions when X^T X is non-invertible.
• Exercise 2: Implement gradient descent for linear regression and plot the loss vs iterations.
Experiment with different learning rates.
• Exercise 3: For a binary classification dataset, compare logistic regression and a decision tree in
terms of accuracy and calibration. Report precision and recall.
• Exercise 4: Use k-means to cluster a 2D synthetic dataset; vary k and describe the effect on
within-cluster variance.
• Exercise 5: Using cross-validation, tune the regularization parameter λ for ridge regression and report
the selected λ and test performance.
Suggested Readings & Resources
1. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
2. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
3. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning. Springer.
4. Online: scikit-learn documentation and tutorial notebooks ([Link]).
Appendix: Practical Tips for Projects and Reports
• Start with exploratory data analysis (EDA): visualize distributions, missing data, and correlations.
• Keep a baseline model (e.g., linear/logistic regression) before trying complex models.
• Use pipelines to preprocess data consistently (scaling, encoding).
• Track experiments and hyperparameters (e.g., using simple spreadsheets or tools like MLflow).