INTRODUCTION TO

MACHINE LEARNING
SUPERVISED LEARNING - LINEAR MODELS
© 2024-2026 Pierre-Henri Paris
This work is licensed under CC BY 4.0


1
INTRODUCTION TO
SUPERVISED
LEARNING
2
WHAT IS SUPERVISED LEARNING?
A type of machine learning where the model learns from labeled data.
Input data: Features (X), Output: Target labels (y).
Common tasks:
Regression: Predicting continuous values (e.g., house prices).

🎯
Classification: Predicting categories (e.g., spam detection).
Find a function that maps inputs to outputs.

3
 SIMPLIFIED WORKFLOW
OF SUPERVISED LEARNING
1. Data collection and preprocessing.
2. Model training with labeled data.
3. Model evaluation and validation.
4. Deployment for predictions on unseen data.
Importance of clean, normalized, and labeled datasets for
effective learning.
Scaling ensures all features contribute equally and improves
optimization convergence.

4
DATA PREPARATION

5
 DATA PREPROCESSING
Preprocessing ensures data is clean and suitable for machine learning
models.
Why preprocess data?
Improves model convergence and stability.
Prevents features with large ranges from dominating others.
Ensures compatibility with algorithms sensitive to feature scaling.
Common preprocessing steps:
Handle missing values (e.g., imputation).
Encode categorical features (e.g., one-hot encoding).
Feature Scaling: Critical for many algorithms. (See next slide for details.)

6
 FEATURE SCALING
Why scale features?
Ensures all features contribute equally to the model.
Improves convergence speed for gradient descent.
Prevents features with large ranges from dominating others.
Methods:
Normalization: Rescales data to [0, 1].
Formula: x′
x−min(x)
= max(x)−min(x)
​

Used for neural networks and distance-based models.

Standardization: Centers data to mean 0 and scales to unit variance.
Formula: x′ = σ
x−μ
​

Used for regression, SVMs, and PCA.

When to use:
Normalization: For models sensitive to absolute ranges.
Standardization: For models sensitive to variance.

7
 NUMERICAL STABILITY
Why it matters:
Prevents overflow or underflow in computations.
Ensures reliable training and evaluation of models.
Techniques for stability:
Clipping values: Limit inputs to functions (e.g., [Link] for sigmoid).
Log-sum-exp trick: Stabilizes logarithmic computations.
Avoid division by zero: Add small values (ϵ) to denominators.
Example: Sigmoid function
Without clipping: Risk of overflow for large inputs.
With clipping: Inputs limited to a safe range.
import numpy as np
def sigmoid_with_clipping(x):
# Clip input values to [-5, 5]
x_clipped = [Link](x, -5, 5)
# Apply sigmoid function
return 1 / (1 + [Link](-x_clipp
# Example usage
x = [Link]([-10, -5, 0, 5, 10])
y = sigmoid_with_clipping(x)
print(f"Input: {x}")
print(f"Output: {y}")
# Output: [0.00669285 0.00669285 0.

8
LINEAR REGRESSION

9
INTRODUCTION TO LINEAR REGRESSION

10
COST FUNCTION FOR LINEAR REGRESSION
The cost function measures how well the model fits the data.
For linear regression, use Mean Squared Error (MSE):

N
1
∑ (yi − y^i )
2
MSE = ​ ​ ​ ​ ​

N
i=1

Smaller MSE means a better fit.

Regularization: Adds a penalty term to the cost function to control model complexity:
Prevents overfitting by penalizing large weights.
Common types: L1 (Lasso), L2 (Ridge).

11
 OPTIMIZATION WITH GRADIENT DESCENT
Gradient descent minimizes the cost function.
Update rules for weights (w) and bias (b):
∂Cost
∂w1
​

∂Cost
∂w2
​

w := w − η∇w Cost, b := b − η∇b Cost where ∇w Cost = .

​

​ ​ ​ ​ ​ ​

⋮
∂Cost
∂wn
​

η : Learning rate (step size).

Gradient clipping: Limits the magnitude of gradients to stabilize training.
Early stopping: Halts training when loss improvement becomes negligible.

12
REGULARIZATION

13
 REGULARIZATION
Helps prevent overfitting by adding a penalty term to the cost function.
Types of regularization:
L1 (Lasso): Encourages sparse models by penalizing absolute weights.
L2 (Ridge): Penalizes the square of the weights for smoother models.
Impact on optimization:
L1: Adds a constant gradient penalty proportional to the sign of weights.
L2: Adds a gradient penalty proportional to the magnitude of weights.
Regularized cost functions:
L1: Cost + λ ∑ ∣w∣
L2: Cost + λ ∑ w 2

14
 LOGISTIC
REGRESSION

15
 INTRODUCTION TO LOGISTIC REGRESSION
Used for binary classification tasks (e.g., spam vs. non-spam).
Maps input features to a probability value using the sigmoid function:
1
σ(z) = −z
, z = Xw + b
1+e
​

Output is a probability between 0 and 1.

Decision boundary separates classes (e.g., 0.5 for binary classification).

16
 LOSS FUNCTION FOR LOGISTIC REGRESSION
Uses binary cross-entropy loss to evaluate model performance:
N
1
Loss = − ∑ [yi log(y^i ) + (1 − yi ) log(1 − y^i )]
​ ​ ​ ​ ​ ​ ​ ​

N
i=1

Minimizes the distance between predicted probabilities and true labels.

Penalty increases for incorrect predictions.

17
 DECISION BOUNDARIES
A decision boundary is a line or surface that separates different classes.
For logistic regression, the boundary is linear.
Examples:
2D classification: A line separating two classes.
Higher dimensions: A hyperplane.

18
MODEL TUNING

19
 HYPERPARAMETER TUNING
What are hyperparameters?
Settings chosen before training the model (e.g., learning rate,
regularization strength).
Do not change during training, unlike model parameters (weights,
biases).
Why tune hyperparameters?
Improves model performance and generalization.
Avoids underfitting and overfitting.
Common approaches:
Grid Search: Try all combinations of specified values.
Random Search: Sample random combinations within a range.
Manual Tuning: Adjust based on intuition and results.
Example: Tuning learning rate and λ for regularization.
20
 CROSS-VALIDATION
Ensures robust evaluation by splitting data into training and validation sets.
k-Fold Cross-Validation:
Splits data into k subsets (folds).
Trains the model on k-1 folds and validates on the remaining fold.
Repeats k times and averages the performance.
Reduces risk of overfitting to a single validation set.

21
MODEL EVALUATION

22
 REGRESSION METRICS
Common metrics for evaluating regression models:
Mean Squared Error (MSE): Average of squared differences between actual and predicted values.
Root Mean Squared Error (RMSE): Square root of MSE for interpretability in original units.
2
R -score: Proportion of variance explained by the model.
Regularization impact:
May increase MSE slightly but improves generalization.
R² should still remain high for a well-regularized model.
Metric Formula Without With Interpretation
Regularization Regularization
MSE 1 n 2 24.5 26.8 Lower is better. Slightly
n
​
∑i=1 (yi − y^i )
​ ​ ​ ​

increases with regularization

due to bias-variance tradeoff.
RMSE RMSE = 4.95 5.18 Interpretable in original units.
MSE ​
Shows actual average
prediction error magnitude.
R²- MSE 0.98 0.96 Proportion of variance
1− Var(y)
​

score explained (0-1). Should remain

high with proper
regularization.
Note: Example values shown for a typical housing price prediction model. Actual values will vary by dataset and model.
23
CLASSIFICATION METRICS
Metrics to evaluate binary classifiers:
Accuracy: Percentage of correct predictions.
Precision: Ratio of true positives to predicted positives.
Recall: Ratio of true positives to actual positives.
F1-Score: Harmonic mean of precision and recall.
Use a confusion matrix to visualize predictions.

24
ADVANCED
CONCEPTS
25
 BIAS-VARIANCE TRADEOFF
Bias: Error from incorrect model assumptions (e.g., underfitting).
Variance: Error from sensitivity to small changes in the training data (e.g.,
overfitting).
Tradeoff:
Low bias → More complex model, higher variance.

🎯 Find the optimal balance for best performance.

Low variance → Simpler model, higher bias.

26
 DEBUGGING TIPS
Common issues:
Vanishing gradients: Use appropriate learning rates and activation functions.
Overfitting: Add regularization or use cross-validation.
Poor scaling: Normalize or standardize features to improve model performance.
Numerical instability:
Clip values (e.g., gradients or sigmoid inputs) to prevent overflow or
underflow.
Adjust initialization of weights for better stability.
Exploding gradients: Apply gradient clipping to limit their magnitude.
Solutions:
Tune hyperparameters (e.g., learning rate, regularization strength) iteratively.
Analyze model performance with evaluation metrics and loss curves.

27
 RECAP
Linear Regression:
Predicts continuous values.
Optimized using gradient descent and evaluated with metrics like
MSE and R².
Logistic Regression:
Used for binary classification tasks.
Outputs probabilities, uses sigmoid function, and evaluated with
metrics like accuracy and F1-score.
Regularization:
Prevents overfitting by penalizing large coefficients (L1/L2).
Evaluation:
Metrics for regression and classification guide model
improvements.
28
To get the PDF of these slides and print them, click
here and then use the PDF printer of your browser.

29