IE406 - Machine Learning

Lab-2: Ridge Regression using Normal Equation

Course Instructor: Arunava Chakravarty

TA: Hetul Kadiya
TA: Kathan Shah

Objective
In this lab, students will:

• Implement Ridge Regression using the Normal Equation based closed-form solution.

• Perform polynomial basis expansion to project the original features into a higher-
dimensional feature space.

• Understand the effect of polynomial degree and L2 regularization on regression perfor-

mance.

• Perform a grid search over polynomial degree and the L2 regularization weight.

• Select the best hyperparameter combination using the validation set.

• Evaluate the selected model on the unseen test set using multiple regression metrics.

Dataset
A CSV file named [Link] is provided along with this assignment.
The dataset contains 11 physicochemical input features and one target variable:

• fixed acidity

• volatile acidity

• citric acid

• residual sugar

• chlorides

• free sulfur dioxide

1
 • total sulfur dioxide

• density

• pH

• sulphates

• alcohol

• quality – target variable

The task is to predict the wine quality score from the 11 physicochemical features.
Note: Students must use only the provided [Link] file for this lab.

Task 1: Dataset Preparation

1. Load the [Link] dataset and store the input features and target
values in NumPy arrays.

2. Randomly split the dataset into three mutually exclusive sets:

• Training Set (60%)

• Validation Set (20%)
• Test Set (20%)

Use a fixed random seed so that the same split is obtained every time the program is
executed. Use:

[Link](9432)

3. Polynomial Basis Expansion: Apply polynomial basis expansion to the input fea-
tures using PolynomialFeatures from [Link].
Perform polynomial expansion for the following degrees:

d ∈ {1, 2, 3, 4}

Use:

PolynomialFeatures(degree=d, include bias=False)

The polynomial basis expansion should be performed before feature normalization.

2
 4. Feature Normalization: After polynomial basis expansion, normalize each feature
dimension to have zero mean and unit standard deviation.
The normalization parameters (mean and standard deviation) must be calculated only
from the training set. The same parameters must then be used to normalize the
validation and test sets.
5. Feature Augmentation: Add a column of ones to the feature matrix to incorporate
the bias term into the matrix formulation.
6. The target variable quality should not be normalized for this lab. Predictions should
therefore remain in the original target scale for computation of all performance metrics.

Task 2: Implement Ridge Regression using Normal Equa-

tion
Implement Ridge Regression using the Normal Equation based closed-form solution.

• Implement the Ridge Regression solution using only Python and NumPy for the matrix
operations.
• Do not use any pre-built regression model such as Ridge, LinearRegression, or any
other regression implementation from sklearn or another machine learning library.
• The Ridge Regression parameter vector must be computed using the following closed-
form solution:

−1
θ = XbT Xb + λI ′ XbT Y

where λ is the L2 regularization weight and I ′ is the modified identity matrix.

• Write the above equation explicitly in the notebook and implement it using matrix
operations.
• The bias parameter should not be regularized. Therefore, the diagonal element
corresponding to the bias term in the regularization matrix should be set to zero.
• The modified identity matrix can be represented as:

I ′ = diag(1, 1, . . . , 1, 0)

where the final zero corresponds to the bias term.

• You may use a numerically stable alternative such as the Moore–Penrose pseudoinverse
instead of explicitly computing the matrix inverse.
• Implement a prediction function that uses the learned parameter vector θ to predict
the target values.

3
Task 3: Grid Search over Polynomial Degree and λ
Perform a grid search over the following two hyperparameters.

Polynomial Degree
Use the following polynomial degrees:

d ∈ {1, 2, 3, 4}

L2 Regularization Weight
Use the following values of λ:

λ ∈ {0.0001, 0.001, 0.01, 0.1, 1}

This results in:

4 × 5 = 20
different hyperparameter combinations.
For every combination of polynomial degree and λ:

1. Generate the corresponding polynomial features.

2. Normalize the expanded features using parameters calculated only from the training
set.

3. Train the Ridge Regression model using the Normal Equation.

4. Compute predictions on the training and validation sets.

5. Calculate the required evaluation metrics.

The test set must not be used during the grid search or hyperparameter selection.

Task 4: Evaluation and Hyperparameter Selection

For every one of the 20 experiments, calculate the following metrics:

• Root Mean Squared Error (RMSE)

• Mean Absolute Percentage Error (MAPE)

• R2 Score

The metrics should be calculated separately for the:

4
 • Training set

• Validation set

Create a table containing, for each experiment:

• Polynomial degree

• λ

• Training RMSE

• Validation RMSE

• Training MAPE

• Validation MAPE

• Training R2

• Validation R2

Select the best combination of polynomial degree and λ based on the validation per-
formance.
The test set must not be used to select the polynomial degree or λ.

Task 5: Visualization of Grid Search Results

Visualize the effect of the two hyperparameters on validation performance.
Create heatmaps for the validation set showing:

• Polynomial degree vs λ for Validation RMSE.

• Polynomial degree vs λ for Validation MAPE.

• Polynomial degree vs λ for Validation R2 .

The rows and columns of each heatmap should clearly indicate the corresponding poly-
nomial degree and λ values.
Clearly identify the best hyperparameter combination based on validation performance.

5
Task 6: Inference on the Unseen Test Set
After completing the grid search and selecting the best polynomial degree and λ using the
validation set:

1. Train the final Ridge Regression model using the selected polynomial degree and λ.

2. Evaluate the selected model on the unseen test set.

3. Report the following metrics for the Training, Validation, and Test sets:

• RMSE
• MAPE
• R2

4. Present the results in a single table for comparison.

5. Plot the Actual vs Predicted Wine Quality values for the test set using a scatter
plot.

The test set must be used only for the final evaluation and must not be used during
hyperparameter search or model selection.

Submission Instructions
Upload only one Jupyter Notebook (.ipynb) file to Google Classroom.
The notebook should contain:

• Complete code

• All required outputs

• Grid search results

• Required tables

• Required plots

• Final test-set evaluation

Name the notebook as:

[studentid] [Link]

The file [Link] is provided along with the lab assignment and must
be used for this lab.
Any other filename or additional files will not be accepted.