Implement the non-parametric Locally Weighted Regression algorithm in order to fit data points.
Select appropriate
data set for your experiment and draw graphs.
Steps:
1. Select a dataset: I will generate a simple nonlinear dataset for demonstration.
2. Implement LWR: The algorithm will use a Gaussian kernel to assign weights based on proximity.
3. Fit and predict: I will fit the model for different values of bandwidth (τ\tauτ) and visualize results.
4. Graphical Analysis: The output will include graphs showing data points, the fitted curve, and explanations.
Explanation:
1. Dataset Creation: The dataset follows y=sin(x)y = \sin(x)y=sin(x) with added Gaussian noise.
2. Kernel Function: A Gaussian kernel is used to assign higher weights to closer points.
3. LWR Algorithm:
o Computes weights for each training point relative to the query point.
o Solves a weighted least squares problem to find the best-fit line for local data.
o Predicts values based on the locally fitted model.
4. Visualization:
o The plot shows three different fits using different values of tau (bandwidth).
o Smaller tau makes the fit too sensitive to noise.
o Larger tau smooths the curve but may underfit.
Program 6
import numpy as np
import [Link] as plt
from [Link] import cdist
# Generate a nonlinear dataset
[Link](42)
X = [Link](-3, 3, 100)
y = [Link](X) + [Link](scale=0.1, size=[Link]) # True function with noise
# Convert to column vector format for computation
X = [Link](-1, 1)
def gaussian_kernel(x, X_train, tau):
"""Compute the weights using a Gaussian kernel."""
distances = cdist(X_train, x, 'sqeuclidean') # Squared Euclidean distance
weights = [Link](-distances / (2 * tau ** 2))
return weights
def locally_weighted_regression(X_train, y_train, x_query, tau):
"""Perform locally weighted regression for a single query point."""
X_bias = [Link]([[Link]((X_train.shape[0], 1)), X_train])
x_query_bias = [Link]([1, x_query]) # Bias term for query point
W = [Link](gaussian_kernel([[x_query]], X_train, tau).flatten()) # Weight matrix
theta = [Link](X_bias.T @ W @ X_bias) @ (X_bias.T @ W @ y_train) # Normal equation
return x_query_bias @ theta # Prediction
def predict_lwr(X_train, y_train, X_test, tau):
"""Compute predictions for all test points."""
return [Link]([locally_weighted_regression(X_train, y_train, x, tau) for x in X_test])
# Define test points for smooth curve
X_test = [Link](-3, 3, 200).reshape(-1, 1)
# Apply LWR for different values of tau
taus = [0.1, 0.5, 1.0]
[Link](figsize=(12, 4))
for i, tau in enumerate(taus):
y_pred = predict_lwr(X, y, X_test, tau)
[Link](1, len(taus), i+1)
[Link](X, y, label='Data', alpha=0.6)
[Link](X_test, [Link](X_test), 'g--', label='True function')
[Link](X_test, y_pred, 'r', label=f'LWR fit (tau={tau})')
[Link]()
[Link]('X')
[Link]('y')
[Link](f'LWR with tau={tau}')
plt.tight_layout()
[Link]()
Analysis of the Graphs:
Leftmost plot (τ=0.1\tau=0.1τ=0.1):
o The model captures fine details but is too sensitive to noise, leading to overfitting.
o The red curve fluctuates heavily.
Middle plot (τ=0.5\tau=0.5τ=0.5):
o A good balance between bias and variance.
o The red curve follows the true function (dashed green) closely while still smoothing noise.
Rightmost plot (τ=1.0\tau=1.0τ=1.0):
o The fit is very smooth but underfits the true function.
o Some sharp variations in y=sin(x)y = \sin(x)y=sin(x) are lost.