Linear Regression Problem Set and Solutions
Uploaded by
aldobakhoslyLinear Regression Problem Set and Solutions
Uploaded by
aldobakhoslyCommon questions
Powered by AINormalizing data is valuable because it ensures that each feature contributes equally to the result, avoiding dominance by features with larger ranges, and helps in achieving better convergence during optimization. Without normalization, features with larger ranges can disproportionately influence the model parameters, leading to biased coefficients and potentially erroneous predictions .
Ridge Regression doesn't incorporate an L1-norm penalty; instead, it uses an L2-norm penalty, which penalizes the sum of squares of coefficients. The statement is incorrect as the L1-norm is used in Lasso Regression, known for setting some coefficients to zero, while Ridge focuses on shrinking them based on the penalty strength .
Introducing an extreme data point, such as a low score at high study hours, significantly affects the slope (ω1) and intercept (ω0) of the regression model. The model will adjust parameters to minimize the impact of this point on the overall error, potentially skewing the line to fit closer to the outlier, indicating an increased bias .
The statement that RSS is non-convex in linear regression is incorrect. In linear regression, the RSS function is convex due to its formulation as a sum of squared differences, which guarantees a single global minimum. Non-convexity occurs in more complex models like those involving neural networks with non-linear activation functions .
No, gradient descent will not lead to multiple solutions when minimizing the Residual Sum of Squares (RSS) for linear regression because the RSS function in linear regression is convex. In a convex optimization problem, there is a single global minimum, ensuring that any initial starting point will converge to the same solution .
Ridge regression is less sensitive to outliers than ordinary least squares because it includes an L2 penalty term that limits the size of the coefficients, thus reducing the influence of extreme data points. This penalty helps in stabilizing the model and improving its generalization ability, although it could introduce a bias in the coefficients .
For linear regression, different initializations should not lead to different solutions because of the convexity of the optimization problem. However, in non-convex problems, such as those encountered with deep learning models, different initializations can indeed lead to convergence at different local minima, impacting the final model quality .
Adding bonus points uniformly affects only the intercept (ω0) of the model, not the slope (ω1), since the relative differences between predictions remain unchanged. In the equation Y = 2.755 * X + 52.013, adding 3 points to each student causes ω0 to increase by 3, leading to a new intercept .
A very small β1 value indicates that changes in x have little to no impact on the predicted value of y, suggesting that the model is too simplistic to capture the underlying relationship. This typically implies underfitting, where the model does not capture the complexity of the data and fails to perform well on unseen samples .
This assumption suggests that the response variable has an underlying linear relationship with the predictors, corrupted by random noise. It allows the model to fit a line to the data by minimizing the discrepancy between observed and predicted values, recognizing that deviations are expected due to randomness rather than model inadequacy .