Supervised Learning Class Notes
Uploaded by
silviuSupervised Learning Class Notes
Uploaded by
silviuCommon questions
Powered by AIRegularization methods such as Lasso (L1 regularization) or Ridge (L2 regularization) are best used when models are experiencing overfitting due to high complexity or multicollinearity among input features. Regularization adds a penalty on the magnitude of the coefficients, helping in reducing variance in predictions and improving the generalization of the model on unseen data .
Increasing model complexity (e.g., using more sophisticated models or deeper networks) can improve a model's ability to capture complex patterns in data, hence reducing underfitting. However, it risks overfitting if the increase is excessive or unjustified by available data . On the other hand, adding relevant features can provide more information for the model to learn from, but care must be taken to avoid multicollinearity, which can lead to misleading coefficients and reduced predictive performance . Both strategies need to be balanced with cross-validation to ensure model generalization .
Missing data in a supervised learning problem can be addressed by imputation methods which fill in missing values with substituted values. Popular techniques include mean or median imputation, forward or backward filling in time series, or model-based imputation using predictive models to estimate missing values based on observed data points . Proper handling of missing data can enhance model accuracy by providing more complete data for learning, reducing bias, and preserving statistical power of the dataset .
Mean Squared Error (MSE) is effective because it emphasizes larger errors by squaring the error terms, which can be beneficial for catching significant underperformance in predictions . However, its scale depenency and sensitivity to outliers can be a disadvantage, as large errors disproportionately affect the metric, potentially distorting model evaluations. Despite this, MSE remains widely used for its straightforward interpretation and mathematical properties conducive for optimization .
Linear regression is used for predicting continuous outcomes based on input features, utilizing a model in the form y = w₀ + w₁x₁ + ... + wₙxₙ, where weights are adjusted to minimize the Mean Squared Error (MSE). Logistic regression, on the other hand, is used for binary classification problems. It employs the logistic sigmoid function, σ(z) = 1/(1 + e⁻ᶻ), to predict probabilities of binary outcomes, with optimization done through minimizing Log Loss or Binary Cross-Entropy using gradient descent .
Overfitting in decision trees can be mitigated through pruning, which reduces the complexity of the final model by removing sections of the tree that provide little power in predicting target variables . Additionally, employing techniques like cross-validation, setting a minimum number of samples required to split a node, or a maximum depth for the tree, can help in reducing overfitting .
Feature normalization is critical because it ensures that all features contribute equally to the model's predictions, preventing some features with larger scales from disproportionately affecting the model's outputs. It typically involves scaling features to a specific range, such as 0 to 1, which can improve the convergence speed of optimization algorithms like gradient descent . In practice, this can be achieved using techniques such as min-max scaling or standardization .
Gradient descent is used to optimize the weights of a linear regression model by iteratively adjusting them to minimize the cost function, typically the Mean Squared Error (MSE). It computes the gradient of the MSE with respect to the weights and updates the weights in the opposite direction of the gradient, proportionate to a learning rate, until convergence is achieved, meaning minimal error .
Decision trees offer interpretability by visually representing decisions made based on feature values, are capable of handling both non-linear data and categorical variables, and do not require feature scaling . However, they are prone to overfitting without techniques such as pruning, and they may not generalize well to unseen data. Decision trees can also be sensitive to small variations in the data, leading to high variance in predictions .
In supervised learning, models are trained on labeled data, meaning the input data is paired with the correct output, such as features paired with their corresponding target values . This label information guides the model's learning process, allowing it to make accurate predictions about unseen data by minimizing error between predicted and actual outcomes. Unsupervised learning lacks such labels and instead focuses on identifying inherent patterns or groupings within input data without predefined outputs .