1.
In convex function -> local and global minima are the same; IN NON CONVEX
THEY ARE DIFFERENT
[Link] gradient descent, we update the values of both m and c to reach the global
minimum of the cost function.
[Link] simple linear regression model can be represented graphically as a best-
fit line between the data points, while the multiple linear regression model can
be represented as a plane (in 2-dimensions) or a hyperplane (in higher
dimensions)
Xavier Initialization:
A smart way to initialize neural network weights.
Keeps gradients stable (prevents vanishing/exploding).
Uses input/output layer sizes to set weight scale.
Best for tanh or sigmoid activations.
Not ideal for ReLU (use He Initialization instead).
Formula:
o Uniform: U(−√6/(n_in+n_out), √6/(n_in+n_out))
o Normal: N(0, √2/(n_in+n_out))
Used in most deep learning frameworks as the default for dense layers
================================================================
Gradient Descent
It is an optimization algorithm used to minimize a cost/loss function by
iteratively adjusting model parameters (like weights in a neural network).
================================================================
1. Cost Function – Measures how far the model's predictions are from the
actual values.
2. Residual – The difference between the actual value yiy_iyi and predicted
value y^i\hat{y}_iy^i.
3. Residual Sum of Squares (RSS) – The sum of squared residuals:
∑(yi−y^i)2\sum (y_i - \hat{y}_i)^2∑(yi−y^i)2.
4. Mean Squared Error (MSE) – Average of squared residuals used as the
cost: 1n∑(yi−y^i)2\frac{1}{n} \sum (y_i - \hat{y}_i)^2n1∑(yi−y^i)2.
5. Minimization Problem – The task of finding the values of mmm and ccc
that minimize the cost function.
6. Gradient Descent – An iterative optimization algorithm that updates
parameters to minimize the cost.
7. Gradient – The derivative of the cost function that shows the direction of
steepest increase.
8. Learning Rate (α\alphaα) – A hyperparameter that controls how big
each update step is in gradient descent.
9. Weight (m) – The slope of the line that the algorithm learns to fit the
data.
[Link] (c) – The y-intercept of the line that the algorithm learns.
[Link] (y^\hat{y}y^) – The output value computed by the model for
a given input xxx.
[Link] Truth (y) – The actual target value from the dataset.
[Link] – A mathematical function (like y=mx+cy = mx + cy=mx+c) that
maps input to output.
[Link] – The process of adjusting model parameters to minimize
the cost function.
[Link] Function – The error for a single data point (e.g., (yi−y^i)2(y_i - \
hat{y}_i)^2(yi−y^i)2).
[Link] – One complete cycle of passing through all training data during
optimization.
[Link] – When the model is too simple to capture the trend in the
data.
[Link] – When the model learns noise and performs poorly on new,
unseen data.
[Link] (x) – An input variable used to predict the target output.
[Link] (y) – The output variable the model aims to predict.
[Link] – The point where further updates no longer significantly
reduce the cost.
[Link]
GLOBAL MINIMA - WHERE ERRORS ARE LOWEST
GRADIENT DESCENT - DESCENDS TOWARDS GLOBAL MINIMA
Assumptions of Linear Regression
1. Linearity
The relationship between inputs and output must be linear—i.e., the
expected value of y changes proportionally with x
2. No multicollinearity
Input features shouldn’t be highly linearly correlated; each should
contribute unique information. High correlation makes coefficient
estimates unstable .
3. Homoscedasticity
Errors (residuals) should have constant variance across all levels of the
predictors. In other words, residuals shouldn’t spread out unevenly.
4. Normality of residuals
The errors should follow a normal (bell-curve) distribution, which is
necessary for valid hypothesis testing and confidence intervals
🧠 Why these matter:
If linearity is violated → model predictions will be biased.
If multicollinearity exists → coefficient estimates become unreliable.
If homoscedasticity doesn't hold → statistical tests (e.g., t-tests) may be
invalid.
If residuals aren’t normal → confidence intervals and hypothesis tests
become inaccurate.
What is Feature Scaling?
Feature scaling means transforming your input features (columns in your
dataset) so that they are on a similar scale.
Fine-tuning model using Recursive Feature
Elimination (RFE
🔍 Main Idea
The blog explains how Recursive Feature Elimination (RFE) is used for feature
selection and model fine-tuning in machine learning. RFE is a wrapper method
that helps select the most important features by iteratively eliminating the
least important ones and building the model using only the top features.
🧠 Key Concepts
🔸 What is RFE?
A backward feature elimination method.
Removes one (or more) least important feature(s) in each iteration.
Continues until the desired number of features (n_features_to_select)
remains.
Can be used with any estimator (like Logistic Regression, Decision Tree,
etc.).
How RFE Works (with Example):
1. Start with all features (e.g. 15 features).
2. Build a model.
3. Remove the least important feature (based on coefficients/importance).
4. Repeat steps 2–3 until only the top N features are left (e.g. 8).
🧪 Code Snippet
python
CopyEdit
from sklearn.feature_selection import RFE
rfe = RFE(estimator=LogisticRegression(), n_features_to_select=8)
model_rfe = [Link](X_train, Y_train)
Y_pred = model_rfe.predict(X_test)
🧾 Understanding RFE Output
rfe.n_features_: Number of selected features.
model_rfe.support_: Boolean array of selected features (True = kept).
rfe.ranking_: Rank of all features (1 = selected, higher = dropped earlier).
✅ Advantages
Selects best features AND builds the model.
Works with any estimator.
Easy to interpret selection process.
Handles multicollinearity well.
Improves model performance by focusing on significant variables.
❌ Disadvantages
Computationally expensive (model is rebuilt multiple times).
Risk of overfitting if wrong number of features is selected.
Doesn’t help pick which estimator to use.
Feature importance may not always be obvious.
Can be slow on large datasets.
🧾 Conclusion
RFE is a powerful technique for fine-tuning ML models by selecting only the
most important features. While it's slow and resource-intensive, its ability to
improve model accuracy makes it a valuable tool in the machine learning
workflow.