Algorithm of Logistic Regression 3.
Apply the Sigmoid Function:
Logistic regression is a statistical method used for binary z to get the predicted probability.
classification problems, where the goal is to predict the Y^=σ(z)=1/ 1+e −z
probability that a given input point belongs to a certain class. [Link] Function:Define the cost function using
Unlike linear regression, which predicts continuous output, the binary cross-entropy (log loss) to measure
logistic regression predicts a probability that falls between 0 the error between the predicted probabilities
and 1. and actual class labels.
[Link] core of logistic regression is the sigmoid function, which J(β)=− 1/m∑ [yilog(y^i)+(1−yi)log(1−y^i)]
𝜎σ(z)= 1/1+e −z
maps any real-valued number to a value between 0 and 1. where m is the number of training examples,yi
is the actual class label
[Link] Combination of Input Features: [Link] Descent Optimization: Use gradient
Calculate the linear combination of input features (weighted descent or another optimization algorithm to
sum). z=β 0+β1x1+β2x2+…+βnxn minimize the cost function and find the best-
𝛽
fitting parameters (β).
𝑗
What is Gradient Boosting? [Link] a weak learner (e.g., a decision
Gradient Boosting is an ensemble machine learning tree with limited depth) to the
technique used for regression and classification tasks, residuals. This learner is trained to
which builds models in a stage-wise fashion and predict the residuals from the previous
generalizes them by optimizing a differentiable loss step. 4. Update the model byadding
function. The idea is to combine the predictions of the weighted prediction of the decision
several base estimators (typically decision trees) to tree. 5. Calculate the residuals again
improve robustness and accuracy. using the updated model. 6. After
[Link] with an initial model, usually a simple mode many iterations, you end up with a
like the mean of the target values for regression or the series of small trees. Each tree corrects
log-odds for classification. [Link] the residuals the errors of the previous one. Your
(errors) of the current model, which represent the final model is the sum of all these small
difference between the actual target values and the trees’ predictions.
predicted values by the model.
Steps to Construct Decision Tree 1. Clustering is an unsupervised machine learning
Select the best Attribute(GI,IG) technique that involves grouping a set of objects in such
[Link] the datasets into subsets 3. a way that objects in the same group (or cluster) are
Create Decision Nodes or Leaf Nodes4. more similar to each other than to those in other
Apply the same process recursively to groups. It helps in identifying patterns and structures in
each child node data without pre-labeled responses.
Entropy and Information Gain: Support Vector Machine (SVM) is a supervised machine
Entropy measures the randomness in the learning algorithm commonly used for classification and
information being processed. The formula regression tasks. The goal of SVM is to find the optimal
for entropy is: hyperplane that separates data points of different
=−∑pi log2(pi) classes with the maximum margin.
A hyperplane is a decision boundary that separates
Information Gain is the reduction in different classes in the feature space. In 2D, it is a line;
entropy achieved by partitioning the in 3D, it is a plane; and in higher dimensions, it is a
examples according to an attribute: hyperplane.
Information Gain(S,A)=Entropy(S)−
×∑ |Sv|/|S|*Entropy(Sv)
Support vectors are the data points that are closest to
the hyperplane. These points are critical in defining the
position and orientation of the hyperplane. The SVM
algorithm seeks to maximize the margin around these
support vectors.
The margin is the distance between the hyperplane and
The main steps include selecting the best the nearest data points from either class. The objective
attribute, splitting the dataset, and of SVM is to maximize this margin to ensure better
recursively applying this process. generalization on unseen data.
Properly handling overfitting with
techniques like pruning is essential to
create an effective decision tree model.