Module 3

Standard Curves
1. Straight line

y=mx+c ;

y=a0x+a1 ;

y=ax+b
2 Quadratic Curve or Parabola Curve

ax2 + bx + c = 0
3. Nth degree polynomial curve

y = an xn + an-1 xn-1+.……….…+ a0, where a0,a1,………,an are all constants

Example:- y=(x+3)(x−2)2(x+1)3
.
4. Exponential curve

y = abx, a ≠ 0, b is a positive real number and b ≠ 1

5. Geometric Curve

y=axb
Lagrange Interpolation
Lagrange Interpolation is a way of finding the value of any function at any given
point when the function is not given.

Other points on the function to get the value of the function at any required point.

Suppose we have a function y = f(x) in which substituting the values of x gives

different values of y. And we are given two points (x1, y1) and (x2, y2) on the
curve then the value of y at x = a(constant) is calculated using Lagrange
Interpolation Formula.
Given few real values x1, x2, x3, …, xn and y1, y2, y3, …, yn and there will be a
polynomial P with real coefficients satisfying the conditions P(xi) = yi,

∀ i = {1, 2, 3, …, n} and degree of polynomial P must be less than the count of

real values i.e., degree(P) < n.

Lagrange Interpolation Formula for the nth order is,

Properties of Lagrange Interpolation Formula
● This formula is used to find the value of the function at any point even when
the function itself is not given.
● It is used even if the points given are not evenly spaced.
● It gives the value of the dependent variable for any independent variable
belong to any function and thus is used in Numerical Analysis for finding the
values of the function, etc.
Uses of Lagrange Interpolation Formula
It is used to find the value of the dependent variable at any particular independent
variable even if the function itself is not given.

It is used in image scaling.

It is used in AI modeling.

It is used to teach NLPs, etc.

Least Square Method
The least-squares method is a statistical
method used to find the line of best fit of
the form of an equation such as y = mx
+ b to the given data.

The curve of the equation is called the

regression line.
Limitations for Least Square Method
This method exhibits only the relationship between the two variables. All other
causes and effects are not taken into consideration.

This method is unreliable when data is not evenly distributed.

This method is very sensitive to outliers. In fact, this can skew the results of the
least-squares analysis.
Curve fitting by method of least squares
Given points of data are (x1, y1), (x2, y2), (x3, y3), …, (xn, yn) in which all x’s are
independent variables, while all y’s are dependent ones.

Suppose that f(x) is the fitting curve and d represents error or deviation from each
given point.

Now, we can write:

d1 = y1 − f(x1)

d2 = y2 − f(x2)

…..

dn = yn – f(xn)
The least-squares explain that the curve that best fits is represented by the
property that the sum of squares of all the deviations from given values must be
minimum, i.e:
Least square straight line curve fitting
Quadratic Curve or Parabola Curve

ax2 + bx + c = 0
Numericals
Linear Weighted Least Squares Approximation
Curve fitting is the process of approximating a function 𝑓(𝑥) that best represents a
given set of data points.

Weighted Least Squares (WLS) is an extension of Ordinary Least Squares (OLS)

that assigns different importance to different data points based on their reliability
or significance.

In this discussion, we will cover:

1. Discrete Data Case – where data points are given explicitly.

2. Continuous Function Case – where we approximate a function over a
continuous domain.
Discrete Data Case

In the discrete case, we are given a set of 𝑛 observations:

where each 𝑦𝑖 is an observed value corresponding to 𝑥𝑖.

However, these observations may not be equally reliable. To account for this, we
introduce a weight 𝑤𝑖 for each data point, leading to the weighted sum of squared
residuals:
Linear Weighted Least Squares for Discrete Data
For a linear function:

the error function to be minimized is:

To minimize 𝑆𝑤Sw , take partial derivatives with respect to 𝑎0 and 𝑎1 and set
them to zero:
Normal Equations for Continuous Case
To find 𝑎0 and 𝑎1, set the derivatives with respect to 𝑎0 and 𝑎1 to zero:
 Key Differences Between Discrete and Continuous Cases

Feature Discrete Case Continuous Case

Data Points Given as individual points (xi,yi) Defined over an interval
Summation vs.
Integration Uses summation ∑wi Uses integral ∫w(x)dx
Weighting Each point has weight wi A function w(x) assigns weight
Solves normal equations using
Approximation Solves normal equations for n points integration
Applications of Weighted Least Squares
1. Handling Measurement Errors
● More precise measurements can be given higher weights.
● Less reliable data (e.g., noisy or outlier points) can be assigned smaller
weights.
2. Regression with Heteroscedasticity
● In real-world datasets, variance may not be constant.
● WLS adjusts for varying uncertainty in different regions.
3. Polynomial Curve Fitting

● The concept extends to higher-order polynomials, where:

● requires solving a system of weighted equations.

4. Numerical Approximation in Computational Science

● Used in finite element methods where different regions have different

accuracy.

5. Time Series Analysis

● Assigning higher weights to recent data ensures better adaptation to trends.

Nonlinear Weighted Least Squares Approximation
Nonlinear Weighted Least Squares (NWLS) is an extension of the Weighted Least
Squares (WLS) method used when the function to be fitted is nonlinear in its
parameters.

Unlike linear regression, where the parameters appear in a linear fashion,

nonlinear models require iterative optimization techniques for parameter
estimation.
Mathematical Formulation
Given a set of 𝑛 observations:
Solution Methods
Since f(x,θ) is nonlinear, we solve the problem using iterative methods.

Gauss-Newton Method
Discrete Data Case
Continuous Function Case
For a continuous function 𝑦(𝑥), we minimize the weighted error integral:

where w(x) is a weight function.

Applications of Nonlinear Weighted Least Squares
1. Machine Learning & AI
● Neural Networks: Training deep learning models involves minimizing
nonlinear loss functions.
● Logistic Regression: A nonlinear function for binary classification.
2. Engineering & Physics
● Sensor Calibration: Assigning different weights to sensor readings.
● Curve Fitting in Experimental Data: Fitting nonlinear models to physical data.
3. Economics & Finance
● Risk Modeling: Weighting different financial instruments based on risk.
● Stock Market Predictions: Nonlinear regression models.
Numerical
Multiple Regression Model
1. Introduction to Regression

Regression analysis is a statistical technique used to model the relationship

between one dependent variable and one or more independent variables.

It helps in predicting values and understanding the impact of independent

variables on the dependent variable.
Linear Regression
Linear regression is the simplest form of regression where the relationship
between a dependent variable and a single independent variable is modeled as a
linear equation:
Least Squares Estimation
Multiple Regression
Multiple regression analysis is a statistical technique that analyzes the relationship
between two or more variables and uses the information to estimate the value of
the dependent variables.

In multiple regression, the objective is to develop a model that describes a

dependent variable y to more than one independent variable.
Multiple Linear Regression
Multiple linear regression is used to estimate the relationship between two or more
independent variables and one dependent variable. You can use multiple linear
regression when you want to know:

How strong the relationship is between two or more independent variables and
one dependent variable (e.g. how rainfall, temperature, and amount of fertilizer
added affect crop growth).

The value of the dependent variable at a certain value of the independent

variables (e.g. the expected yield of a crop at certain levels of rainfall,
temperature, and fertilizer addition).
Assumptions of Multiple Linear Regression
● Linearity: The relationship between the dependent and independent variables
is linear.
● Independence: Observations are independent of each other.
● Homoscedasticity: The variance of errors is constant across all levels of
independent variables.
● Normality: The error terms follow a normal distribution.
● No multicollinearity: Independent variables are not highly correlated with each
other.
Estimation Using Ordinary Least Squares (OLS)
The following equation for the two independent variables X1 and X2
Linear Multiple Regression with k-Independent Variables
When there are independent variables, the multiple regression model is:

Interpretation of Coefficients
Model Evaluation Metrics
● R-Squared (R^2): Measures the proportion of variance in explained by the
independent variables.
● Adjusted R-Squared: Adjusts for the number of predictors to prevent
overfitting.
● F-Test: Tests the overall significance of the model.
● t-Tests: Tests the significance of individual regression coefficients.
● Residual Analysis: Checks the validity of regression assumptions.
Numericals on multiple linear regression
Q. Fit a regression equation to estimate to the following data of a
transport company on the weights of 6 shipments, the distances they were moved
and the damage of the goods that was incurred. Estimate the damage when a
shipment of 3700 kg is moved to a distance of 260 km.

Weight X1 (1000kg) 4.0 3.0 1.6 1.2 3.4 4.8

Distance X2 (100km) 1.5 2.2 1.0 2.0 0.8 1.6
Damage Y (Rs.) 160 112 69 90 123 168
Find Y when X1=10 and X2 =6 from the least squares regression equation of Y on
X1 and X2 for the following data

X1 3 5 6 8 12 14

X2 16 10 7 4 3 2
Y 90 72 54 42 30 12
Multivariate Linear Regression

Multivariate linear regression extends multiple regression by modeling multiple

dependent variables simultaneously:
This is quite similar to the simple linear regression model, multiple independent
variables contributing to the dependent variable and hence multiple coefficients to
determine and complex computation due to the added variables.

Multivariate regression shows the linear relationship between more than one
predictor or independent variable and more than one output or dependent
variable.

Differences from Multiple Regression

● Multiple Regression: One dependent variable, multiple independent variables.

● Multivariate Regression: Multiple dependent variables modeled together.
Examples of Multivariate Regression
Data analysis applies statistical analysis and logical techniques to describe,
visualize, reduce, revise, summarize, and assess data into useful information that
provides a better context for the data.

1. Buyer wants to estimate the price of a house. He will collect details such as
the location, number of bedrooms, size of square feet, amenities available,
etc. Based on these details, he can predict the price of the house and how
each variable is interrelated.
2. An agriculture scientist wants to predict the total crop yield expected for the summer.
He collected details of the expected amount of rainfall, fertilizers to use, and soil
[Link] building a Multivariate regression model, scientists can predict crop yield.
With the crop yield, the scientist also tries to understand the relationship among the
variables.
3. Suppose an organization wants to know how much it has to pay a new hire. In that
case, it will consider details such as education level, years of experience, job location,
and whether the employee has niche skills. Based on this information, you can predict an
employee’s salary, and these variables help estimate the salary.
4. Economists can use Multivariate regression to predict the GDP growth of a state or a
country based on parameters such as the total amount spent by consumers, import
expenditures, total gains from exports, and total savings.
5. A company wants to predict an apartment’s electricity bill. The details needed here are
the number of flats, the number of appliances used, the number of people at home, etc.
These variables can help predict the electricity bill.
Steps of Multivariate Regression analysis
● Feature selection-
○ The selection of features is an important step in multivariate regression.
○ Feature selection also known as variable selection.
○ It becomes important for us to pick significant variables for better model building.
● Normalizing Features-
○ We need to scale the features as it maintains general distribution and ratios in data.
This will lead to an efficient analysis.
○ The value of each feature can also be changed.
● Select Loss function and Hypothesis-
○ The loss function predicts whenever there is an error. Meaning, when the hypothesis
prediction deviates from actual values.
○ Here, the hypothesis is the predicted value from the feature/variable.
● Set Hypothesis Parameters-
○ The hypothesis parameter needs to be set in such a way that it reduces the loss
function and predicts well.
● Minimize the Loss Function-
○ The loss function needs to be minimized by using a loss minimization algorithm on
the dataset, which will help in adjusting hypothesis parameters.
○ After the loss is minimized, it can be used for further action.
○ Gradient descent is one of the algorithms commonly used for loss minimization.
● Test the hypothesis function-
○ The hypothesis function needs to be checked on as well, as it is predicting values.
○ Once this is done, it has to be tested on test data.
Advantages of Multivariate Regression
1. Improved Predictive Accuracy: Multivariate regression can provide a more
accurate and nuanced model than simple linear regression by incorporating
multiple predictors.
2. Handles Complex Relationships: It can capture the relationships between
the dependent variable and multiple predictors, including interactions and
combined effects, leading to a better understanding of complex data
structures.
3. Reduces Bias: Including several variables helps reduce bias by accounting
for factors that might influence the dependent variable, leading to more
reliable estimates.
4. Identifies Key Predictors: It helps determine which predictors significantly
impact the outcome, aiding in feature selection and model refinement.
5. Improves Model Fit: Multivariate regression, by considering multiple
variables, can often improve the fit of the model to the data, providing more
detailed insights into the underlying relationships.
Disadvantages of Multivariate Regression

● Multivariate regression analysis is complex and requires a high level of

mathematical calculation.
● The output produced by multivariate models is sometimes not accessible to
interpret because it has some loss and error outputs that are not identical.
● These models do not have much scope for smaller datasets. Hence, the
same cannot be applied to them. The results are better for larger datasets.
Numerical
Given the following data, find the multivariate regression equations.

X0 0 1 2 3 4

Y1 1 4 3 8 9

Y2 -1 -1 2 3 2

Solution
Regularized Regression
Regularized regression is an extension of linear regression that adds a penalty
term to the loss function to prevent overfitting.

Overfitting occurs when a model fits the training data too closely and performs
poorly on unseen data.

Regularization techniques help by imposing constraints on the model parameters,

reducing complexity, and improving generalization.

Regularized regression puts constraints on the magnitude of the coefficients and

will progressively shrink them towards zero.

This constraint helps to reduce the magnitude and fluctuations of the coefficients
and will reduce the variance of our model.
Properties of Regularization
● Reduces overfitting: Regularization techniques such as L1 and L2
regularization add a penalty to the loss function for which the model is
optimized. This penalty term encourages the model to use simpler weights,
which helps avoid overfitting the training data.
● Reduce weights: Regularization techniques reduce model weights toward
zero, reducing the impact of input variables that are less important to model
predictions. This makes the model more interpretable and easier to
understand.
● Improves generalization performance: Regularization helps the model
generalize better to new and invisible data by reducing the model’s
dependence on the training data. This makes the model more robust and
reliable for predictions.
● Requires hyperparameter tuning: Regularization techniques require
hyperparameter tuning as the regularization parameter that controls the
strength of the penalty term. The optimal value of the hyperparameter
depends on the specific problem and the data used and finding the best value
can be a challenging task.
● Works well with high-dimensional data: Regularization techniques work
well with high-dimensional data where the number of input variables is much
larger than the number of observations. In such cases, regularization helps to
avoid the dimensionality curse, where the model becomes too complex and
overfits the data.
Advantages of Regularization
● Regularization reduces overfitting in machine learning models.
● Regularization improves generalization performance and makes models more
reliable.
● Regularization increases the interpretability of the model and makes it easier
to understand.
● Regularization works well with multidimensional data and helps avoid the
dimensionality curse.
● Regularization provides a compromise between bias and variance in the
model.
● Adjusting the normalization parameter can help correct the bias-variance
trade-off for the best results.
Limitations of Regularization
● Regularization can make the model too simple and unsuitable for the data if
the regularization parameter is too large. This can cause the model to miss
important patterns in the data.
● Regularization assumes that all input variables are of equal importance, which
may not be true in some cases. Using domain knowledge to weigh the
importance of input variables can help improve model performance.
● Regularization can be computationally expensive, especially for
multidimensional data. This can make it difficult to use normalization in
real-time applications.
● Regularization assumes that the data is linearly separable, which may not be
true in some cases. Non-linear models, such as neural networks, maybe a
better fit for that data.
● Regularization assumes that the data are independent and identically
distributed (IIDs), which may not be true for some types of data, such as time
series or spatial data.
What Is Regularization in Machine Learning
Regularization restricts a model to prevent overfitting by penalizing large
coefficient values, with some techniques shrinking coefficients to zero.
When a model suffers from overfitting, we should control the model's complexity.
Technically, regularization avoids overfitting by adding a penalty to the model's
loss function:

We add coefficients to the cost function which is the Mean Squared Error (Sum of
Squared Residuals Divided by Degrees of Freedom) of the regression model,
which as a result increases the cost.
The optimizer would try to minimize the coefficient to decrease the cost function.
In regularization, penalizes all the parameters except the intercept.
Types of Regularized Regression
There are three primary types of regularized regression:

● Ridge Regression (L2 Regularization)

● Lasso Regression (L1 Regularization)
● Elastic Net Regression (Combination of L1 and L2 Regularization)
Methods of Regularization
Regularization works by modifying the cost function of standard linear regression.
The typical objective function for linear regression is:
Ridge Regression (L2 Regularization)
Definition

Ridge regression adds an L2 penalty, which penalizes large coefficient values by

adding the sum of squared coefficients to the loss function:
Working Mechanism
The L2 penalty prevents large coefficients by shrinking them towards zero but
does not eliminate them entirely.

Ridge regression is useful when many predictors contribute to the outcome and
need to be regularized without setting coefficients to zero.

The parameter λ controls the degree of shrinkage; higher λ values lead to smaller
coefficients.
Advantages
Reduces multicollinearity among predictors.

Helps in stabilizing model predictions.

Maintains all features in the model

Lasso Regression (L1 Regularization)
Definition

Lasso regression introduces an L1 penalty, which adds the absolute values of

coefficients to the loss function:
Working Mechanism
The L1 penalty encourages sparsity by setting some coefficients exactly to zero.

This results in feature selection, making Lasso useful for high-dimensional

datasets where only a few predictors are relevant.

The strength of regularization is controlled by λ, with higher values leading to more

coefficients being set to zero.
Advantages
Performs automatic feature selection.

Reduces model complexity by eliminating irrelevant features.

Helps in handling high-dimensional datasets.

Elastic Net Regularization (Combination of L1 and L2)
Definition

Elastic Net Regularization combines both L1 (Lasso) and L2 (Ridge) penalties to

overcome their individual limitations. The cost function is given by:
Working Mechanism
The L1 term encourages sparsity (feature selection), while the L2 term prevents
excessive shrinking of coefficients.

Elastic Net is useful when predictors are highly correlated, where Lasso alone may
randomly select one predictor and ignore the rest.

The balance between L1 and L2 penalties is controlled by two hyperparameters,

λ1 and λ2.
Advantages
Provides a balance between Ridge and Lasso.

Useful for handling collinearity among predictors.

Helps in selecting important features while maintaining model stability.

 Difference
Characteristic Ridge Regression Lasso Regression
Applies L2 regularization, adding a penalty term proportional to Applies L1 regularization, adding a penalty term proportional
Regularization Type the square of the coefficients to the absolute value of the coefficients.
Does not perform feature selection. All predictors are retained, Performs automatic feature selection. Less important
although their coefficients are reduced in size to minimize predictors are completely excluded by setting their
Feature Selection overfitting coefficients to zero.
Best suited for situations where all predictors are potentially Ideal when you suspect that only a subset of predictors is
relevant, and the goal is to reduce overfitting rather than important, and the model should focus on those while
When to use eliminate features ignoring the irrelevant ones.
Produces a model that is simpler, retaining only the most
Produces a model that includes all features, but their significant features and ignoring the rest by setting their
Output model coefficients are smaller in magnitude to prevent overfitting coefficients to zero.
Reduces the magnitude of coefficients, shrinking them towards Shrinks some coefficients to exactly zero, effectively removing
Impact on zero, but does not set any coefficients exactly to zero. All their influence from the model. This leads to a simpler model
Prediction predictors remain in the model with fewer features

Computation Generally faster as it doesn’t involve feature selection May be slower due to the feature selection process
Use when you have many predictors, all contributing to the Use when you believe only some predictors are truly
outcome (e.g., predicting house prices where all features like important (e.g., genetic studies where only a few genes out of
Example Use Case size, location, etc., matter) thousands are relevant).
Features Lasso Regression Ridge Regression Elastic Net Regression

L1 Penalty: Lasso uses the L2 Penalty: Ridge uses L1 + L2 Penalty: Elastic Net
Penalty Type absolute values of the square of the uses both absolute and
coefficients. coefficients. square penalties together.
Lasso completely removes Ridge makes all
Elastic Net removes some
Effect on unnecessary features by coefficients smaller but
features and reduces others,
Coefficients setting their coefficients to doesn’t set them to
balancing both.
zero. zero.
It is good when all
It is best when you have
features are useful, but It is best for when you have
It is best for many features and want to
you want to reduce many correlated features
remove irrelevant ones
their impact.
Alpha + L1_ratio: Two
Alpha: Controls how much
Alpha: Similar to Lasso, parameters. Alpha controls
Hyperparameters regularization is applied. A
controls the strength of regularization strength, and
involved higher alpha means more
regularization. L1_ratio adjusts the balance
shrinkage.
between Lasso and Ridge.
Features Lasso Regression Ridge Regression Elastic Net Regression
High bias, low variance: Low bias, high variance: Balance of bias and variance:
Bias and Lasso makes the model Ridge keeps all features, Elastic Net tries to find the right
Variance simpler, leading to higher leading to less bias but balance between simplicity and
bias but less overfitting. possibly more overfitting. complexity.
Ridge works well when
Lasso is great for Elastic Net combines Lasso’s
features are related but
Strengths automatically choosing feature selection and Ridge’s
shouldn’t be completely
important features. handling of correlations.
removed.
Ridge keeps all features,
Lasso can sometimes
which may not help in Elastic Net is a bit harder to tune
Weaknesses remove useful features if
high-dimensional data with due to having two parameters.
not tuned properly.
irrelevant features.
Imagine you have 100
If you have 100 features,
features to predict house If you have features like “size”
Ridge will reduce the
prices. Lasso will set the and “rooms” that are similar,
Example impact of every feature but
coefficients of irrelevant Elastic Net will remove one and
won’t completely remove
features (like house color) shrink the other.
any.
Support Vector Machine
Introduction

● Definition: "Support Vector Machine is a

supervised machine learning algorithm used for
classification and regression tasks.”

● “Support Vector Machine” (SVM) is a supervised

machine learning algorithm which can be used
for both classification or regression challenges.
However, it is mostly used in classification
problems.
Introduction

● Key Idea: Finding the hyperplane that best separates data into classes.
● In this algorithm, we plot each data item as a point in n-dimensional space
(where n is number of features you have) with the value of each feature
being the value of a particular coordinate.
● Then, we perform classification by finding the hyperplane that differentiate
the two classes very well.
Hyperplane in SVM
How SVM works ?

● The main objective is to segregate the given dataset in the best possible
way.
● The distance between the either nearest points is known as the margin.
● The objective is to select a hyperplane with the maximum possible margin
between support vectors in the given dataset. SVM searches for the
maximum marginal hyperplane in the following steps:
– Generate hyperplanes which segregates the classes in the best way.
– Select the right hyperplane with the maximum segregation from the
either nearest data points.
Non-linear and inseparable planes

● Some problems can’t be solved using linear hyperplane.

● In such situation, SVM uses a kernel trick to transform the input space to a
higher dimensional space as shown on the right.
● The data points are plotted on the x-axis and zaxis (Z is the squared sum
of both x and y: z=x^2=y^2).
● Now you can easily segregate these points using linear separation.
Non-linear and inseparable planes
Soft Margin Vs Hard Margin
Learning a linear SVM with quadratic
programming
Quadratic programming (QP) is a technique for optimizing a quadratic
objective function, subject to certain linear constraints.
There is a large number of QP solvers available, for example GNU Octave’s
qp, MATLAB’s Optimization Toolbox, Python’s CVXOPT framework etc., and
they are all available within the Domino Data Science Platform.
For this tutorial we will use CVXOPT. In order to use convex optimization, we
first need to construct a Lagrangian function of the constrained-optimization
problem.
where w are the
weights and b is the
bias meant to be
learned. This can be
solved using any
quadratic
programming solver,
but we will transform
this constrained
problem into its dual
using Lagrange
multipliers
We again transform this problem using Lagrange multipliers
and get the dual as :
Soft margin can be also solved using gradient descent by minimizing the loss function below :
Quadratic Programming using CVXOPT

CVXOPT is an optimization library in python. We can use qp solver of

CVXOPT to solve quadratic problems like our SVM optimization problem.
We just need to create matrices P, q, A, G, h and initialize a value for b.

Comparing our optimization problems to the figure above, we can easily

deduce the values of these matrices.
SVM Kernels

● The SVM algorithm is implemented in practice using a kernel. A kernel

transforms an input data space into the required form.
● SVM uses a technique called the kernel trick. Here, the kernel takes a
low-dimensional input space and transforms it into a higher dimensional
space.
● In other words, you can say that it converts non-separable problem to
separable problems by adding more dimension to it.
● It is most useful in non-linear separation problem. Kernel trick helps you to
build a more accurate classifier.
Kernel Types
● Linear Kernel
1. The linear kernel is the simplest and most straightforward kernel function.
2. This kernel is used when the data is already linearly separable. It
effectively means that no transformation is applied to the data.
● Polynomial Kernel
1. The polynomial kernel allows for more complex decision boundaries by
adding polynomial features to the data. It is defined as:
2. This kernel can capture interactions between features up to a certain
degree.
● Radial Basis Function (RBF) Kernel
1. The RBF kernel, also known as the Gaussian kernel, is a popular choice
due to its flexibility. It is defined as:
2. This kernel can handle very complex and non-linear relationships.
● Sigmoid
1. The Sigmoid Kernel introduces non-linearity by squashing the feature
space into a bounded range (-1 to 1) through the hyperbolic tangent
function.
2. It is similar to the activation function used in artificial neural networks
(ANNs), making it a bridge between SVMs and ANNs.
Basic Parameters for SVM

● We thought we have plenty of parameters option, we will be using only

very few alone because of its importance and for its impact,
1. Regularization parameter (C)
2. Gamma parameter
3. Kernel
4. Degree
Regularization parameter C

1. The C parameter in SVM is mainly used for the Penalty parameter of the
error term.
2. You can consider it as the degree of correct classification that the
algorithm has to meet or the degree of optimization the SVM has to meet.
3. Controls the tradeoff between the classification of training points
accurately and a smooth decision boundary or in a simple word, it suggests
the model choose data points as a support vector.
 Regularization parameter C
For large C – then model choose For small C – If the value of C is
more data points as a support vector small then the model chooses fewer
and we get the higher variance and data points as a support vector and
lower bias, which may lead to the gets lower variance/high bias.
problem of overfitting

The value of gamma and C should not be very high because it leads to overfitting
or it shouldn’t be very small (underfitting). Thus we need to choose the optimal
value of C.
Gamma Parameter:

1. Gamma is used when we use the Gaussian RBF kernel.

[Link] you use linear or polynomial kernel then you do not need gamma only you
need C hypermeter.
3. It decides that how much curvature we want in a decision boundary.
4. High Gamma value – More curvature
5. Low Gamma value – Less curvature
Gamma Parameter:
Degree:

1. It controls the flexibility of the decision boundary.

2. Higher degrees yield more flexible decision boundaries.
3. Highly recommended for polynomial kernel