1.

Why Soft Margin SVM is needed

� Real-world data is not perfectly separable.
Points overlap, have noise, errors, and there is no clean straight-line separation.
So Hard-Margin SVM (which requires perfect separation) fails.
Solution:
Use Soft Margin SVM
Soft Margin SVM allows some misclassification.

Soft Margin SVM

• Real-world datasets are rarely perfectly linearly separable; they are often

nearly or completely non-linearly separable.

• Methods for linearly separable data don’t work for these cases.

• Support Vector Machines (SVM) are a powerful machine learning tool.

• SVMs can handle both nearly and non-linearly separable datasets.

• They provide effective solutions for classification problems in diverse real-

world scenarios.
To tackle this problem what we do is modify that equation in such a way that it
allows few misclassifications that means it allows few points to be wrongly
classified. We know that max[f(x)] can also be written as min[1/f(x)], it is common
practice to minimize a cost function for optimization problems; therefore, we can
invert the function. To make a soft margin equation we add 2 more terms to this
equation 1 which is zeta (or slack variable) and multiply that by hyperparameter
‘c’
The first term in the above equation which is ||w||/2 is used to minimize the margin error or

increase the margin (distance between hyperplane and the support vector): During SVM training,

+ve points are correctly classified if they lie above the positive hyperplane and -ve points are

correctly classified if they lie below the negative hyperplane.

hyperparameter C controls margin vs misclassification.

Case 1 : If we increase the value of C which is a hyperparameter to a very big value, it

means we are giving more importance to classification error and less importance to

margin error, which means the margin will be very small. Let’s say you take a high

value of ‘c’ =1000, this would mean that you don’t want to focus on margin error and

just want a model which doesn’t misclassify any data point.

Case 2 : If we decrease the value of C say 0.1, 0.01, then more importance is given to

margin error which means the margin will increase but number of classifications

errors will also increase

If someone asks you which is a better model, the one where the margin is

maximum and has 2 misclassified points or the one where the margin is very less,

and all the points are correctly classified?

Well, there’s no correct answer to this question, but rather we can use SVM Error

= Margin Error + Classification Error to justify this. If you don’t want any

misclassification in the model then you can choose figure 2. That means we’ll

increase ‘c’ to decrease Classification Error but if you want that your margin should
be maximized then the value of ‘c’ should be minimized. That’s why ‘c’ is a

hyperparameter and we find the optimal value of ‘c’ using GridsearchCV and cross-

validation.

1. Ideal, perfectly separable case (no mistakes)

In a perfect world:

✔ Positive class (triangles)

𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 ≥ +1

✔ Negative class (crosses)

𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 ≤ −1

All positive points lie above the +1 line,

All negative points lie below the –1 line.

But real data is NOT perfect, and your diagram correctly shows that.

2. What about points that are:

• Positive class triangles BELOW the +1 hyperplane?

• Negative class crosses ABOVE the –1 hyperplane?

These points do not satisfy the margin constraints.

These are NOT outliers.

These are NOT test mistakes.
These are margin violations during training.
SVM knows that real data may not be perfectly separable.
So SVM introduces slack variables (ξᵢ) to deal with these points.

3. Soft-Margin SVM — the real-world SVM

Soft-margin SVM allows two kinds of violations:
A. Inside the margin, but on the correct side
Example:
• A positive triangle below the +1 hyperplane
• A negative cross above the –1 hyperplane
These points are correctly classified but too close to the boundary.
SVM says:
“OK, I allow this, but I will penalize it slightly.”
These points have:
0 < 𝜉𝜉𝑖𝑖 < 1

B. Completely misclassified points

Example:
• A positive triangle below the decision boundary
• A negative cross above the decision boundary
These points are classified incorrectly.
SVM says:
“I will allow some misclassifications, but I will penalize them more.”
These points have:
𝜉𝜉𝑖𝑖 > 1
SVM calls all these points as Margin violators and assigns them a slack penalty in the
optimization.

SVM identifies these points (margin violators) during training, not during testing.
During training, SVM:
• Detects which points violate the margin
• Assigns slack ξᵢ to them
• Penalizes them in the objective function
✔ The triangle points below the +1 hyperplane
and
✔ The cross points above the –1 hyperplane
are not outliers.
They are valid training data that:
• violate the margin
• but SVM handles them using slack variables
• and penalizes them without removing them or treating them as test errors
These are margin violation points, not test mistakes and not outliers.

1. HARD MARGIN SVM (Perfect, noise-free world)

Hard-margin SVM assumes:

✔ Data is perfectly linearly separable
✔ No overlap between the two classes
✔ No noise
✔ Every point must lie outside its correct margin
✔ NO violations allowed

So in hard-margin SVM:

� What happens to points inside the margin in your diagram?

In your sketch:

• Some triangles lie below the +1 hyperplane

• Some crosses lie above the –1 hyperplane
• Some triangles and crosses even come close to (or touch) the decision boundary

In Hard-Margin SVM, ALL these points are illegal.

Why?

Because hard-margin SVM requires:

𝑦𝑦𝑖𝑖 (𝑤𝑤 ⋅ 𝑥𝑥𝑖𝑖 + 𝑏𝑏) ≥ 1∀𝑖𝑖

No exceptions.

� Points that violate this in hard margin cannot exist

For hard-margin SVM:
• A positive point must be above +1
• A negative point must be below –1
Any point inside the margin:
• Would destroy separability
• Would violate constraints
• Would make optimization infeasible
So:

� Hard-margin SVM fails completely on your diagram.

Hard-margin SVM cannot be trained on such data.

2. SOFT MARGIN SVM (Real-world, noisy, overlapping data)

Soft-margin SVM allows violations using slack variables 𝜉𝜉𝑖𝑖 :

𝑦𝑦𝑖𝑖 (𝑤𝑤 ⋅ 𝑥𝑥𝑖𝑖 + 𝑏𝑏) ≥ 1 − 𝜉𝜉𝑖𝑖

Where:

• 0 < 𝜉𝜉𝑖𝑖 < 1→ inside margin but correctly classified

• 𝜉𝜉𝑖𝑖 > 1→ misclassified point
• 𝜉𝜉𝑖𝑖 = 0→ perfect margin point
• Larger 𝜉𝜉𝑖𝑖 → bigger violation
• Penalty controlled by C

What happens to the specific points in your diagram?

(A) Triangles below +1 but still above 0 (decision boundary)
These are:
• “Too close to the boundary”
• Correctly classified
• But inside margin
➡ Soft-margin SVM allows these.
➡ Slack value: 0 < 𝜉𝜉 < 1

(B) Crosses above –1 but still below 0

Same as above:
➡ Allowed
➡ Margin violation
➡ Slack 0 < 𝜉𝜉 < 1
(C) Points on the wrong side of the decision boundary

✔ Soft-Margin SVM summary

Type of point in your diagram Hard margin SVM Soft margin SVM
Correctly classified + outside margin Allowed Allowed
Correctly classified but inside margin � Not allowed ✔ Allowed (ξ between 0 and 1)
Misclassified points � Not allowed ✔ Allowed (ξ > 1)
Noisy data, overlaps � Not allowed ✔ Allowed
Real-world messy data � Impossible ✔ Works perfectly

Inside-margin points do NOT sit ON the margin hyperplanes, but they DO influence the final
position of the hyperplanes through the optimization process.

1. The +1 and –1 hyperplanes are defined by w and b

These equations define them:
𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = +1
 𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = −1

But these hyperplanes are not fixed.

They depend entirely on:
• w
• b
And w and b are learned using ALL TRAINING POINTS, including:
• Perfect points (ξ = 0)
• Inside-margin points (0 < ξ < 1)
• Misclassified points (ξ > 1)
So inside-margin points affect w, which means they affect the final hyperplanes.

2. But… inside-margin points are not on the ±1 lines

Inside-margin points are:
• Positive points between 0 and +1
• Negative points between –1 and 0
These points are not support vectors on the margins, because support vectors ON the margins
satisfy:
𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = ±1
3. So do they affect the hyperplanes?

� They do NOT define the margin directly (since they are not exactly on ±1)
✔ But they DO influence the position of the margin
Why?
Because the optimization objective includes slack:
1
min ∥ 𝑤𝑤 ∥ + 𝐶𝐶∑𝜉𝜉𝑖𝑖
2

Whenever a point is inside the margin (0 < ξ < 1):

• A penalty is added
• This influences the optimal w and b
• And therefore, moves the hyperplanes a little

4. Who defines the margin exactly?

In Soft-Margin SVM:
✔ Support vectors with 0 ≤ α ≤ C
including:
• points ON the margin (ξ = 0, “classic” support vectors)
• points INSIDE the margin (0 < ξ < 1)
• misclassified points (ξ > 1)
These points all influence the solution.
But only points that satisfy:
𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = +1 or 𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = −1

are geometrical support vectors lying ON the margin.

First: What is α (alpha)?
αᵢ is a Lagrange multiplier that comes from the dual form of the SVM optimization problem.
But you can think of αᵢ in simple English as:
αᵢ tells you how important each point is in deciding the SVM boundary.
If αᵢ = 0 → the point does NOT influence the boundary
If αᵢ > 0 → the point DOES influence the boundary

Second: Why 0 ≤ α ≤ C ?
In hard-margin SVM, αᵢ ≥ 0 (because C = ∞).
But in soft-margin SVM, αᵢ is limited:
0 ≤ 𝛼𝛼𝑖𝑖 ≤ 𝐶𝐶

Why?
Because:
• C is the maximum influence a point is allowed to have
• C controls how much penalty the model allows
• If α exceeds C, the point would be "over-punished"
So in soft-margin SVM:
αᵢ has a ceiling limit = C.

� Third: What does α tell us about each point?

This is the key part.
There are 3 types of points, and each type has a different α value.

� Case 1: α = 0 → Point is NOT a support vector

These are:
• Far from boundary
• Correctly classified
• Outside the margin
These points DO NOT influence w or the margin.
α = 0 → Irrelevant point

� Case 2: 0 < α < C → TRUE Support Vectors

These points are:
• Exactly ON the margin
• They satisfy:
𝑦𝑦𝑖𝑖 (𝑤𝑤 ⋅ 𝑥𝑥𝑖𝑖 + 𝑏𝑏) = 1

These points define the margin lines:

𝑤𝑤 ⋅ 𝑥𝑥 + 𝑏𝑏 = ±1
These are the geometric support vectors.
0 < α < C → This point DEFINES the margin exactly.

� Case 3: α = C → Margin violators / Misclassified points

These are:
• Inside the margin (0 < ξ < 1)
• Or misclassified (ξ > 1)
• Heavy violators → α hits the maximum allowed value = C
They influence w, but they do NOT lie on the margin.
α = C → Violator point (inside margin or misclassified).

Putting It All Together: Who Defines the Margin?

✔ ONLY points with 0 < α < C
These lie exactly on ±1 hyperplanes.
These points satisfy:
𝑦𝑦𝑖𝑖 (𝑤𝑤 ⋅ 𝑥𝑥𝑖𝑖 + 𝑏𝑏) = 1

These are the true geometric margin-defining support vectors.

Inside-margin violators (α = C) do influence the boundary, but they do NOT define the ±1 margin
hyperplanes. Outside-margin points (α = 0) do nothing.

Simple Example
Imagine a positive point lies slightly inside the margin (at 0.7 instead of 1.0):
SVM notices it is inside the margin → assigns slack
Slack contributes to cost → Optimization pushes w and b slightly to reduce the violation.
So the hyperplane shifts a little.
That is how inside-margin points influence the final model.

Inside-margin points do not define the margin, but they influence the margin by
affecting w during optimization.
• Blue line = decision boundary

• Dashed lines = +1 and –1 hyperplanes

• Points on dashed lines = TRUE support vectors

• Points inside margin = violators (0 < ξ < 1)

• Misclassified points (ξ > 1) shown on wrong side

Inside-margin points are NOT on ±1 hyperplanes BUT they influence the model because they add
slack penalty.

Influence of C on how much inside-margin points shift the margin

• Small C → margin expands, violators allowed

• Big C → margin shrinks, violators punished
• Inside-margin points matter more when C is large

Key idea:

Inside-margin points affect w more when C is high, and less when C is low.
• Points ON ±1 hyperplanes → define the margin (true support vectors)

• Points inside margin (0 < ξ < 1) → do not lie on boundary but influence w

• Misclassified points (ξ > 1) → influence w strongly

• w changes → hyperplanes shift, so inside-margin points DO affect the final hyperplanes

• But the inside-margin points themselves do NOT define the margin equations

α = 0 → useless point
• 0 < α < C → margin-defining support vector
• α = C → margin violator / misclassified

And:

Only points with 0 < α < C define the margin.