Gradient Descent and Convergence Modes

1. Gradient Descent and Convergence Modes

Goal: Estimating the Optimal Filter Weights

We're in the setting of designing an adaptive filter that tries to minimize the mean
square error (MSE) between its output and a desired signal. Mathematically, we want to
find a weight vector 𝑤 ₒₚₜ such that the error:
T 2
J (w) = E[(d[n] − w x[n]) ]

is minimized. Here:

d[n] = desired output at time n,

x[n] = input vector (e.g., recent input samples),
w = filter coefficient vector.

This function J (w) is called the cost function. It's a bowl-shaped quadratic function
(paraboloid in higher dimensions), and our goal is to find the bottom of the bowl — where
the error is minimized.

Gradient Descent Update Rule

The general update rule is:

(i+1) (i) (i)
w = w − μ∇J (w )

Where:

w
(i)
= weight vector at iteration i,
μ = step size (learning rate),
∇J (w
(i)
) = gradient of the cost function at the current weights.

If μ is too small: convergence is very slow.

If μ is too large: you might overshoot or diverge.

What’s the Gradient for Our Cost Function?

The cost function:

T 2
J (w) = E[(d[n] − w x[n]) ]
Its gradient with respect to w is:

∇J (w) = −2r xd + 2R x w

Where:

R x = E[x[n]x
T
[n]] is the autocorrelation matrix of the input (also called R in the y

video),
r xd = E[x[n]d[n]] is the cross-correlation vector between input and desired output.

Setting this gradient to zero gives the Wiener-Hopf equation:

R x w opt = r xd

Solving this gives the optimal weights directly, but it's computationally expensive
(inverting matrices) — so we use iterative methods like gradient descent.

Weight Error Vector and Convergence

Let’s now define an important quantity:

(i) (i)
Δw = w − w opt

This is the error in the weights at iteration i (each step). Here, we track how far w is
from the optimal value. This error evolves with time when using gradient descent and
becomes:
(i+1) (i)
Δw = (I − μR x )Δw

How?

We start from:

(i+1) (i) (i)

w = w + μ [r xd − R x w ]

Substitute r xd = R x w opt (from the Wiener-Hopf equation):

(i+1) (i) (i) (i) (i) (i) (i)

w = w + μ [R x w opt − R x w ] = w + μR x (w opt − w ) = w − μR x (w − w opt )

Rewriting this:
(i+1) (i) (i)
w = w − μR x Δw

Now subtract w opt from both sides:

(i+1) (i+1) (i) (i) (i) (i) (i

Δw = w − w opt = [w − μR x Δw ] − w opt = Δw − μR x Δw = (I − μR x )Δw

This recurrence tells you how the error evolves from one step to the next.
 If μ is chosen correctly, then (I − μR x
) acts like a shrinking operator, reducing the
error at each step.

So in each iteration:

The error vector is multiplied by (I − μR ). Hence after each iteration, the power of
x

(I − μR x ) increases.
This recursive relationship describes how error shrinks over time.

Now here’s the catch: understanding how a matrix raised to a power like (I − μR ) x
i

behaves over time isn’t straightforward. That’s where eigenvalue decomposition helps.

Spectral Decomposition: Simplifying the Math

Since R (aka R ) is a real symmetric matrix, we can write it as:

x y

T
R x = QΛQ

Where:

Q is an orthonormal matrix of eigenvectors,

Λ is a diagonal matrix of eigenvalues.

And since QQ = I , this decomposition allows us to switch to a rotated coordinate

T

system where everything is easier.

Let’s define:
(i) T (i)
v = Q Δw

This is like rotating the error vector into a new basis - the eigenvector basis of R . Now x

the recursion becomes:

(i+1) (i)
v = (I − μΛ)v

Because everything is diagonal now, this means:

(i+1) (i)
v = (1 − μλ k )v
k k

Where λ is the k-th eigenvalue of R .

k x

This tells us: each component of the error decays independently, and the rate
depends on μλ . k
Modes of Convergence

v = Error vector in the eigenvector basis of R

(i)
x

Each v is called a mode/component. The convergence of each mode is geometric:

k

(i) i (0)
v = (1 − μλ k ) ⋅ v
k k

So:

If |1 − μλ k
, the mode decays to zero.
| < 1

If it’s close to 0, it decays quickly.

If it’s close to 1, it decays slowly.
If it's > 1 or < -1, the error grows — divergence!

To guarantee convergence, we need:

2
0 < μ <
λ max

Where λ is the largest eigenvalue of R . That’s the most conservative bound — it

max x

ensures that even the slowest-decaying mode won’t blow up.

Gradient descent (in the least-squares setting) converges if and only if the step size
μ satisfies:

2
|1 − μλ k | < 1 ∀k ⟺ 0 < μ <
λ max

This ensures each mode of the error vector shrinks over time, leading the weight vector
(i)
w → w opt

Error Energy and Convergence Curve

We can now express the total error energy as:

(i) (i)T (i) (i)T (i)
E = Δw R x Δw = v Λv

Which expands to:

2
(i) (0) i
E = ∑ λ k (v (1 − μλ k ) )
k

Each mode contributes a term that decays as:

2i
(1 − μλ k )

So on a plot of error vs. iteration:

Initially, modes with larger λ (faster decay) dominate.

k
 Over time, slower modes (smaller λ ) dominate the tail of the curve.
k

You get a long tail — convergence slows down.

This explains the "elbow shape" of the learning curve — fast decay at first, then slow.

What Happens Visually?

Imagine plotting v vs i (number of iterations):

(i)

For a well-chosen μ, you’ll see smooth exponential decay like:

|\
| \
| \
| \
| \
| \
| \_______
|
+------------------->
Iterations

If μ is too large, the curve may oscillate (flip sign) or diverge:

| /\
| / \
| / \
| / \
| / \______
|
+------------------->

Ideal Case vs. Reality

Best case:

All λ are equal ⇒ All modes decay at the same rate.

k

You can choose μ = 1

λ
and converge quickly.

Real-world case:
 Eigenvalues are spread out ⇒ No perfect μ.
Choosing μ to make fast modes decay quickly causes slow modes to decay painfully
slowly.

Fastest Convergence
2. Fastest Convergence

Recap: Why are we even talking about modes?

We had:

(i) i (0)
v = (1 − μλ k ) v
k k

This tells us that each component of the error vector decays exponentially over time —
how fast it decays depends on:

the step size μ,

the eigenvalue λ of the autocorrelation matrix R .
k x

So the rate of convergence in each mode depends on |1 − μλ |. We want this value to k

be less than 1 so it shrinks over time.

Objective

We want to choose μ so that all modes converge as fast as possible.

To do that, we:

Look at how each mode decays: |1 − μλ k

|
i

Want to minimize the slowest-decaying mode

What the diagram shows (Refer the video)

The professor plots:

|1 − μλ k | vs μ

for different values of λ (each eigenvalue).

k

Each curve looks like a V shape:

At μ = 0, all |1 − μλ k
| = 1 → no convergence.
 As μ increases:
|1 − μλ k | decreases (good), reaches a minimum, and then increases again (bad).
If μ becomes too large, |1 − μλ k| , and we get divergence.
> 1

So each eigenvalue λ gives us a curve (V-shape), and we want to pick a single value
k

of μ that makes all these curves as low as possible — i.e., all error components
converge fast.

Best strategy

We ask:

What value of μ minimizes the worst case value of |1 − μλ k

| across all k?

This is a minimax problem:

We want the maximum of all |1 − μλ k

| to be as small as possible.

This minimum of the maximum happens at the point where the slowest and fastest
modes decay at the same rate:

|1 − μλ max | = |1 − μλ min |

Solve for optimal μ

From the equation above:

1 − μλ max = −(1 − μλ min )

(We put a minus sign to reflect symmetry — one side is decreasing, the other increasing)

Simplify:

1 − μλ max = −1 + μλ min

2 = μ(λ max + λ min )

2
μ opt =
λ max + λ min

This is the best step size for fastest convergence across all error modes.

Summary
 Each component of the error decays like (1 − μλ k
)
i

The closer |1 − μλ k| is to 0, the faster the decay

But if it’s >1, it diverges
So we choose μ so that all components converge quickly and safely
The sweet spot is when both fastest and slowest modes converge equally fast
This gives:

2
μ opt =
λ max + λ min

Summary of the last two notes (1&2):

Gradient descent is the algorithm to go to in order to find the best optimal weights
to transform the noisy input into the desired signal.
You can use it in either wiener filter or any other adaptive filters. In case of wiener
filters, using the Wiener-Hopf equation can be too expensive computationally. So,
using gradient descent instead to get the optimal weights works better.
To ensure that all the weights converge, the step size should be:

2
0 < μ <
λ max

which is the maximum eigen value of the autocorrelation matrix.

We can get the fastest convergence if step size :

2
μ opt =
λ max + λ min

All those mathematical derivations were just to get to this point! To know what the
range of step size should be in order for the weights to converge and what the
value of step size should be to get the fastest convergence.

LMS Adaptive Filter

3. LMS Adaptive Filter

Goal of LMS

We want to find the optimal filter weights w to minimize the mean squared error
opt

(MSE) between the desired signal d(n) and the filter output y(n).

Normally, we could do this with the Wiener-Hopf equation, but computing the
autocorrelation matrix R and its inverse is expensive.
x

So, LMS is a clever approximation of gradient descent that avoids all that.
Intuitive LMS Idea in 5 Key Steps:

1. Why It’s Called LMS

L: Least → we're minimizing something.

M: Mean → expected value over time.
S: Squares → squaring the error (d(n) − y(n)) . 2

So we’re minimizing the expected value of squared error:

2 T 2
E[e(n) ] = E[(d(n) − w x(n)) ]

2. Why LMS is "Approximate" Gradient Descent

In ideal gradient descent, we would compute the gradient:

2
∇E[e(n) ] = −2E[e(n)x(n)]

but this needs an expectation → i.e., averaging over lots of data.

LMS shortcut: “Let’s just use one sample at time n”:

2
∇E[e(n) ] ≈ −2e(n)x(n)

That's the crazy but clever approximation #1:

We treat one noisy sample as if it were the expectation.

3. One Step Per Sample

In theory, gradient descent would do multiple steps per sample to reach w opt .
LMS only takes one step per sample, then moves on to the next one.

That’s approximation #2:

w(n + 1) = w(n) + μe(n)x(n)

So, at every new time step, you just:

1. Compute the output y(n) = w(n) T

x(n)

2. Compute error e(n) = d(n) − y(n)

3. Update weights with that error and input

4. What Makes LMS So Attractive

 No matrix inversion, no autocorrelation matrix, no big calculations.
Just vector additions and multiplications.
Very efficient — great for real-time systems like noise cancellation, echo
suppression, or system ID.

5. Conceptual Understanding of Error Convergence

Initially, the weights are bad → high error.

With every new sample, LMS takes one small step toward optimal weights.
Over time, weights converge (assuming step size μ is well chosen), and error
decreases.

So you don’t get the best performance immediately, but you learn as you go.

Why This Works (Despite Being Approximate)

Even though each weight update is based on noisy info (just one sample), over time,
these updates act like a running average of the true gradient:
n

w(n) ≈ something like w opt + μ ⋅ ∑ e(i)x(i)

i=0

So slowly, you’re estimating the ideal gradient by just accumulating over time.

Final LMS Summary (Core Equations)

At each time step n, LMS does:

1. Compute filter output:

T
y(n) = w(n) x(n)

2. Compute error:

e(n) = d(n) − y(n)

3. Update weights:

w(n + 1) = w(n) + μe(n)x(n)

That’s it. That's LMS in three steps.

 Application Example: System Identification

Say you have an unknown system (like room acoustics). If you:

Feed in a known signal (e.g., white noise)

Observe the system output (microphone signal)
Use LMS to match that output using your adaptive filter

Eventually, the filter weights will converge to approximate the unknown system’s
impulse response.

Summary

In case of Wiener filter, we needed the autocorrelation matrix in order to compute

weights.
Even with gradient descent, we used eigenvalues of the autocorrelation matrix to
update the weights.
This is not practical in real time applications. That's where LMS is useful.
Instead of calculating the expected value of the samples (a lot of samples), we
make use of every sample that we get (noisy or clean) to update the weights.
This saves a lot of computation. No matrix inversion, no eigenvalues, no
expectations — just vector dot products and scalar multiplications. Extremely
efficient.
But the thing is, this takes time to give out the perfect output. The more samples it
gets over time, the better will be its performance.

LMS Convergence
4. LMS Convergence

1. Why LMS was introduced:

In Wiener filter theory, to find the optimal weights w opt

, we need to know:
Autocorrelation matrix R x = E[x(n)x
T
(n)]

Cross-correlation r dx = E[d(n)x(n)]

These expectations require lots of data and can be computationally expensive

(especially due to matrix inversion).

2. LMS: A Shortcut Approach

 Gradient descent gives us a way to iteratively find the optimal weights, but:
It still uses expectations (which are hard to compute in real-time).
LMS replaces the expected value E[e(n)x(n)] with a single sample approximation:

w(n + 1) = w(n) + μe(n)x(n)

This is cheap to compute, requiring just:

1 multiplication (step size),

1 inner product (error computation),
1 vector addition.

3. Convergence – The Problem and the Shift to Expectation

LMS uses noisy estimates to update weights.

So, each weight update "jumps" around.
Hence, it's hard to prove convergence to the exact Wiener solution w opt
.
Instead, we ask:
Does the expected value of the weights, E[w(n)], converge to w opt ?
In other words - If I keep running LMS on more and more samples (for large n), will
my weight vector approach the optimal Wiener solution on average?

4. Dealing with E[e(n)x(n)]

Substituting error: e(n) = d(n) − w T

(n)x(n)

So:

E[w(n + 1)] = E[w(n)] + μ (r dx − R x E[w(n)])

This is similar to steepest descent, but with expectations.

5. Independence Assumption

w(n) is dependent on past values of x(n), since it is updated using

x(n − 1), x(n − 2), …

To simplify analysis, we assume independence between x(n) and w(n).

This is an approximation, justified when:
μ is small (so each update step is tiny), or...
x(n) is white (uncorrelated samples).
6. Back to Familiar Ground:

Now we can analyze LMS similarly to steepest descent:

E[w(n + 1)] = (I − μR x )E[w(n)] + μr dx

In rotated coordinate system (eigenbasis of R ): x

E[v k (n + 1)] = (1 − μλ k )E[v k (n)] + constant

where v k (n) is the projection of the weight vector in the eigen-direction (aka mode).

Woah - why are we dealing with Rx , eigenbasis and λk again? I thought we were
done with that?

In LMS, the weights are updated using every sample we get (noisy or clean), no
doubt.
But when it converges to optimal weights, then those weights are similar to the
weights we get when we use the traditional method to compute weights (using R x

etc.,).
We use R etc., again just to study and analyze the behavior, not for
x

computation.
And in order to analyze the behavior, we have come back to diagonalization,
eigenvalue and change of basis for simplification of our analysis.

7. What Does "Converges in the Mean" Mean?

LMS does not guarantee that w(n) → w opt

for every run.
But on average across many trials, E[w(n)] → w opt

8. Step Size μ Matters:

If μ is too large:
We get faster learning, but more oscillations and possibly instability.
If μ is small:
LMS learns slowly, but is more stable and variance is lower.
The condition for mean convergence:

2
0 < μ <
λ max
 But in practice, we choose μ ≪ 1
λ max
to also control variance.

In real-time or practical application, you don't compute λ directly. Instead, you use
max

heuristics or prior knowledge to choose a step size μ that’s small enough to

ensure stability and convergence — without knowing λ exactly.
max

The theoretical bound is used only for analysis, to understand the behavior, not in
real-time computation.

9. Why Do the Weights Keep Jumping?

Because we’re using a single sample (not the expected value) to update weights.
This single sample is random, so the update is noisy.
Even near convergence, LMS weight updates will bounce around the optimal
solution.

Summary:

LMS sacrifices precision (true expectations) for efficiency and simplicity.

It converges in the mean, assuming a small enough step size.
Variance in the weights is the next concern, which will be analyzed in future.

LMS Excess Error

5. LMS Excess Error

This is a dense and math-heavy lecture, but the main ideas and intuition, skipping
the long derivations and focusing on what the results mean and why they matter in
understanding LMS behavior is presented.

Goal of the Lecture:

To understand the Excess Mean Squared Error (EMSE) of LMS.

We already know that:

LMS converges in the mean, i.e., average filter weights converge to optimal weights.
But even after convergence, the weights keep bouncing around due to the
stochastic nature of the algorithm.
 So we want to quantify how much error remains due to this bouncing — this is
EMSE.

Step-by-step Intuition

1. Two sources of error in LMS:

At any time, the output error e(n) consists of:

v(n): the unavoidable error — even with the best filter you can't predict this (e.g.,
noise, randomness).
e a (n): the excess error — due to LMS weights not being exactly optimal.

e(n) = v(n) + e a (n)

So:
2 2 2
MSE = E[e (n)] = E[v (n)] + E[e a (n)]



      
minimum possible MSE EMSE

2. Why is EMSE nonzero?

Even after convergence, the filter weights keep changing slightly (due to random input
x(n) and error signals). This “misadjustment” causes small extra error, which is what

we’re trying to analyze.

3. We need assumptions to make analysis tractable:

To simplify the math, the following assumptions are made:

All signals are zero-mean.

The unavoidable error v(n) is white, stationary, and independent of the input x(n).
LMS has converged, i.e., we're in steady-state.

4. We define EMSE and derive an approximate expression for it:

The result (after lots of derivations) is:

μ 2
EMSE ≈ ⋅ σ v ⋅ Tr(R x )
2
Where:

μ is the LMS step size.

σv
2
is the variance (power) of the unpredictable part of the signal.
Tr(R x ) is the trace of the input autocorrelation matrix (i.e., sum of eigenvalues —
total signal power spread across modes).

What does this formula tell us?

1. Smaller μ ⇒ smaller EMSE.

​ Because weights fluctuate less.
But too small ⇒ slow convergence.
2. More input power ⇒ more EMSE.
​ Bigger trace means more energy in the input → bigger wiggles in weight update
→ more error.
3. Larger filters (more taps) ⇒ usually larger EMSE.
​ Because Tr(R ) grows with filter size.
x

4. There's a trade-off:
​ Bigger μ: faster adaptation but larger steady-state error.
Smaller μ: slower adaptation, smaller error.

Introduction of β (Normalized Step Size)

They define:

2β
μ = with 0 < β < 1
Tr(R x )

So that the EMSE becomes:

2
EMSE ≈ β ⋅ σ v

This is very nice because now:

EMSE is controlled by β, which is dimensionless and easy to tune.

You can forget about R , μ, eigenvalues — just set β ≈ 0.05 to 0.5, and you're good.
x

Great, but how do we use β in the LMS equation?

The original LMS update equation is written in terms of the step size μ, like this:
 w(n + 1) = w(n) + μ e(n) x(n)

Now, to incorporate the normalized step size β, we simply redefine μ using:

2β
μ =
Tr(R x )

So the update becomes:

2β
w(n + 1) = w(n) + ( )e(n) x(n)
Tr(R x )

This means:

You don’t change the LMS equation itself — you just plug in this expression for μ.
If you don’t know Tr(R ), then you estimate it or replace it with something easy to
x

compute, like ∥x(n)∥ . That leads us to...

2

Normalized LMS (NLMS)

When you don’t know Tr(R ), you can normalize using the current input vector energy:
x

β
μ(n) =
∥x(n)∥ 2 + ϵ

Then the update becomes:

β
w(n + 1) = w(n) + ⋅ e(n) ⋅ x(n)
2
∥x(n)∥ + ϵ

Here, ϵ is a small positive number (like 10 ) to avoid division by zero.

−6

This is the Normalized LMS (NLMS) algorithm. (Detailed explanation comes later)

LMS misadjustment ratio

6. LMS misadjustment ratio

Goal: How far are we from ideal performance?

In theory, the ideal performance of our adaptive filter is to reach the minimum mean
squared error (MMSE) — denoted as:
2
MMSE = σ v

But in practice, the LMS algorithm doesn't settle exactly at MMSE. Due to the continual
adaptation (i.e., small fluctuations in weights), there’s a small persistent error left —
called the Excess Mean Squared Error (EMSE).
Misadjustment Ratio (M)

The misadjustment ratio tells us:

How big is this extra error compared to the minimum error?

Mathematically:

EMSE ξ ex
M = =
2
MMSE σv

If M = 0, the algorithm performs perfectly (no excess error).

If M = 0.1 , you're 10% worse than ideal.
If M = 1, your excess error is as big as the minimum possible error — not great.

Expression for Misadjustment

From previous derivations, we know:

2
ξ ex = μ ⋅ σ v ⋅ Tr(R x )

So plugging into the formula:

2
μ ⋅ σ v ⋅ Tr(R x )
M = = μ ⋅ Tr(R x )
2
σv

Thus:

M = μ ⋅ Tr(R x )

or sometimes written in an equivalent form:

μ ⋅ Tr(R x )
M =
2 − μ ⋅ Tr(R x )

(both are valid under small-μ approximations)

Intuition

When μ → 0: M → 0 → minimal excess error (ideal performance).

When μ increases: M increases → more jitter, more deviation from ideal behavior.
But you can't keep reducing μ forever:
If μ is too small → filter becomes too slow to adapt.
If μ is too large → filter diverges.
 Practical Trick — Using β

To make life easier, we define:

2β
μ = ⇒ M ≈ β
Tr(R x )

This way, you don't need to calculate misadjustment every time. You choose β (say,
0.1 for 10% misadjustment), and compute μ accordingly.

Example (from video)

Suppose two eigenvalues of R : λ x 1 = 20 ,λ 2 = 1.5

Then: Tr(R x) = 21.5

Choose β = 0.1 ⇒ μ = 2⋅0.1

21.5
≈ 0.0093

Predicted misadjustment:
M = μ ⋅ Tr(R x ) = 0.0093 ⋅ 21.5 ≈ 0.2

(which shows that with a higher β, misadjustment increases)

If you try μ = 0.5, then μ ⋅ Tr(R ) = 10.75 > 2, violating stability bounds
x

= 0.1. So convergence fails.

2 2
μ < ⇒ μ <
λ max 20

NLMS Algorithm
7. NLMS Algorithm

1. LMS has problems choosing step size μ

"Mu must be chosen to be sufficiently small... values of μ are dependent on the

eigenvalues of the autocorrelation matrix R ..." x

In basic LMS, your step size μ affects convergence and stability.

To ensure convergence, we need:

2
0 < μ <
λ max

But how do we get λ max

? We'd need the eigenvalues of R , the input correlation
x

matrix**.

This is a problem because:

​ Computing R is expensive.
x

Inputs like speech signals are non-stationary — R changes over time.x

2. Idea of Normalization: Adapt step size to the input

"We want a way to choose μ without computing λ max or R ..."

x

So we approximate. Since:

2
Tr(R x ) = ∑ λ i ≈ E[∥x n ∥ ]

We just use:
2 T
∥x n ∥ = xn xn

as a rough real-time proxy for the total energy in the input vector.

Then we adapt μ like this:

β
μ(n) =
∥x n ∥ 2 + ε

β is a small constant (<2) to control convergence speed.

ε avoids division by zero.

This makes the step size smaller when input energy is high, and larger when input
energy is low. That’s smart.

3. Final NLMS update rule

You replace the fixed μ in LMS:

w n+1 = w n + μe(n)x n

with the new adaptive μ(n):

β
w n+1 = w n + e(n)x n
2
∥x n ∥ + ε

This is the Normalized LMS update.

4. Why is NLMS better in practice?

Handles non-stationary signals like speech

​ Speech has huge dynamic range (soft → loud, silence → bursts).
LMS must be tuned for the worst-case energy; NLMS adapts on-the-fly.
Avoids instability due to large x(n)
 ​ In LMS, large inputs can blow up updates.
NLMS scales them down: this prevents gradient noise amplification.
No need to estimate R x

​ This was the main reason we avoided Wiener filter to begin with!
NLMS respects that principle: learn adaptively without statistical estimation.

5. Trade-offs: More computation

You need to compute ∥x n

∥
2
at every step.
Requires division, which is expensive on embedded systems.
Workaround:
​ Use approximations, e.g., powers of two (bit-shifting).
Shift ∥x n∥
2
until it’s 1 → fast division.

Doubt: Why are we now talking about eigenvalues of Rx ​again in

LMS/NLMS?

Question: In the beginning of LMS, you said we are going for it because we don't
need to compute Rx as we do in Wiener filter, we just use the samples (noisy or
clean) to update the weights. But now while learning NLMS, you are saying step size
depends on Rx eigen values in LMS. Confusion!

Here’s the catch:

We avoid computing the autocorrelation matrix R , but... x

We still need to pick a step size μ that satisfies:

2
0 < μ <
λ max

So, we're not completely free from λ max — even if only to estimate a safe μ.
In practice, since λ max
is unknown, we pick a small μ, to "play it safe", which:
​ Ensures stability
But often leads to slow convergence

The idea of NLMS is to improve over LMS by making μ adapt to the signal's energy at
time n. So, instead of picking a fixed small μ, we do:

β
μ(n) =
2
∥x(n)∥ + ϵ
So the weight update becomes:

β
w(n + 1) = w(n) + e(n)x(n)
2
∥x(n)∥ + ϵ

This way:

Step size becomes smaller when input signal energy is high (e.g., during loud
speech)
And larger when input energy is low (e.g., during silence)
It removes the need to even approximate or think about λ max