BGP MED Churn
Daniel Walton
dwalton@[Link]
QOS
dwalton@[Link] ©2000, Cisco Systems, Inc. 1
Description
• MED in a RR or Confederation environment can
cause an endless convergence loop
• Happens as a result of two things:
RRs and Confeds “hide” path information
MEDs are only compared among like Neighbor ASs
• Two types of “The Churn”
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 2
The Churn – Type I
• Network must have multiple paths to a prefix via multiple Neighbor ASs
• The MED values for these paths must be unique
• Network must have a single tier of RRs or Sub ASs to have Type I
churn
• Type I can be fixed today
Network must use “deterministic-med”
Network must follow the deployment guidelines of the RR and Confed drafts
Drafts state that “intra cluster/SubAS paths must be preferred over inter
cluster/SubAS paths”
Result is that “intra” IGP metrics must ALWAYS be lower than “inter” IGP metrics
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 3
The Churn – Type I
• Still not a great solution
IGP change could trigger The Churn
Networks are bound to a single tier
Hands are tied in terms of setting IGP metrics
• For more details please see:
“Endless BGP Convergence Problem” -
[Link]/warp/public/770/[Link]
Includes information on how to identify MED Churn
Includes an example of Type I churn
Includes information on the solution for Type I
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 4
The Churn – Type II
• Network must have multiple paths to a prefix via multiple
Neighbor ASs
• The MED values for these paths must be unique
• Network must have more than one tier of RRs or SubASs
• Solution for Type I does not apply
• Type II cannot be fixed today with the current decision
algorithm
• Example …
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 5
The Churn – Type II
SubAS 65000 Step 1
= Advertisement
– E selects Y1
C D = Withdrawal
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C
B E
10 3 2 D
A F G
E X 3
AS Y AS X AS Y * Y 1 2
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 6
The Churn – Type II
SubAS 65000 Step 2
= Advertisement
– C selects Y0
C D = Withdrawal
– D selects Y1
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C * Y 0 50
B E
10 3 2 D
* Y 1 42
A F G
E X 3
AS Y AS X AS Y * Y 1 2
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 7
The Churn – Type II
SubAS 65000 Step 3
= Advertisement
– D selects Y0
C D = Withdrawal
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C * Y 0 50
B E
Y 1 44
10 3 2 D * Y 0 52
Y 1 42
A F G
E X 3
AS Y AS X AS Y * Y 1 2
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 8
The Churn – Type II
SubAS 65000 Step 4
= Advertisement
– E selects X
C D = Withdrawal
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C * Y 0 50
B E
10 3 2 D * Y 0 52
Y 1 42
A F G
E * X 3
Y 0 92
AS Y AS X AS Y
Y 1 2
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 9
The Churn – Type II
SubAS 65000 Step 5
= Advertisement
– D selects X
C D = Withdrawal
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C * Y 0 50
B E
10 3 2 D Y 0 52
* X 43
A F G
E * X 3
Y 0 92
AS Y AS X AS Y
Y 1 2
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 10
The Churn – Type II
SubAS 65000 Step 6
= Advertisement
– C selects X
C D = Withdrawal
– E selects Y1
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C Y 0 50
B E * X 45
10 3 2 D Y 0 52
* X 43
A F G
E X 3
* Y 1 2
AS Y AS X AS Y
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 11
The Churn – Type II
SubAS 65000 Step 7
= Advertisement
– D selects Y1
C D = Withdrawal
2
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C Y 0 50
B E * X 45
10 3 2 D
* Y 1 42
A F G
E X 3
* Y 1 2
AS Y AS X AS Y
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 12
The Churn – Type II
SubAS 65000 Step 8
= Advertisement
– C selects Y0
C D = Withdrawal
– This is the same as Step 2
2 – BGP is in a loop
40 40 AS_PATH MED IGP
SubAS 65001 SubAS 65002
C * Y 0 50
B E
Y 1 44
10 3 2 D
* Y 1 42
A F G
E X 3
* Y 1 2
AS Y AS X AS Y
QOS
MED 0 MED 1
dwalton@[Link] © 2000, Cisco Systems, Inc. 13
The Churn – Type II
SubAS 65000 • In a nutshell, the churn happens
because E does not always know about
C D the Y0 path but the Y0 path has an affect
2 on what E considers to be his best path.
• Without Y0, E considers Y1 as best
40 40
• With Y0, E considers X as best
SubAS 65001 SubAS 65002 • From C and D’s point of view
Y0 < Y1 < X < Y0 This happens
B E because MED is not compared every
time
10 3 2 • Sequence
C selects Y0 and Y0 is propagated to D, E
E receives Y0 which forces E to select X
D receives X and selects it over Y0
A F G
C receives X and selects it over Y0
C sends a withdrawal for Y0
AS Y AS X AS Y E receives the withdrawal for Y0 so E now
MED 0 MED 1 prefers Y1
QOS C, D receive Y1 but select Y0 14
dwalton@[Link] © 2000, Cisco Systems, Inc.
Possible Solutions
• Solution #1 – Make sure E has the Y0 path
BGP Peers will need to advertise multiple paths
BGP will need a new Attribute that will allow a speaker to
advertise multiple paths for the same prefix (draft coming soon)
A BGP speaker will then need to advertise a best path per
“Neighbor AS” group IF that path came from an internal peer. This
will force C and D to always advertise Y0 to D
• Solution #2 – Eliminate “Y0 < Y1 < X < Y0” problem
Always comparing MEDs accomplishes this
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 15
Spotting “The Churn”
• Two steps to ID the churn in your network
• 1 – Run “show ip route bgp | include , 00:00” once every 60 seconds for ~5 minutes. This
will give you a list of routes that have changed within the past minute. If a route is
changing every minute then there is a good chance it is churning.
Router#show ip route bgp | include , 00:00
B [Link]/22 [200/1] via [Link], 00:00:32
B [Link]/23 [200/1] via [Link], 00:00:58
Router#
Wait 60 seconds…
Router#show ip route bgp | include , 00:00
B [Link]/24 [200/1] via [Link], 00:00:17
B [Link]/23 [200/1] via [Link], 00:00:57
Router#
[Link]/23 has changed twice in the last 2 minutes. It is possible that this prefix is churning.
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 16
Spotting “The Churn”
• 2 – Take a prefix from #1 and do “show ip bgp x.x.x.x | include best #“
for a little over 1 minute. If you see a pattern in the best path
transition then this prefix is churning. If not, select another
prefix from #1 and try again.
Router#show ip bgp [Link] | include best # Next, the best path changes to #18.
Paths: (23 available, best #17)
Router#show ip bgp [Link] | include best # Router#show ip bgp [Link] | include best #
Paths: (23 available, best #17) Paths: (24 available, best #18)
Router#show ip bgp [Link] | include best #
Paths: (23 available, best #17) Now, the best path is #17 again.
Router#show ip bgp [Link] | include best #
Paths: (23 available, best #17) Router#show ip bgp [Link] | include best #
Paths: (23 available, best #17)
Then, the best path changes to #14. Router#show ip bgp [Link] | include best #
Paths: (23 available, best #17)
Router#show ip bgp [Link] | include best #
Paths: (23 available, best #14) Notice the transition “17->17->14->18->17->17”!!
Repeat Step #2 for another minute just to be sure
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 17
Summary
• Single Tier Networks
The churn can be eliminated by using deterministic-
med and tweaking your IGP metrics. Another option is
to always compare MED.
• Multi Tier Networks
Currently the only “solution” is to always compare
MED. A more feasible solution is in the works but it will
require BGP to propagate more than one path for a
prefix.
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 18
BGP MED Churn
• Questions
• Comments
QOS
dwalton@[Link] © 2000, Cisco Systems, Inc. 19