BGP Techniques for Internet Service Providers

Philip Smith <pfs@[Link]> NANOG 44 12-14 October 2008 Los Angeles

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

Presentation Slides
Will be available on
[Link] /pfs/seminars/[Link] And on the NANOG44 website

Feel free to ask questions any time

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

BGP Techniques for Internet Service Providers

BGP Basics Scaling BGP Using Communities Deploying BGP in an ISP network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

BGP Basics

What is BGP?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

Border Gateway Protocol

A Routing Protocol used to exchange routing information between different networks
Exterior gateway protocol

Described in RFC4271
RFC4276 gives an implementation report on BGP RFC4277 describes operational experiences using BGP

The Autonomous System is the cornerstone of BGP

It is used to uniquely identify networks with a common routing policy

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

Autonomous System (AS)

AS 100

Collection of networks with same routing policy Single routing protocol Usually under single ownership, trust and administrative control Identified by a unique number (ASN)

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

Autonomous System Number (ASN)

An ASN is a 32 bit integer Two ranges
0-65535 65536-4294967295 (original 16-bit range) (32-bit range - RFC4893) (public Internet) (private use only) (represent 32-bit range in 16-bit world) (reserved) (public Internet)

Usage:
1-64511 64512-65534 23456 0 and 65535 65536-4294967295

32-bit range representation in IETF last call

[Link] Defines asplain (traditional format) as standard notation
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

Autonomous System Number (ASN)

ASNs are distributed by the Regional Internet Registries
They are also available from upstream ISPs who are members of one of the RIRs

Current 16-bit ASN allocations up to 49151 have been made to the RIRs
Around 29400 are visible on the Internet

The RIRs also have received 1024 32-bit ASNs each

12 are visible on the Internet (early adopters)

See [Link]/assignments/as-numbers
8

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

BGP Basics
Peering
A C

AS 100
B E D

AS 101

Runs over TCP port 179 Path vector protocol Incremental updates Internal & External BGP

AS 102

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

Demarcation Zone (DMZ)

A

AS 100
B

DMZ Network

AS 101
D

AS 102
Shared network between ASes
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

10

BGP General Operation

Learns multiple paths via internal and external BGP speakers Picks the best path and installs in the forwarding table Best path is sent to external BGP neighbours Policies are applied by influencing the best path selection

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

11

eBGP & iBGP

BGP used internally (iBGP) and externally (eBGP) iBGP used to carry
some/all Internet prefixes across ISP backbone ISPs customer prefixes

eBGP used to
exchange prefixes with other ASes implement routing policy

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

12

BGP/IGP model used in ISP networks

Model representation

eBGP

eBGP

eBGP

iBGP IGP

iBGP IGP

iBGP IGP

iBGP IGP

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

13

External BGP Peering (eBGP)

A

AS 100
B

AS 101

Between BGP speakers in different AS Should be directly connected Never run an IGP between eBGP peers
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

14

Internal BGP (iBGP)

BGP peer within the same AS Not required to be directly connected
IGP takes care of inter-BGP speaker connectivity

iBGP speakers must to be fully meshed:

They originate connected networks They pass on prefixes learned from outside the ASN They do not pass on prefixes learned from other iBGP speakers

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

15

Internal BGP Peering (iBGP)

AS 100
A C B

D
Topology independent Each iBGP speaker must peer with every other iBGP speaker in the AS
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

16

Peering to Loopback Interfaces

AS 100

Peer with loop-back interface

Loop-back interface does not go down ever!

Do not want iBGP session to depend on state of a single interface or the physical topology
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

17

BGP Attributes

Information about BGP

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

18

AS-Path
Sequence of ASes a route has traversed Used for:
Loop detection Applying policy

AS 200
[Link]/16

AS 100
[Link]/16

AS 300

[Link]/16 300 200 100 [Link]/16 300 200

AS 400
[Link]/16

AS 500

[Link]/16 [Link]/16 [Link]/16

300 200 100 300 200 300 400

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

19

AS-Path (with 16 and 32-bit ASNs)

Internet with 16-bit and 32-bit ASNs
32-bit ASNs are 65536 and above

AS 80000
[Link]/16

AS 70000
[Link]/16

AS-PATH length maintained

AS 300

[Link]/16 300 23456 23456 [Link]/16 300 23456

AS 400
[Link]/16

AS 90000

[Link]/16 [Link]/16 [Link]/16

300 80000 70000 300 80000 300 400

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

20

AS-Path loop detection

AS 200
[Link]/16

AS 100
[Link]/16 [Link]/16 [Link]/16 500 300 500 300 200

AS 300
[Link]/16

AS 500
[Link]/16 [Link]/16 [Link]/16 300 200 100 300 200 300

[Link]/16 is not accepted by AS100 as the prefix has AS100 in its ASPATH this is loop detection in action

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

21

Next Hop
[Link] [Link]

[Link]/16

AS 200

iBGP
A

eBGP

AS 300
[Link]/16 [Link] [Link]/16 [Link]

[Link]/16

AS 100

eBGP address of external neighbour iBGP NEXT_HOP from eBGP Mandatory non-transitive attribute

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

22

iBGP Next Hop

[Link]/23 [Link]/24

iBGP
Loopback [Link]/32

Loopback [Link]/32

AS 300
D A
[Link]/24 [Link] [Link]/23 [Link]

Next hop is ibgp router loopback address Recursive route look-up

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

23

Third Party Next Hop

AS 200
[Link]/24 [Link]

[Link]

eBGP between Router A and Router C

[Link]

[Link]

eBGP between RouterA and RouterB 120.68.1/24 prefix has next hop address of [Link] this is passed on to RouterC instead of [Link] More efficient No extra config needed

B
[Link]/24

AS 201

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

24

Next Hop Best Practice

BGP default is for external next-hop to be propagated unchanged to iBGP peers
This means that IGP has to carry external next-hops Forgetting means external network is invisible With many eBGP peers, it is unnecessary extra load on IGP

ISP Best Practice is to change external next-hop to be that of the local router

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

25

Next Hop (Summary)

IGP should carry route to next hops Recursive route look-up Unlinks BGP from actual physical topology Change external next hops to that of local router Allows IGP to make intelligent forwarding decision

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

26

Origin
Conveys the origin of the prefix Historical attribute
Used in transition from EGP to BGP

Transitive and Mandatory Attribute Influences best path selection Three values: IGP, EGP, incomplete
IGP generated by BGP network statement EGP generated by EGP incomplete redistributed from another routing protocol

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

27

Aggregator
Conveys the IP address of the router or BGP speaker generating the aggregate route Optional & transitive attribute Useful for debugging purposes Does not influence best path selection

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

28

Local Preference
AS 100
[Link]/16

AS 200
D
500 800

AS 300
E

A
[Link]/16 > [Link]/16 500 800

AS 400
C

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

29

Local Preference
Non-transitive and optional attribute Local to an AS non-transitive
Default local preference is 100 (IOS)

Used to influence BGP path selection

determines best path for outbound traffic

Path with highest local preference wins

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

30

Multi-Exit Discriminator (MED)

[Link]/24 > [Link]/24

2000 1000

AS 200
C D

[Link]/24

2000

[Link]/24

1000

B
[Link]/24

AS 201
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

31

Multi-Exit Discriminator
Inter-AS non-transitive & optional attribute Used to convey the relative preference of entry points
determines best path for inbound traffic

Comparable if paths are from same AS

Implementations have a knob to allow comparisons of MEDs from different ASes

Path with lowest MED wins Absence of MED attribute implies MED value of zero (RFC4271)

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

32

Multi-Exit Discriminator metric confusion

MED is non-transitive and optional attribute
Some implementations send learned MEDs to iBGP peers by default, others do not Some implementations send MEDs to eBGP peers by default, others do not

Default metric varies according to vendor implementation

Original BGP spec (RFC1771) made no recommendation Some implementations said that absence of metric was equivalent to 0 Other implementations said that absence of metric was equivalent to 232-1 (highest possible) or 232-2 Potential for metric confusion

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

33

Community
Communities are described in RFC1997
Transitive and Optional Attribute

32 bit integer
Represented as two 16 bit integers (RFC1998) Common format is <local-ASN>:xx 0:0 to 0:65535 and 65535:0 to 65535:65535 are reserved

Used to group destinations

Each destination could be member of multiple communities

Very useful in applying policies within and between ASes

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

34

Community Example (before)

ISP 2
X
[Link]/16 permit [Link]/16 in permit [Link]/16 out permit [Link]/16 out

E D

AS 400

ISP 1
C

AS 300
permit [Link]/16 in permit [Link]/16 in

AS 100

AS 200
[Link]/16

[Link]/16

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

35

Community Example (after)

ISP 2
X
[Link]/16 [Link]/16 300:9 [Link]/16 [Link]/16 300:1 300:1

E D

AS 400

ISP 1
C

AS 300
[Link]/16 300:1 [Link]/16 300:1

AS 100

AS 200
[Link]/16

[Link]/16

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

36

Well-Known Communities
Several well known communities
[Link]/assignments/bgp-well-known-communities

no-export no-advertise no-export-subconfed

65535:65281 65535:65282 65535:65283

do not advertise to any eBGP peers do not advertise to any BGP peer do not advertise outside local AS (only used with confederations)

no-peer

65535:65284

do not advertise to bi-lateral peers (RFC3765)

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

37

No-Export Community
[Link]/16 105.7.X.X No-Export 105.7.X.X

A B C E

D AS 200 F G

[Link]/16

AS 100

AS100 announces aggregate and subprefixes

Intention is to improve loadsharing by leaking subprefixes

Subprefixes marked with no-export community Router G in AS200 does not announce prefixes with no-export community set
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

38

No-Peer Community
[Link]/16 105.7.X.X No-Peer upstream

D
[Link]/16

C&D&E are peers e.g. Tier-1s

C A
upstream upstream

Sub-prefixes marked with no-peer community are not sent to bi-lateral peers
They are only sent to upstream providers
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

39

Community Implementation details

Community is an optional attribute
Some implementations send communities to iBGP peers by default, some do not Some implementations send communities to eBGP peers by default, some do not

Being careless can lead to community confusion

ISPs need consistent community policy within their own networks And they need to inform peers, upstreams and customers about their community expectations

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

40

BGP Path Selection Algorithm

Why Is This the Best Path?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

41

BGP Path Selection Algorithm for IOS Part One

Do not consider path if no route to next hop Do not consider iBGP path if not synchronised (Cisco IOS only) Highest weight (local to router) Highest local preference (global within AS) Prefer locally originated route Shortest AS path

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

42

BGP Path Selection Algorithm for IOS Part Two

Lowest origin code
IGP < EGP < incomplete

Lowest Multi-Exit Discriminator (MED)

If bgp deterministic-med, order the paths before comparing (BGP spec does not specify in which order the paths should be compared. This means best path depends on order in which the paths are compared.) If bgp always-compare-med, then compare for all paths otherwise MED only considered if paths are from the same AS (default)

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

43

BGP Path Selection Algorithm for IOS Part Three

Prefer eBGP path over iBGP path Path with lowest IGP metric to next-hop Lowest router-id (originator-id for reflected routes) Shortest Cluster-List
Client must be aware of Route Reflector attributes!

Lowest neighbour IP address

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

44

BGP Path Selection Algorithm

In multi-vendor environments:
Make sure the path selection processes are understood for each brand of equipment Each vendor has slightly different implementations, extra steps, extra features, etc Watch out for possible MED confusion

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

45

Applying Policy with BGP

Controlling Traffic Flow & Traffic Engineering

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

46

Applying Policy in BGP: Why?

Network operators rarely plug in routers and go External relationships:
Control who they peer with Control who they give transit to Control who they get transit from

Traffic flow control:

Efficiently use the scarce infrastructure resources (external link load balancing) Congestion avoidance Terminology: Traffic Engineering

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

47

Applying Policy in BGP: How?

Policies are applied by:
Setting BGP attributes (local-pref, MED, AS-PATH, community), thereby influencing the path selection process Advertising or Filtering prefixes Advertising or Filtering prefixes according to ASN and ASPATHs Advertising or Filtering prefixes according to Community membership

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

48

Applying Policy with BGP: Tools

Most implementations have tools to apply policies to BGP:
Prefix manipulation/filtering AS-PATH manipulation/filtering Community Attribute setting and matching

Implementations also have policy language which can do various match/set constructs on the attributes of chosen BGP routes

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

49

BGP Capabilities

Extending BGP

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

50

BGP Capabilities
Documented in RFC2842 Capabilities parameters passed in BGP open message Unknown or unsupported capabilities will result in NOTIFICATION message Codes:
0 to 63 are assigned by IANA by IETF consensus 64 to 127 are assigned by IANA first come first served 128 to 255 are vendor specific

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

51

BGP Capabilities
Current capabilities are:
0 1 2 3 4 64 65 66 67 68 Reserved Multiprotocol Extensions for BGP-4 Route Refresh Capability for BGP-4 Outbound Route Filtering Capability Multiple routes to a destination capability Graceful Restart Capability Support for 4 octet ASNs Deprecated 2003-03-06 Support for Dynamic Capability Multisession BGP [ID] [ID] [RFC3392] [RFC4760] [RFC2918] [RFC5291] [RFC3107] [RFC4724] [RFC4893]

See [Link]/assignments/capability-codes

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

52

BGP Capabilities
Multiprotocol extensions
This is a whole different world, allowing BGP to support more than IPv4 unicast routes Examples include: v4 multicast, IPv6, v6 multicast, VPNs Another tutorial (or many!)

Route refresh is a well known scaling technique covered shortly 32-bit ASNs have recently arrived The other capabilities are still in development or not widely implemented or deployed yet

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

53

BGP for Internet Service Providers

BGP Basics Scaling BGP Using Communities Deploying BGP in an ISP network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

54

BGP Scaling Techniques

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

55

BGP Scaling Techniques

How does a service provider:
Scale the iBGP mesh beyond a few peers? Implement new policy without causing flaps and route churning? Keep the network stable, scalable, as well as simple?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

56

BGP Scaling Techniques

Route Refresh Route Reflectors Confederations

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

57

Dynamic Reconfiguration

Route Refresh

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

58

Route Refresh
BGP peer reset required after every policy change
Because the router does not store prefixes which are rejected by policy

Hard BGP peer reset:

Terminates BGP peering & Consumes CPU Severely disrupts connectivity for all networks

Soft BGP peer reset (or Route Refresh):

BGP peering remains active Impacts only those prefixes affected by policy change

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

59

Route Refresh Capability

Facilitates non-disruptive policy changes For most implementations, no configuration is needed
Automatically negotiated at peer establishment

No additional memory is used Requires peering routers to support route refresh capability RFC2918

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

60

Dynamic Reconfiguration
Use Route Refresh capability if supported
find out from the BGP neighbour status display Non-disruptive, Good For the Internet

If not supported, see if implementation has a workaround Only hard-reset a BGP peering as a last resort

Consider the impact to be equivalent to a router reboot

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

61

Route Reflectors

Scaling the iBGP mesh

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

62

Scaling iBGP mesh

Avoid n(n-1) iBGP mesh 13 Routers 78 iBGP Sessions!

n=1000 nearly half a million ibgp sessions!

Two solutions
Route reflector simpler to deploy and run Confederation more complex, has corner case advantages
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

63

Route Reflector: Principle

Route Reflector

AS 100
B C

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

64

Route Reflector
Reflector receives path from clients and non-clients Selects best path If best path is from client, reflect to other clients and non-clients If best path is from non-client, reflect to clients only Non-meshed clients Described in RFC4456
B

Clients

Reflectors
C

AS 100

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

65

Route Reflector: Topology

Divide the backbone into multiple clusters At least one route reflector and few clients per cluster Route reflectors are fully meshed Clients in a cluster could be fully meshed Single IGP to carry next hop and local routes

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

66

Route Reflector: Loop Avoidance

Originator_ID attribute
Carries the RID of the originator of the route in the local AS (created by the RR)

Cluster_list attribute
The local cluster-id is added when the update is sent by the RR Best to set cluster-id is from router-id (address of loopback) (Some ISPs use their own cluster-id assignment strategy but needs to be well documented!)

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

67

Route Reflector: Redundancy

Multiple RRs can be configured in the same cluster not advised!
All RRs in the cluster must have the same cluster-id (otherwise it is a different cluster)

A router may be a client of RRs in different clusters

Common today in ISP networks to overlay two clusters redundancy achieved that way Each client has two RRs = redundancy

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

68

Route Reflector: Redundancy

PoP3

AS 100

PoP1 PoP2 Cluster One Cluster Two

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

69

Route Reflector: Benefits

Solves iBGP mesh problem Packet forwarding is not affected Normal BGP speakers co-exist Multiple reflectors for redundancy Easy migration Multiple levels of route reflectors

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

70

Route Reflector: Deployment

Where to place the route reflectors?
Always follow the physical topology! This will guarantee that the packet forwarding wont be affected

Typical ISP network:

PoP has two core routers Core routers are RR for the PoP Two overlaid clusters

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

71

Route Reflector: Migration

Typical ISP network:
Core routers have fully meshed iBGP Create further hierarchy if core mesh too big
Split backbone into regions

Configure one cluster pair at a time

Eliminate redundant iBGP sessions Place maximum one RR per cluster Easy migration, multiple levels

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

72

Route Reflector: Migration

AS 300
A B C D F G

AS 100 AS 200
E

Migrate small parts of the network, one part at a time

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

73

BGP Confederations

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

74

Confederations
Divide the AS into sub-AS
eBGP between sub-AS, but some iBGP information is kept Preserve NEXT_HOP across the sub-AS (IGP carries this information) Preserve LOCAL_PREF and MED

Usually a single IGP Described in RFC5065

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

75

Confederations (Cont.)
Visible to outside world as single AS Confederation Identifier
Each sub-AS uses a number from the private AS range (6451265534)

iBGP speakers in each sub-AS are fully meshed

The total number of neighbours is reduced by limiting the full mesh requirement to only the peers in the sub-AS Can also use Route-Reflector within sub-AS

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

76

Confederations
Sub-AS 65530
A

AS 200 Sub-AS 65531

B

Sub-AS 65532
C

Configuration (Router C):

router bgp 65532 bgp confederation identifier 200 bgp confederation peers 65530 65531 neighbor [Link] remote-as 65530 neighbor [Link] remote-as 65531
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

77

Confederations: AS-Sequence
[Link]/16 A 200

Sub-AS 65002
[Link]/16 {65004 65002} 200 B C [Link]/16 {65002} 200

Sub-AS 65004
H

Sub-AS 65003

E F

Sub-AS 65001

[Link]/16
NANOG 44

100

Confederation 100 200

78

2008 Cisco Systems, Inc. All rights reserved.

Route Propagation Decisions

Same as with normal BGP:
From peer in same sub-AS only to external peers From external peers to all neighbors

External peers refers to

Peers outside the confederation Peers in a different sub-AS
Preserve LOCAL_PREF, MED and NEXT_HOP

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

79

RRs or Confederations
Internet Connectivity Multi-Level Hierarchy Policy Control Scalability Migration Complexity

Anywhere Confederations in the Network

Yes

Yes

Medium

Medium to High

Route Reflectors

Anywhere in the Network

Yes

Yes

Very High

Very Low

Most new service provider networks now deploy Route Reflectors from Day One
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

80

More points about Confederations

Can ease absorbing other ISPs into you ISP e.g., if one ISP buys another
Or can use AS masquerading feature available in some implementations to do a similar thing

Can use route-reflectors with confederation sub-AS to reduce the sub-AS iBGP mesh

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

81

Route Flap Damping

Network Stability for the 1990s Network Instability for the 21st Century!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

82

Route Flap Damping

For many years, Route Flap Damping was a strongly recommended practice Now it is strongly discouraged as it appears to cause far greater network instability than it cures But first, the theory

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

83

Route Flap Damping

Route flap
Going up and down of path or change in attribute BGP WITHDRAW followed by UPDATE = 1 flap eBGP neighbour going down/up is NOT a flap Ripples through the entire Internet Wastes CPU

Damping aims to reduce scope of route flap propagation

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

84

Route Flap Damping (continued)

Requirements
Fast convergence for normal route changes History predicts future behaviour Suppress oscillating routes Advertise stable routes

Implementation described in RFC 2439

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

85

Operation
Add penalty (1000) for each flap
Change in attribute gets penalty of 500

Exponentially decay penalty

half life determines decay rate

Penalty above suppress-limit

do not advertise route to BGP peers

Penalty decayed below reuse-limit

re-advertise route to BGP peers penalty reset to zero when it is half of reuse-limit

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

86

Operation
4000 Penalty 3000 Suppress limit

Penalty
2000 Reuse limit 1000

0
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Time

Network Announced
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

Network Not Announced

Network Re-announced
87

Operation
Only applied to inbound announcements from eBGP peers Alternate paths still usable Controllable by at least:
Half-life reuse-limit suppress-limit maximum suppress time

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

88

Configuration
Implementations allow various policy control with flap damping
Fixed damping, same rate applied to all prefixes Variable damping, different rates applied to different ranges of prefixes and prefix lengths

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

89

Route Flap Damping History

First implementations on the Internet by 1995 Vendor defaults too severe
RIPE Routing Working Group recommendations in ripe-178, ripe-210, and ripe-229 [Link] But many ISPs simply switched on the vendors default values without thinking

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

90

Serious Problems:
"Route Flap Damping Exacerbates Internet Routing Convergence
Zhuoqing Morley Mao, Ramesh Govindan, George Varghese & Randy H. Katz, August 2002

What is the sound of one route flapping?

Tim Griffin, June 2002

Various work on routing convergence by Craig Labovitz and Abha Ahuja a few years ago Happy Packets
Closely related work by Randy Bush et al

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

91

Problem 1:
One path flaps:
BGP speakers pick next best path, announce to all peers, flap counter incremented Those peers see change in best path, flap counter incremented After a few hops, peers see multiple changes simply caused by a single flap prefix is suppressed

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

92

Problem 2:
Different BGP implementations have different transit time for prefixes
Some hold onto prefix for some time before advertising Others advertise immediately

Race to the finish line causes appearance of flapping, caused by a simple announcement or path change prefix is suppressed

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

93

Solution:
Do NOT use Route Flap Damping whatever you do! RFD will unnecessarily impair access
to your network and to the Internet

More information contained in RIPE Routing Working Group recommendations:

[Link]/ripe/docs/ripe-378.[pdf,html,txt]

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

94

BGP for Internet Service Providers

BGP Basics Scaling BGP Using Communities Deploying BGP in an ISP network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

95

Service Provider use of Communities

Some examples of how ISPs make life easier for themselves

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

96

BGP Communities
Another ISP scaling technique Prefixes are grouped into different classes or communities within the ISP network Each community means a different thing, has a different result in the ISP network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

97

ISP BGP Communities

There are no recommended ISP BGP communities apart from
RFC1998 The five standard communities [Link]/assignments/bgp-well-known-communities

Efforts have been made to document from time to time

[Link]/publications/papers-elec-versions/[Link] But so far nothing more Collection of ISP communities at [Link]/communities NANOG Tutorial: [Link]/meetings/nanog40/presentations/[Link]

ISP policy is usually published

On the ISPs website Referenced in the AS Object in the IRR
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

98

Some ISP Examples: Sprintlink

More info at [Link]/policy/[Link]

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

99

Some ISP Examples AAPT

Australian ISP Run their own Routing Registry
[Link]

Offer 6 different communities to customers to aid with their traffic engineering

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

100

Some ISP Examples AAPT

aut-num: as-name: descr: admin-c: tech-c: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: notify: mnt-by: changed: source: AS2764 ASN-CONNECT-NET AAPT Limited CNO2-AP CNO2-AP Community support definitions Community Definition -----------------------------------------------2764:2 Don't announce outside local POP 2764:4 Lower local preference by 15 2764:5 Lower local preference by 5 2764:6 Announce to customers and all peers (incl int'l peers), but not transit 2764:7 Announce to customers only 2764:14 Announce to AANX routing@[Link] CONNECT-AU nobody@[Link] 20050225 CCAIR

More at [Link]
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

101

Some ISP Examples Verizon Business EMEA

Verizon Business European operation Permits customers to send communities which determine
local preferences within Verizon Business network Reachability of the prefix How the prefix is announced outside of Verizon Business network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

102

Some ISP Examples Verizon Business Europe

aut-num: AS702 descr: Verizon Business EMEA - Commercial IP service provider in Eur remarks: VzBi uses the following communities with its customers: 702:80 Set Local Pref 80 within AS702 702:120 Set Local Pref 120 within AS702 702:20 Announce only to VzBi AS'es and VzBi customers 702:30 Keep within Europe, don't announce to other VzBi AS 702:1 Prepend AS702 once at edges of VzBi to Peers 702:2 Prepend AS702 twice at edges of VzBi to Peers 702:3 Prepend AS702 thrice at edges of VzBi to Peers Advanced communities for customers 702:7020 Do not announce to AS702 peers with a scope of National but advertise to Global Peers, European Peers and VzBi customers. 702:7001 Prepend AS702 once at edges of VzBi to AS702 peers with a scope of National. 702:7002 Prepend AS702 twice at edges of VzBi to AS702 peers with a scope of National. (more)

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

103

Some ISP Examples VzBi Europe

(more) 702:7003 Prepend AS702 thrice at edges of VzBi to AS702 peers with a scope of National. 702:8020 Do not announce to AS702 peers with a scope of European but advertise to Global Peers, National Peers and VzBi customers. 702:8001 Prepend AS702 once at edges of VzBi to AS702 peers with a scope of European. 702:8002 Prepend AS702 twice at edges of VzBi to AS702 peers with a scope of European. 702:8003 Prepend AS702 thrice at edges of VzBi to AS702 peers with a scope of European. -------------------------------------------------------------Additional details of the VzBi communities are located at: [Link] -------------------------------------------------------------WCOM-EMEA-RICE-MNT RIPE

mnt-by: source:

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

104

Some ISP Examples BT Ignite

One of the most comprehensive community lists around
Seems to be based on definitions originally used in Tiscalis network whois h [Link] AS5400 reveals all

Extensive community definitions allow sophisticated traffic engineering by customers

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

105

Some ISP Examples BT Ignite

aut-num: descr: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: <snip> notify: mnt-by: source:
NANOG 44

AS5400 BT Ignite European Backbone Community to Not announce To peer: Community to AS prepend 5400 5400:2000 5400:2500 5400:2501 5400:2502 5400:2503 5400:2504 5400:2506 5400:2001 5400:2002 5400:2004

5400:1000 All peers & Transits 5400:1500 5400:1501 5400:1502 5400:1503 5400:1504 5400:1506 All Transits Sprint Transit (AS1239) SAVVIS Transit (AS3561) Level 3 Transit (AS3356) AT&T Transit (AS7018) GlobalCrossing Trans(AS3549)

5400:1001 Nexica (AS24592) 5400:1002 Fujitsu (AS3324) 5400:1004 C&W EU (1273) notify@[Link] CIP-MNT RIPE

And many many more!

106

2008 Cisco Systems, Inc. All rights reserved.

Some ISP Examples Level 3

Highly detailed AS object held on the RIPE Routing Registry Also a very comprehensive list of community definitions
whois h [Link] AS3356 reveals all

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

107

Some ISP Examples Level 3

aut-num: descr: <snip> remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: remarks: <snip> remarks: remarks: remarks: remarks: <snip> mnt-by: source:
NANOG 44

AS3356 Level 3 Communications ------------------------------------------------------customer traffic engineering communities - Suppression ------------------------------------------------------64960:XXX - announce to AS XXX if 65000:0 65000:0 - announce to customers but not to peers 65000:XXX - do not announce at peerings to AS XXX ------------------------------------------------------customer traffic engineering communities - Prepending ------------------------------------------------------65001:0 - prepend once to all peers 65001:XXX - prepend once at peerings to AS XXX 3356:70 3356:80 3356:90 3356:9999 set local set local set local blackhole preference to 70 preference to 80 preference to 90 (discard) traffic

LEVEL3-MNT RIPE

And many many more!

108

2008 Cisco Systems, Inc. All rights reserved.

BGP for Internet Service Providers

BGP Basics Scaling BGP Using Communities Deploying BGP in an ISP network

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

109

Deploying BGP in an ISP Network

Okay, so weve learned all about BGP now; how do we use it on our network??

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

110

Deploying BGP
The role of IGPs and iBGP Aggregation Receiving Prefixes Configuration Tips

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

111

The role of IGP and iBGP

Ships in the night? Or Good foundations?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

112

BGP versus OSPF/ISIS

Internal Routing Protocols (IGPs)
examples are ISIS and OSPF used for carrying infrastructure addresses NOT used for carrying Internet prefixes or customer prefixes design goal is to minimise number of prefixes in IGP to aid scalability and rapid convergence

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

113

BGP versus OSPF/ISIS

BGP used internally (iBGP) and externally (eBGP) iBGP used to carry
some/all Internet prefixes across backbone customer prefixes

eBGP used to
exchange prefixes with other ASes implement routing policy

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

114

BGP/IGP model used in ISP networks

Model representation

eBGP

eBGP

eBGP

iBGP IGP

iBGP IGP

iBGP IGP

iBGP IGP

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

115

BGP versus OSPF/ISIS

DO NOT:
distribute BGP prefixes into an IGP distribute IGP routes into BGP use an IGP to carry customer prefixes

YOUR NETWORK WILL NOT SCALE

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

116

Injecting prefixes into iBGP

Use iBGP to carry customer prefixes
Dont ever use IGP

Point static route to customer interface Enter network into BGP process
Ensure that implementation options are used so that the prefix always remains in iBGP, regardless of state of interface i.e. avoid iBGP flaps caused by interface flaps

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

117

Aggregation

Quality or Quantity?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

118

Aggregation
Aggregation means announcing the address block received from the RIR to the other ASes connected to your network Subprefixes of this aggregate may be:
Used internally in the ISP network Announced to other ASes to aid with multihoming

Unfortunately too many people are still thinking about class Cs, resulting in a proliferation of /24s in the Internet routing table

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

119

Aggregation
Address block should be announced to the Internet as an aggregate Subprefixes of address block should NOT be announced to Internet unless special circumstances (more later) Aggregate should be generated internally
Not on the network borders!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

120

Announcing an Aggregate
ISPs who dont and wont aggregate are held in poor regard by community Registries publish their minimum allocation size
Anything from a /20 to a /22 depending on RIR Different sizes for different address blocks

No real reason to see anything longer than a /22 prefix in the Internet
BUT there are currently >141000 /24s!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

121

Aggregation Example
[Link]/23 [Link]/24 [Link]/22

customer

Internet

AS100

[Link]/23

Customer has /23 network assigned from AS100s /19 address block AS100 announces customers individual networks to the Internet

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

122

Aggregation Bad Example

Customer link goes down
Their /23 network becomes unreachable /23 is withdrawn from AS100s iBGP

Customer link returns

Their /23 network is now visible to their ISP Their /23 network is readvertised to peers Starts rippling through Internet Load on Internet backbone routers as network is reinserted into routing table Some ISPs suppress the flaps Internet may take 10-20 min or longer to be visible Where is the Quality of Service???
123

Their ISP doesnt aggregate its /19 network block

/23 network withdrawal announced to peers starts rippling through the Internet added load on all Internet backbone routers as network is removed from routing table
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

Aggregation Example
[Link]/19

[Link]/19 aggregate

customer

Internet

AS100

[Link]/23

Customer has /23 network assigned from AS100s /19 address block AS100 announced /19 aggregate to the Internet

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

124

Aggregation Good Example

Customer link goes down
their /23 network becomes unreachable /23 is withdrawn from AS100s iBGP

Customer link returns Their /23 network is visible again

The /23 is re-injected into AS100s iBGP

/19 aggregate is still being announced

no BGP hold down problems no BGP propagation delays no damping by other ISPs

The whole Internet becomes visible immediately Customer has Quality of Service perception

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

125

Aggregation Summary
Good example is what everyone should do!
Adds to Internet stability Reduces size of routing table Reduces routing churn Improves Internet QoS for everyone

Bad example is what too many still do!

Why? Lack of knowledge? Laziness?

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

126

The Internet Today (October 2008)

Current Internet Routing Table Statistics
BGP Routing Table Entries Prefixes after maximum aggregation Unique prefixes in Internet Prefixes smaller than registry alloc /24s announced only 5753 /24s are from [Link]/8 ASes in use 29392 270153 130372 131760 132678 141064

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

127

The New Swamp

Swamp space is name used for areas of poor aggregation
The original swamp was [Link]/8 from the former class C block
Name given just after the deployment of CIDR

The new swamp is creeping across all parts of the Internet

Not just RIR space, but legacy space too

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

128

The New Swamp RIR Space February 1999

RIR blocks contribute 49393 prefixes or 88% of the Internet Routing Table
Block
24/8 41/8 58/8 59/8 60/8 61/8 62/8 63/8 64/8 65/8 66/8 67/8 68/8 69/8 70/8 71/8 72/8 73/8 74/8 75/8 76/8 77/8 78/8
NANOG 44

Networks
165 0 0 0 0 3 87 20 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Block
79/8 80/8 81/8 82/8 83/8 84/8 85/8 86/8 87/8 88/8 89/8 90/8 91/8 96/8 97/8 98/8 99/8 112/8 113/8 114/8 115/8 116/8 117/8

Networks
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Block
118/8 119/8 120/8 121/8 122/8 123/8 124/8 125/8 126/8 173/8 174/8 186/8 187/8 189/8 190/8 192/8 193/8 194/8 195/8 196/8 198/8 199/8 200/8

Networks
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6275 2390 2932 1338 513 4034 3495 1348

Block
201/8 202/8 203/8 204/8 205/8 206/8 207/8 208/8 209/8 210/8 211/8 212/8 213/8 216/8 217/8 218/8 219/8 220/8 221/8 222/8

Networks
0 2276 3622 3792 2584 3127 2723 2817 2574 617 0 717 1 943 0 0 0 0 0 0

2008 Cisco Systems, Inc. All rights reserved.

129

The New Swamp RIR Space February 2008

RIR blocks contribute 219688 prefixes or 89% of the Internet Routing Table
Block
24/8 41/8 58/8 59/8 60/8 61/8 62/8 63/8 64/8 65/8 66/8 67/8 68/8 69/8 70/8 71/8 72/8 73/8 74/8 75/8 76/8 77/8 78/8
NANOG 44

Networks
3103 1087 1479 1317 853 2653 2303 3069 5953 4012 7172 2652 2858 4203 1798 1186 3543 254 3002 1086 1029 1515 1169

Block
79/8 80/8 81/8 82/8 83/8 84/8 85/8 86/8 87/8 88/8 89/8 90/8 91/8 96/8 97/8 98/8 99/8 112/8 113/8 114/8 115/8 116/8 117/8

Networks
588 2162 1724 1641 1215 1290 2316 768 1484 900 2824 220 2227 255 162 389 282 0 0 4 4 1011 960

Block
118/8 119/8 120/8 121/8 122/8 123/8 124/8 125/8 126/8 173/8 174/8 186/8 187/8 189/8 190/8 192/8 193/8 194/8 195/8 196/8 198/8 199/8 200/8

Networks
649 469 0 1054 1600 1225 1787 2217 46 0 0 2 6 1475 3203 6929 6220 4926 4480 1769 4799 4116 8626

Block
201/8 202/8 203/8 204/8 205/8 206/8 207/8 208/8 209/8 210/8 211/8 212/8 213/8 216/8 217/8 218/8 219/8 220/8 221/8 222/8

Networks
3632 10934 11000 5601 3008 3863 4285 5444 5590 4931 2875 3015 3310 7129 2666 1375 1320 2153 969 1268

2008 Cisco Systems, Inc. All rights reserved.

130

The New Swamp Summary

RIR space shows creeping deaggregation
It seems that an RIR /8 block averages around 5000 prefixes once fully allocated So their existing 88 /8s will eventually cause 440000 prefix announcements

Food for thought:

Remaining 39 unallocated /8s and the 88 RIR /8s combined will cause: 635000 prefixes with 5000 prefixes per /8 density 762000 prefixes with 6000 prefixes per /8 density Plus 12% due to non RIR space deaggregation Routing Table size of 853440 prefixes

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

131

The New Swamp Summary

Rest of address space is showing similar deaggregation too What are the reasons?
Main justification is traffic engineering

Real reasons are:

Lack of knowledge Laziness Deliberate & knowing actions

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

132

BGP Report ([Link])

199336 total announcements in October 2006 129795 prefixes
After aggregating including full AS PATH info
i.e. including each ASNs traffic engineering

35% saving possible

109034 prefixes
After aggregating by Origin AS
i.e. ignoring each ASNs traffic engineering

10% saving possible

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

133

Deaggregation: The Excuses

Traffic engineering causes 10% of the Internet Routing table Deliberate deaggregation causes 35% of the Internet Routing table

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

134

Efforts to improve aggregation

The CIDR Report
Initiated and operated for many years by Tony Bates Now combined with Geoff Hustons routing analysis [Link] Results e-mailed on a weekly basis to most operations lists around the world Lists the top 30 service providers who could do better at aggregating

RIPE Routing WG aggregation recommendation

RIPE-399 [Link]

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

135

Efforts to Improve Aggregation The CIDR Report

Also computes the size of the routing table assuming ISPs performed optimal aggregation Website allows searches and computations of aggregation to be made on a per AS basis
Flexible and powerful tool to aid ISPs Intended to show how greater efficiency in terms of BGP table size can be obtained without loss of routing and policy information Shows what forms of origin AS aggregation could be performed and the potential benefit of such actions to the total table size Very effectively challenges the traffic engineering excuse

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

136

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

137

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

138

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

139

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

140

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

141

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

142

Importance of Aggregation
Size of routing table
Memory is no longer a problem Routers can be specified to carry 1 million prefixes

Convergence of the Routing System

This is a problem Bigger table takes longer for CPU to process BGP updates take longer to deal with BGP Instability Report tracks routing system update activity [Link]

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

143

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

144

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

145

Aggregation Potential (source: [Link]/as2.0/)

AS Path AS Origin

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

146

Aggregation Summary
Aggregation on the Internet could be MUCH better
35% saving on Internet routing table size is quite feasible Tools are available
Commands on the routers are not hard CIDR-Report webpage

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

147

Receiving Prefixes

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

148

Receiving Prefixes
There are three scenarios for receiving prefixes from other ASNs
Customer talking BGP Peer talking BGP Upstream/Transit talking BGP

Each has different filtering requirements and need to be considered separately

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

149

Receiving Prefixes: From Customers

ISPs should only accept prefixes which have been assigned or allocated to their downstream customer If ISP has assigned address space to its customer, then the customer IS entitled to announce it back to his ISP If the ISP has NOT assigned address space to its customer, then:
Check the five RIR databases to see if this address space really has been assigned to the customer The tool: whois

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

150

Receiving Prefixes: From Customers

Example use of whois to check if customer is entitled to announce address space:
pfs-pc$ whois inetnum: netname: descr: descr: descr: descr: country: admin-c: tech-c: mnt-by: changed: status: source: -h [Link] [Link] [Link] - [Link] APNIC-AP-AU-BNE APNIC Pty Ltd - Brisbane Offices + Servers Level 1, 33 Park Rd PO Box 2131, Milton Brisbane, QLD. AU HM20-AP Portable means its an assignment NO4-AP to the customer, the customer can APNIC-HM announce it to you hm-changed@[Link] 20030108 ASSIGNED PORTABLE APNIC

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

151

Receiving Prefixes: From Customers

Example use of whois to check if customer is entitled to announce address space:
$ whois -h [Link] [Link] inetnum: [Link] - [Link] descr: Wood Mackenzie country: GB admin-c: DB635-RIPE ASSIGNED PA means that it is tech-c: DB635-RIPE Provider Aggregatable address space status: ASSIGNED PA and can only be used for connecting mnt-by: AS1849-MNT changed: davids@[Link] 20020211 to the ISP who assigned it source: RIPE route: descr: origin: notify: mnt-by: changed: source: [Link]/14 PIPEX-BLOCK1 AS1849 routing@[Link] AS1849-MNT beny@[Link] 20020321 RIPE

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

152

Receiving Prefixes: From Peers

A peer is an ISP with whom you agree to exchange prefixes you originate into the Internet routing table
Prefixes you accept from a peer are only those they have indicated they will announce Prefixes you announce to your peer are only those you have indicated you will announce

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

153

Receiving Prefixes: From Peers

Agreeing what each will announce to the other:
Exchange of e-mail documentation as part of the peering agreement, and then ongoing updates OR Use of the Internet Routing Registry and configuration tools such as the IRRToolSet [Link]/sw/IRRToolSet/

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

154

Receiving Prefixes: From Upstream/Transit Provider

Upstream/Transit Provider is an ISP who you pay to give you transit to the WHOLE Internet Receiving prefixes from them is not desirable unless really necessary
special circumstances see later

Ask upstream/transit provider to either:

originate a default-route OR announce one prefix you can use as default

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

155

Receiving Prefixes: From Upstream/Transit Provider

If necessary to receive prefixes from any provider, care is required
dont accept RFC1918 etc prefixes [Link] dont accept your own prefixes dont accept default (unless you need it) dont accept prefixes longer than /24

Check Team Cymrus bogon pages

[Link] [Link] bogon route server

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

156

Receiving Prefixes
Paying attention to prefixes received from customers, peers and transit providers assists with:
The integrity of the local network The integrity of the Internet

Responsibility of all ISPs to be good Internet citizens

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

157

Configuration Tips

Of passwords, tricks and templates

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

158

iBGP and IGPs Reminder!

Make sure loopback is configured on router
iBGP between loopbacks, NOT real interfaces

Make sure IGP carries loopback /32 address Consider the DMZ nets:
Use unnumbered interfaces? Use next-hop-self on iBGP neighbours Or carry the DMZ /30s in the iBGP Basically keep the DMZ nets out of the IGP!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

159

iBGP: Next-hop-self
BGP speaker announces external network to iBGP peers using routers local address (loopback) as nexthop Used by many ISPs on edge routers
Preferable to carrying DMZ /30 addresses in the IGP Reduces size of IGP to just core infrastructure Alternative to using unnumbered interfaces Helps scale network Many ISPs consider this best practice

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

160

Limiting AS Path Length

Some BGP implementations have problems with long AS_PATHS
Memory corruption Memory fragmentation

Even using AS_PATH prepends, it is not normal to see more than 20 ASes in a typical AS_PATH in the Internet today
The Internet is around 5 ASes deep on average Largest AS_PATH is usually 16-20 ASNs

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

161

Limiting AS Path Length

Some announcements have ridiculous lengths of ASpaths:
*> [Link]/24 22 11537 145 12199 10318 10566 13193 1930 2200 3425 293 5609 5430 13285 6939 14277 1849 33 15589 25336 6830 8002 2042 7610 i

This example is an error in one IPv6 implementation

*> [Link]/22 2497 3257 29686 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 16327 i

This example shows 20 prepends (for no obvious reason)

If your implementation supports it, consider limiting the maximum AS-path length you will accept
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

162

BGP TTL hack

Implement RFC5082 on BGP peerings
(Generalised TTL Security Mechanism) Neighbour sets TTL to 255 Local router expects TTL of incoming BGP packets to be 254 No one apart from directly attached devices can send BGP packets which arrive with TTL of 254, so any possible attack by a remote miscreant is dropped due to TTL mismatch

ISP
R1

TTL 254

AS 100
R2 TTL 254

Attacker

TTL 253
NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

163

BGP TTL hack

TTL Hack:
Both neighbours must agree to use the feature TTL check is much easier to perform than MD5 (Called BTSH BGP TTL Security Hack)

Provides security for BGP sessions

In addition to packet filters of course MD5 should still be used for messages which slip through the TTL hack See [Link]/mtg-0302/[Link] for more details

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

164

Templates
Good practice to configure templates for everything
Vendor defaults tend not to be optimal or even very useful for ISPs ISPs create their own defaults by using configuration templates

eBGP and iBGP examples follow

Also see Team Cymrus BGP templates [Link]

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

165

iBGP Template Example

iBGP between loopbacks! Next-hop-self
Keep DMZ and external point-to-point out of IGP

Always send communities in iBGP

Otherwise accidents will happen

Hardwire BGP to version 4

Yes, this is being paranoid!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

166

iBGP Template Example continued

Use passwords on iBGP session
Not being paranoid, VERY necessary Its a secret shared between you and your peer If arriving packets dont have the correct MD5 hash, they are ignored Helps defeat miscreants who wish to attack BGP sessions

Powerful preventative tool, especially when combined with filters and the TTL hack

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

167

eBGP Template Example

BGP damping
Do NOT use it unless you understand the impact Do NOT use the vendor defaults without thinking

Remove private ASes from announcements

Common omission today

Use extensive filters, with backup

Use as-path filters to backup prefix filters Keep policy language for implementing policy, rather than basic filtering

Use password agreed between you and peer on eBGP session

NANOG 44 2008 Cisco Systems, Inc. All rights reserved.

168

eBGP Template Example continued

Use maximum-prefix tracking
Router will warn you if there are sudden increases in BGP table size, bringing down eBGP if desired

Limit maximum as-path length inbound Log changes of neighbour state

and monitor those logs!

Make BGP admin distance higher than that of any IGP

Otherwise prefixes heard from outside your network could override your IGP!!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

169

Summary
Use configuration templates Standardise the configuration Be aware of standard tricks to avoid compromise of the BGP session Anything to make your life easier, network less prone to errors, network more likely to scale Its all about scaling if your network wont scale, then it wont be successful

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

170

BGP Techniques for Internet Service Providers

End of Tutorial!

NANOG 44

2008 Cisco Systems, Inc. All rights reserved.

171