Packet Journey Inside

ASR 9000
Mike Mikhail, Technical Leader
BRKARC-2017

mamikhai@[Link]

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

Introduction

About me

Mike Mikhail, Technical Leader, email: mamikhai@[Link]

Available at Meet the Engineer for scheduling 1:1 discussions
Interests: SP edge routers [7600, ASR 9000], SP technologies, QoS

ASR 9000 today

A very popular SP and WAN edge router.

High bandwidth Ethernet services
A hardware forwarding platform: 100s of Mpps per LC
Distributed processing and distributed forwarding
Continued development and busy product roadmap: New hardware for higher density
and bandwidth, and a wealth of new features

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

ASR 9000 Models

Chassis, cards, power, air flow
> 1.5
Tbps/slot
[future]

Fixed hw:

Not a router!
A 10G to 1G
fan out.

RP+SP+LC+p
orts+bays

ASR 9000v

ASR 9001

ASR 9904

ASR 9006

ASR 9010

ASR 9912

ASR 9922

None

Built-in

1+1 RSP

1+1 RSP

1+1 RSP

1+1 RP

1+1 RP

None

Built-in

2x RSP

2x RSP

2x RSP

6+1

6+1

4x SFP+
44x SFP

4x SFP+
2x MPA

10

20

10

21

30

44

Power modules

1x AC or 2x DC

2x AC or 2x DC

4x AC or 4x DC

4x AC or 4x DC

Air flow

Right to left

Right to left

Right to left

Right to back

RP
Fabric
Line cards &
ports
Rack units

8x AC or 8x DC 12x AC or 12x DC 16x AC or 16x DC

Front to back

Front to back

Front to back

System Architecture
9904, 9006, 9010
Distributed control
plane
L2 protocols, ARP,
BFD, CFM, Netflow
run on LC CPU

Distributed data
plane

Line Card
RSP
C

CPU
P BITS/DTI
U FIA
FIC

Forwarding
distributed to NPs

Active-active switch
fabric
Each RSP houses
half of the fabric

Switch Fabric

The Switch Fabric

9904, 9006, 9010
3-Stage fabric
Allows for variable number of
FIAs and FIA links on LC

Super-framing for
unicast
Super-frame same-priority
same-egress frames in a
jumbo frame

[880/440] Gbps per slot

4x [110/55 Gbps] links per
slot per RSP
[440/220 Gbps] if an RSP is
removed

Fabric load sharing

Unicast: per super-frame
Multicast: per flow

110G
110Glinks
links[RSP880]
[RSP880]
55G
55Glinks
links[RSP440]
[RSP440]

The Switch Fabric

9912, 9922

Super-framing for
unicast

2x [110/55 Gbps] links per

slot per fabric card

Linecard
Fabric Stage

Bandwidth per slot:

Linecard
Fabric Stage

Super-frame same-priority
same-egress frames in a
jumbo frame

Linecard
Fabric Stage

Allows for variable number of

FIAs and FIA links on LC

Linecard
Fabric Stage

3-Stage fabric

Fabric load sharing

Unicast: per super-frame
Multicast: per flow

Fabric cards

110G links [RSP880]

55G links [RSP440]

Slot Address
Internal slot address, slot mask, and fabric group ID
Slot

Slot Mask

Slot

Logical

Physical

Binary

Hex

LC7

1000000000

0x0200

LC6

0100000000

0x0100

LC5

0010000000

0x0080

LC4

0001000000

0x0040

RSP0

0000100000

0x0020

RSP1

0000010000

0x0010

LC3

0000001000

0x0008

LC2

0000000100

0x0004

LC1

0000000010

0x0002

LC0

0000000001

0x0001

9006

9010

Slot Mask

Logical

Physical

Binary

Hex

LC3

0000100000

0x0020

LC2

0000010000

0x0010

LC1

0000001000

0x0008

LC0

0000000100

0x0004

RSP1

0000000010

0x0002

RSP0

0000000001

0x0001

Slot

9912/9922

Follows the sequence of slots in chassis

4/6/12/22 RP slots: 000011 which is 0x0003, decimal 3
9010 RSP slots: 0000110000 which is 0x0030, decimal 48

Slot Mask

Logical

Physical

Binary

Hex

LC19

21

10000 00000000
00000000

0x10
0000

LC1-18

3-20

LC0

0000000100

0x0004

RP1

0000000010

0x0002

RP0

0000000001

0x0001

Fabric
Link status
RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric
arbiter linkstatus 3 location 0/RSP0/CPU0
Wed Nov 20 20:44:29.615 EST
Slot 3
PG:9
up
RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric
arbiter linkstatus 0 location 0/RSP0/CPU0
Wed Nov 20 20:44:22.995 EST
Slot 0
PG:24
up

Internal slot
number

RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric

crossbar link-status instance 1 location 0/RSP0/CPU0
Tue Dec 17 02:59:30.110 EST
PORT
Remote Slot Remote Inst
Logical ID Status
======================================================
02
03
00
0
Up
16
00
00
0
Up
18
02
00
1
Up
20
02
00
0
Up
24
03
00
1
Up

Internal slot
number

LC3 in 9010
LC0 in 9010

Fabric
Link statistics
Instance 0
RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric
crossbar statistics instance 1 location 0/RSP0/CPU0
Tue Dec 17 02:59:36.376 EST
Port statistics for xbar:1 port:2
==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read
: 17347357500
Ingress Channel Utilization Count
: 5
Output Buffer Queued Packet Count
: 1
Egress Packet Count Since Last Read
: 19006087016
Egress Channel Utilization Count
: 4
.

Port statistics for xbar:1 port:16

==============================
Hi priority stats (unicast)
===========================
Ingress Packet Count Since Last Read
Egress Packet Count Since Last Read

: 49365
: 323

Low priority stats (multicast)

===========================
Ingress Packet Count Since Last Read
Egress Packet Count Since Last Read

: 1623
: 716

.
Total
Total
Total
Total

Unicast In:
Unicast Out:
Multicast In:
Multicast Out:

63038489128
63038489275
1625
1252

Instance 1

Line Cards
Second generation [Typhoon NP]

A9K-24x10GE

A9K-2x100GE, A9K-1x100GE

A9K-MOD80

A9K-MOD160

MPAs
20x1GE
2x10GE
4x10GE
8x10GE
1x40GE
2x40GE

A9K-36x10GE

Line Cards
Notes on Typhoon line cards

SE and TR versions

SE System Edge: More memory, scalable QoS

TR Transport: Basic QoS, mostly sufficient for Transport or core-facing

CPU and memory

Processor: Quad core processor

RAM: 8GB

A9K-SIP-700 for TDM

Different architecture
Supports a variety of channelized and clear OC192/48/12/3 STM64/16/4/1 T3/1 E3/1

RSP 3 Switch Fabric

Line Card Architecture

Second generation Ethernet line cards
LC CPU
Complex

Physical 1/10/40/100
Gbps
No frame processing there

Typhoon NP
FIB, MAC, ACL, QoS,
encap/decap, LPTS, all hw
features, buffer, i/f stats
60 Gbps 45 Mpps
bidirectional
Or 120 Gbps unidirectional

Fabric Interface ASIC

2PQ+1BE into fabric, and
VOQ
System priority queueing
Separate unicast and
multicast queueing
Super-framing and buffering

RSP 3
Switch
Fabric

forwarding slice
physical
interfaces

NP
Typhoon NP

physical
interfaces

FIA

NP

physical
interfaces

FIA

NP

physical
interfaces

replicate slices of
components to add
density and
performance

FIA

FIA

NP

physical
interfaces
physical
interfaces

FIA

NP
NP

FIA

LC Fabric
Complex

Line Card Architecture

24x 10 GE: A9K-24X10GE-SE and A9K-24X10GE-TR
3x10GE
SFP +
3x10GE
SFP +

3x10GE
SFP +

3x10GE
SFP +
3x10GE
SFP +
3x10GE
SFP +
3x10GE
SFP +

3x 10G

3x 10G

3x 10G

3x 10G

3x 10G

3x 10G

3x 10G

CPU
Typhoon
NP
Typhoon
NP

FIA

Typhoon
NP
Typhoon
NP

Typhoon
NP
Typhoon
NP
Typhoon
NP
Typhoon
NP

FIA
FIA

Switch Fabric
ASIC

3x10GE
SFP +

3x 10G

RSP 3
Switch
Fabric

Switc
h
Fabric
RSP
0

Switc
h
Fabric

FIA

RSP
1

Line Card Architecture

2x 100 GE: A9K-2X100GE-SE and A9K-2X100GE-TR
CPU
100G

Ingress
Typhoon
NP

FIA

Egress
Typhoon
NP

FIA

RSP 3 Switch
100GEFabric

100G

100G

Ingress
Typhoon
NP

100G

Egress
Typhoon
NP

100GE
MAC/PHY

FIA

Switch Fabric ASIC

MAC/PHY

Switch
Fabric
RSP0

Switch
Fabric

FIA
MUX
FPG
A

RSP1

Line Card Architecture

Modular line card: A9K-MOD160-SE and A9K-MOD160-TR
CPU
Supported
MPA

Typhoon
NP

FIA

Typhoon
NP

FIA

RSP 3
Switch Fabric

RSP 1x40GE
3 Switch Fabric

2x40GE

20xGE
Supported
MPA

Typhoon
NP

FIA

1x40GE
2x40GE
2x10GE
4x10GE
20xGE

Typhoon
NP

Switch Fabric ASIC

2x10GE
4x10GE

Switch
Fabric
RSP0

Switch
Fabric

FIA

RSP1

Line Card Architecture

Modular line card: A9K-MOD80-SE and A9K-MOD80-TR
CPU
Supported
MPA

Typhoon
NP

RSP 3
Switch Fabric

FIA

RSP 1x40GE
3 Switch Fabric

20xGE
Supported
MPA
1x40GE

2x10GE
4x10GE
20xGE

Typhoon
NP

FIA

Switch Fabric ASIC

2x10GE
4x10GE

Switch
Fabric
RSP0

Switch
Fabric
RSP1

ASR 9001 Architecture

A router + LC in 2 RU
MPAs
2,4x10GE
20xGE
1x40GE

SFP+ 10GE
On-board
4x10 SFP+
ports

SFP+ 10GE
Internal
EOBC

SFP+ 10GE

LC
CPU

SFP+ 10GE
MPAs
2,4x10GE
20xGE
1x40GE

Typhoon
NP

FIA

RP
CPU

Switch Fabric ASIC

FIA

Typhoon
NP

ASR 9001-S Architecture

A router + LC in 2 RU
MPAs
2,4x10GE
20xGE
1x40GE

SFP+ 10GE
On-board
2x10 SFP+
ports

SFP+ 10GE
Internal
EOBC

LC
CPU

RP
CPU

Switch Fabric ASIC

FIA

Typhoon
NP

Line Cards
Third generation [Tomahawk NP]

A9K-8X100G

MPO24 TO 10X DUPLEX LC SM

CPAK

MPO24 TO 10X DUPLEX LC MM

Tomahawk Line Card

Notes on 8x 100G line card

SE and TR versions

CPU and memory

Processor: Six core processor

RAM: SE 24 GB TR 12 GB

Port breakout

SE System Edge: More memory, scalable QoS

TR Transport: Basic QoS, mostly sufficient for Transport or core-facing

1x 100G or 2x 40G or 10x 10G with breakout cables (and box)

Router(config)#hw-module 0/2/cpu0 port 3 breakout 10xTenGigE
Interface TenGigE 0/slot/bay/port/breakout

Power control per slice [110W/slice]

Router(config)#hw-module power saving slice 3 location 0/2/cpu0

Line Card Architecture

8x 100 GE: A9K-8X100G

CPAK 0
Slice 0

PHY

NP

FIA

PHY

NP

FIA

CPAK 1
CPAK 2
Slice 1

CPAK 3

Switch
Fabric

CPAK 4

PHY

Slice 2

NP

CPAK 5

PHY
CPAK 7

FIA
Up to
14x115G

CPAK 6
Slice 3

LC CPU

NP

FIA

Tomahawk Interface Flexibility

Single CPAK Product ID Three SW selectable Options
Configurable 100GE Interconnect
Options for 10GE interfaces:
Interface HunGigE 0/x/y/z
Breakout Interface Convention:
Rack/Slot/Bay/Port (phy)/Breakout#

LGX
Panel
CPAK-100G-LR4

10GE Interconnect Options

hw-module 0/x/cpu0 port z breakout 10xTenGigE
Interface TenGigE 0/x/y/z/0
Interface TenGigE 0/x/y/z/1

Interface TenGigE 0/x/y/z/9

CPAK-100G-SR10

CPAK-10X10G-LR

DUPLEX SC TO
LC /SC/ST SM

MPO24 TO 10X
DUPLEX
LC/SC/ST MM

Interface FortyGigE 0/x/y/z/0

Interface FortyGigE 0/x/y/z/1

LGX
Panel

MPO24 TO 10X
DUPLEX LC
/SC/ST SM

LGX
Panel

40GE Interconnect Options

hw-module 0/x/cpu0 port z breakout 2xFortyGigE

LGX
Panel

CPAK-2X40G-LR4

LC TO DUPLEX
LC/SC/ST SM

ASR 9000v Satellite

Local or remote 10G to 1G fan out
Local or remote
All configuration done
on host

ASR 9000
host

Up to 4x 10GE
links or bundle

L2/3/4 operation done

on host
No local switching on
satellite
Ingress & egress QoS
done on host

1GE ports can be

assigned to specific
10G ports or bundle

ASR 9000v
satellite

Up to 44x 1GE
user links

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

Traffic: Transit, For us, and Exceptions

Differentiate on ingress NP
RP

Line Card (LC)

Transit

CPU
LC-CPU

Look up, re-write,

forward

sRP

For us
Destined to RP, or
link local scope
Punt to RP or ingress
LC CPU

ucode

PIFIB
(TCAM, dynamic)

Ingress
NP

Exception
MTU failure, TTL
failure, etc. Should
have been transit
Punt to LC CPU

ucode

F
A
B
RI
C

CPU

Egress LC

Exceptions, & some Forus traffic: L2, BFD, ARP

For-us traffic processed by
LPTS: L3 control traffic,
management
Transit traffic

Control [For-us] Traffic

For Us Frame Path

From ingress NP to RP CPU or LC CPU

Control /
for-us
packets

For Us Frame Path

The internal FIB [IFIB]
RP/0/RSP0/CPU0:rasr9000-2w-b#show lpts ifib brief
Mon Dec

9 11:58:43.726 EST

Slice
-------RAWIP4
RAWIP4
RAWIP4
RAWIP4
RAWIP4
BGP4
BGP4
BGP4
BGP4
UDP4
UDP4
TCP4
TCP4
TCP4
TCP4
TCP4
ISIS
ISIS

VRF-ID
-------TRAFFIC
default
default
default
default
default
default
default
default
default
default
default
default
default
default
default
default
default

L4
-----112
RSVP
RSVP
RSVP
IGMP
TCP
TCP
TCP
TCP
UDP
UDP
TCP
TCP
TCP
TCP
TCP
-

Interface
Dlvr
Local-Address,Port Remote-Address,Port
------------ ----------- -------------------------------------Te0/0/0/2.200 0/RSP0/CPU0 [Link] any
Gi0/1/0/3.400 [0x0003]
any any
BE1
[0x0003]
any any
Physical slot mask: 0003 is first
Te0/0/0/4.100 [0x0003]
any any
2 slots in 9006: RSP0 & RSP1
any
[0x0003]
any any
any
0/RSP0/CPU0 [Link],179 [Link],48462
any
0/RSP0/CPU0 [Link],179 [Link],53724
any
0/RSP0/CPU0 any,179 [Link]
any
0/RSP0/CPU0 any,179 [Link]
any
[0x0003]
[Link],646 [Link]
any
[0x0003]
[Link],646 [Link]
Mg0/RSP1/CPU0/0 0/RSP0/CPU0 any,23 any
any
[0x0003]
[Link],59192 [Link],646
Gi0/1/0/1
0/RSP0/CPU0 any,38751 any
Mg0/RSP1/CPU0/0 0/RSP0/CPU0 any,38751 any
any
[0x0003]
[Link],63675 [Link],646
BE1
[0x0003]
- Te0/0/0/4.100 [0x0003]
- -

For Us Frame Path

From ingress NP to RP CPU or LC CPU

Internal I/O process

Interrupt switching
process

For Us Frame Path

Processes to watch on RP CPU
netio on RP CPU
Example for BGP
from unknown
LPTS relaxed for
simulation

RP/0/RSP0/CPU0:rasr9k-1y#show processes cpu location 0/RSP0/CPU0 |

exclude " 0%
0%
0%"
Wed Nov 28 01:36:52.203 UTC
CPU utilization for one minute: 26%; five minutes: 25%; fifteen minutes: 22%
PID
1Min
94243
3%
254074 23%

5Min
3%
22%

15Min Process
3% spp
19% netio

RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib hardware police location 0/0/CPU0 | e$

Wed Nov 28 01:23:10.907 UTC
------------------------------------------------------------Node 0/0/CPU0:
------------------------------------------------------------Burst = 100ms for all flow types
------------------------------------------------------------FlowType
Policer Type
Cur. Rate Def. Rate Accepted
Dropped
---------------------- ------- ------- ---------- ---------- -------------------- ----------------.
BGP-default
108
Local
150000
1500
89395477
3845915191
.
TCP-default
164
Local
150000
2000
49872016
8066163019
.
-----------------------statistics:
Packets accepted by deleted entries: 19477
Packets dropped by deleted entries: 0
Run out of statistics counter errors: 0

For Us Frame Path

Processes to watch on LC CPU
netio and spp on RP
CPU
netio for internal
in/out (like ip input in
IOS)
spp for software
switched (similar to
interrupt switching in
IOS)

Example for for-us

fragments
LPTS relaxed for
simulation

RP/0/RSP0/CPU0:rasr9k-1y#show processes cpu location 0/0/CPU0 |

exclude " 0%
0%
0%"
Wed Nov 28 01:28:52.281 UTC

CPU utilization for one minute: 46%; five minutes: 48%; fifteen
minutes: 39%
PID
1Min
45085
22%
180316 23%

5Min
23%
23%

15Min Process
22% spp
23% netio

RP/0/RSP0/CPU0:rasr9k-1y#show
Wed Nov 28 01:23:10.907 UTC

lpts pifib hardware police location 0/0/CPU0

------------------------------------------------------------Node 0/0/CPU0:
------------------------------------------------------------Burst = 100ms for all flow types
------------------------------------------------------------FlowType
Policer Type
Cur. Rate Def. Rate Accepted
---------------------- ------- ------- ---------- ---------- -------------------Fragment
101
Local
100000
2500
142076716
.
UDP-default
163
Local
1000000
3500
38336274
.
------------------------

statistics:
Packets accepted by deleted entries: 19477
Packets dropped by deleted entries: 0
Run out of statistics counter errors: 0

Dropped
----------------5033837819
2376859

For Us Frame Path

Traffic rate from/to each CPU
RP/0/RSP0/CPU0:rasr9k-1y#show netio rates
location 0/RSP0/CPU0
Thu Jan 3 06:56:28.745 UTC

RP/0/RSP0/CPU0:rasr9k-1y#show netio rates

location 0/0/CPU0
Thu Jan 3 06:56:20.329 UTC

Netio packet rate for node 0/RSP0/CPU0

----------------------------------Current rate
(updated 0 seconds ago):
Input:
82811 pkts/s
Output:
100 pkts/s
Driver Output:
100 pkts/s

Netio packet rate for node 0/0/CPU0

----------------------------------Current rate
(updated 0 seconds ago):
Input:
14759 pkts/s
Output:
0 pkts/s
Driver Output: 14760 pkts/s

1 minute rate
Input:
Output:
Driver Output:

(updated 7 seconds ago):

82668 pkts/s
98 pkts/s
98 pkts/s

5 minute rate
Input:
Output:
Driver Output:

(updated 7 seconds ago):

57073 pkts/s
65 pkts/s
65 pkts/s

RSP: routing protocols,

management, etc.

1 minute rate
Input:
Output:
Driver Output:

(updated 0 seconds ago):

14770 pkts/s
0 pkts/s
14771 pkts/s

5 minute rate
Input:
Output:
Driver Output:

(updated 0 seconds ago):

10178 pkts/s
0 pkts/s
10179 pkts/s

LC: fragments, BFD, ARP, L2, etc.

Control Plane Protection

LPTS flow type policers
RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib hardware police location 0/0/CPU0
Sun Dec 2 06:07:36.931 UTC
-------------------------------------------------------------

Node 0/0/CPU0:
-------------------------------------------------------------

Pre-Internal FIB

Burst = 100ms for all flow types

-------------------------------------------------------------

FlowType

Policer Type

Cur. Rate

Def. Rate

Accepted

Dropped

---------------------- ------- ------- ---------- ---------- -------------------- --------------------

unconfigured-default
Fragment
OSPF-mc-known
OSPF-mc-default
OSPF-uc-known
OSPF-uc-default
ISIS-known
ISIS-default
.
BGP-known
BGP-cfg-peer
BGP-default
PIM-mcast-default
PIM-mcast-known
PIM-ucast
IGMP

100
101
102
103
104
105
143
144

Static
Local
Static
Local
Static
Local
Static
Local

2500
0
2000
0
2000
0
2000
0

2500
2500
2000
1500
2000
1000
2000
1500

0
0
0
53
0
0
20890
0

0
0
0
26
0
0
0
0

106
107
108
109
176
110
111

Static
Static
Local
Local
Static
Static
Static

2500
2000
0
0
2000
1500
3000

2500
2000
1500
2000
2000
1500
3000

4070
17
335787
0
0
0
0

0
0
15570288947
0
0
0
0

Control Plane Protection

LPTS flow type policers
FlowType

Policer Type

Cur. Rate

Def. Rate

Accepted

Dropped

---------------------- ------- ------- ---------- ---------- -------------------- -------------------.

ICMP-local
ICMP-app
ICMP-control
ICMP-default
ICMP-app-default
LDP-TCP-known
LDP-TCP-cfg-peer
LDP-TCP-default
LDP-UDP
All-routers
LMP-TCP-known
LMP-TCP-cfg-peer
LMP-TCP-default
LMP-UDP
RSVP-UDP
RSVP-default
RSVP-known
IKE
IPSEC-known
IPSEC-default
MSDP-known
.

112
152
140
153
152
113
114
115
116
117
168
169
170
171
118
154
177
119
120
121
122

Static
Local
Static
Local
Local
Static
Static
Local
Static
Local
Static
Static
Local
Local
Static
Local
Static
Static
Static
Local
Static

1500
100
1000
100
100
2500
2000
0
2000
0
2500
2000
0
0
2000
0
7000
100
400
0
300

1500
1500
1000
1500
1500
2500
2000
1500
2000
1000
2500
2000
1500
2000
2000
500
7000
100
400
100
300

20044
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

Control Plane Protection

LPTS flow types: BGP example
RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib hardware police location 0/0/CPU0
Wed Nov 28 03:01:48.271 UTC
-------------------------------------------------------------

Node 0/0/CPU0:
-------------------------------------------------------------

Burst = 100ms for all flow types

-------------------------------------------------------------

Established session packets

Configured peer packets
BGP packets from unknown

FlowType
Policer Type Cur. Rate Def. Rate
---------------- ------- ----- --------- --------.
BGP-known
106
Local 50000
2500

2590

BGP-cfg-peer

107

Static 2000

2000

13

BGP-default

108

Local

1500

138918630

3848639925

400000

.
-----------------------statistics:
Packets accepted by deleted entries: 19477
Packets dropped by deleted entries: 0
Run out of statistics counter errors: 0

Accepted
Dropped
--------------- ----------

Control Plane Protection

Customize LPTS flow rates
RP/0/RSP0/CPU0:rasr9k-1y(config)#lpts pifib hardware police location 0/0/CPU0
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow isis default rate 0
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)#flow bgp configured rate 500
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow bgp default rate 0
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow pim multicast default rate 0
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow icmp application rate 100
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow icmp default rate 100
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow ldp tcp default rate 0
RP/0/RSP0/CPU0:rasr9k-(config-pifib-policer-per-node)# flow all-routers rate 0
.

RP/0/RSP0/CPU0:rasr9k-1y#show running-config lpts pifib hardware police location 0/0/CPU0

Sun Dec 2 06:29:11.493 UTC
lpts pifib hardware police location 0/0/CPU0
flow bgp default rate 0
flow pim multicast default rate 0
flow icmp application rate 100
flow icmp default rate 100
flow ldp tcp default rate 0
flow all-routers rate 0
flow lmp tcp default rate 0
flow lmp udp rate 0
flow rsvp default rate 0
flow ipsec default rate 0
flow msdp default rate 0
flow ssh known rate 0
flow ssh default rate 0

Control Plane Protection

LPTS flow policers
RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib hardware police location 0/0/CPU0
Sun Dec 2 06:32:04.344 UTC
-------------------------------------------------------------

Node 0/0/CPU0:
-------------------------------------------------------------

Burst = 100ms for all flow types

-------------------------------------------------------------

FlowType
---------------------unconfigured-default
Fragment
OSPF-mc-known
OSPF-mc-default
OSPF-uc-known
OSPF-uc-default
ISIS-known
ISIS-default
TCP-known
TCP-listen
TCP-cfg-peer
TCP-default
Mcast-known
RADIUS
TACACS
NTP-default
NTP-known

Policer
------100
101
102
103
104
105
143
144
156
157
158
164
159
174
175
126
180

Type
------Static
Local
Static
Local
Static
Local
Static
Local
Static
Static
Static
Local
Static
Local
Static
Local
Local

Cur. Rate
---------2500
0
2000
0
2000
0
2000
0
2500
2500
2000
0
2500
0
2000
0
0

Def. Rate
---------2500
2500
2000
1500
2000
1000
2000
1500
2500
2500
2000
2000
2500
2000
2000
200
200

Accepted
-------------------0
0
0
54
0
0
21078
0
0
0
0
95977990
0
0
0
0
0

Dropped
-------------------0
0
0
27
0
0
0
0
0
0
0
1995220219679
0
0
0
0
0

Control Plane Protection

LPTS PIFIB

LPTS is the group of processes to transport for-us packets

Destination is either RP CPUs or ingress LC CPU
5 queues of different priorities in Typhoon NP

LPTS policers

Configured LC flow rate applied to LC, if not then configured global flow rate applied, if
not then a default rate applied
Enforced by each NP
Flow entries created and installed based on: configuration and neighbor flow state [e.g.
BGP TCP]

For Us Packet Forwarding Entries

LPTS flow entries
RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib

hardware entry location 0/0/CPU0

Sun Dec 2 00:46:50.573 UTC
Node: 0/0/CPU0:
---------------------------------------M - Fabric Multicast;
L - Listener Tag; T - Min TTL;
F - Flow Type;
DestNode - Destination Node;
DestAddr - Destination Fabric queue;
SID - Stream ID;
Po - Policer; Ct - Stats Counter;
Lp - Lookup priority; Sp - Storage Priority;
Ar - Average rate limit; Bu - Burst;
HAr - Hardware Average rate limit; HBu - Hardware
Burst;
Cir - Committed Information rate in HAL
Rsp - Relative sorting position;
Rtp - Relative TCAM position;
na - Not Applicable or Not Available
.

Show flow policers

in LC TCAM.

BGP-known
Session already
established. Flow
parameters in
hardware policer.

.
---------------------------------------------------VRF ID
: 0x60000000
Destination IP : any
Source IP
: [Link]
Is Fragment
:0
Interface
: any
M/L/T/F
: 1/IPv4_STACK/0/BGP-known
DestNode
: FGID 48
DestAddr
: 48
SID
:7
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:58549
Destination Port : 179
Ct
: 0x5f0690
Accepted/Dropped : 3189/0
Lp/Sp
: 1/255
# of TCAM entries : 1
HPo/HAr/HBu/Cir :
15794309/2500pps/1250ms/2500pps
State
: Entry in TCAM
Rsp/Rtp
: 5/15
---------------------------------------------------.

For Us Packet Forwarding Entries

LPTS flow entries
---------------------------------------------------VRF ID
: 0x60000000
Destination IP : any
Source IP
: [Link]
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/IPv4_STACK/255/BGP-known
DestNode
: 48
DestAddr
: 48
SID
:7
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:179
Destination Port : 41243
Ct
: 0x5f0670
Accepted/Dropped : 0/0
Lp/Sp
: 1/255
# of TCAM entries : 1
HPo/HAr/HBu/Cir :
15794309/2500pps/1250ms/2500pps
State
: Entry in TCAM
Rsp/Rtp
: 6/16
----------------------------------------------------

BGP-known
Active session
with a configured
peer.

BGP-cfg-peer
Open to receiving
peer attempts to
establish.

---------------------------------------------------VRF ID
: 0x60000000
Destination IP : any
Source IP
: [Link]
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/IPv4_LISTENER/255/BGP-cfgpeer
DestNode
: 48
DestAddr
: 48
SID
:8
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:any
Destination Port : 179
Ct
: 0x5f0340
Accepted/Dropped : 0/0
Lp/Sp
: 1/255
# of TCAM entries : 1
HPo/HAr/HBu/Cir :
15794310/2000pps/1000ms/2000pps
State
: Entry in TCAM
Rsp/Rtp
: 7/17
----------------------------------------------------

For Us Packet Forwarding Entries

LPTS flow entries
---------------------------------------------------VRF ID
: any
Destination IP : any
Source IP
: any
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/BGP4_FM/0/BGP-default
DestNode
: 48
DestAddr
: 48
SID
:9
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:179
Destination Port : any
Ct
: 0x5f01b0
Accepted/Dropped : 300890/13952472426
Lp/Sp
: 1/0
# of TCAM entries : 1
HPo/HAr/HBu/Cir : 15794311/0pps/750ms/1pps
State
: Entry in TCAM
Rsp/Rtp
: 13/23
----------------------------------------------------

BGP-default
Any TCP from port
179 not matching
previous entries.

BGP-default
Any TCP to port
179 not matching
previous entries.

---------------------------------------------------VRF ID
: any
Destination IP : any
Source IP
: any
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/BGP4_FM/0/BGP-default
DestNode
: 48
DestAddr
: 48
SID
:9
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:any
Destination Port : 179
Ct
: 0x5f01a0
Accepted/Dropped : 0/0
Lp/Sp
: 1/0
# of TCAM entries : 1
HPo/HAr/HBu/Cir : 15794311/0pps/750ms/1pps
State
: Entry in TCAM
Rsp/Rtp
: 15/25
----------------------------------------------------

For Us Packet Forwarding Entries

LPTS flow entries
---------------------------------------------------VRF ID
: any
Destination IP : any
Source IP
: any
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/TCP4_FM/0/TCP-default
DestNode
: 48
DestAddr
: 48
SID
:9
L4 Protocol
: TCP
TCP flag byte : any
Source port
: Port:any
Destination Port : any
Ct
: 0x5f0170
Accepted/Dropped : 95947801/1817465391676
Lp/Sp
: 1/0
# of TCAM entries : 1
HPo/HAr/HBu/Cir : 15794367/0pps/1000ms/1pps
State
: Entry in TCAM
Rsp/Rtp
: 24/34
----------------------------------------------------

TCP-default
Any IPv4 TCP not
matched by
previous entries.

Raw-default
Any IPv4 not
matched by
previous entries.

---------------------------------------------------VRF ID
: any
Destination IP : any
Source IP
: any
Is Fragment
:0
Interface
: any
M/L/T/F
: 0/RAWIP4_FM/0/Raw-default
DestNode
: 48
DestAddr
: 48
SID
:9
L4 Protocol
: any
Source port
: any
Destination Port : any
Ct
: 0x5f01f0
Accepted/Dropped : 10272/18857
Lp/Sp
: 1/0
# of TCAM entries : 1
HPo/HAr/HBu/Cir : 15794370/0pps/1250ms/1pps
State
: Entry in TCAM
Rsp/Rtp
: 28/38
----------------------------------------------------

Control Plane Protection

LPTS: PIFIB ACL-Based Policers
RP/0/RSP0/CPU0:rasr9000-2w-b#show running-config
ipv4 access-list

RP/0/RSP0/CPU0:rasr9000-2w-b#show running-config
lpts pifib hardware police

Thu Apr

Thu Apr

3 18:21:35.034 EDT

ipv4 access-list PE
10 remark PE LOOPBACKS
20 permit ipv4 [Link] [Link] any
40 deny ipv4 any any
!
ipv4 access-list CORE
10 permit ipv4 [Link]/16 any
20 deny ipv4 any any
!
ipv4 access-list OFFENDERS
10 permit ipv4 host [Link] any
20 permit ipv4 host [Link] any
30 permit ipv4 [Link]/29 any
40 deny ipv4 any any
!
.

3 18:25:22.831 EDT

lpts pifib hardware police

acl PE rate 11000
flow bgp known rate 6000
flow bgp configured rate 1000
flow bgp default rate 0
acl CORE rate 33000
acl OFFENDERS rate 0
!

per-ACL PPS
rate

Control Plane Protection

LPTS: PIFIB ACL-Based Policers
RP/0/RSP0/CPU0:rasr9000-2w-b#show lpts pifib hardware entry brief location 0/1/CPU0
Thu Apr

3 18:28:57.713 EDT

Node: 0/0/CPU0:
---------------------------------------L3 - L3 Protocol;L4 - Layer4 Protocol; Intf - Interface;
Dest - Destination Node; V - Virtual;
na - Not Applicable or Not Available;
LU - Local chassis fabric unicast;
LM - Local chassis fabric multicast;
RU - Multi chassis fabric unicast;
RM - Multi chassis fabric multicast;
def - default
Offset
-----.
18
19
20
21
22
23
24
25
26
.

L3
VRF id
L4
Intf
Dest
---- ------------ ------ --------------- ---------

laddr,Port raddr,Port
----------

IPV4
IPV4
IPV4
IPV4
IPV4
IPV4
IPV4
IPV4
IPV4

any,any any,any
any,179 [Link],28603
any,40607 [Link],179
any,38362 [Link],646
any,646 [Link],any
any,646 [Link],any
any,179 [Link],any
any,179 [Link],any
any,23 any,any

default
default
default
default
default
default
default
default
default

RSVP
TCP
TCP
TCP
UDP
UDP
TCP
TCP
TCP

Gi0/1/0/3.400
any
any
any
any
any
any
any
any

Local
LU(30)
LU(30)
LM[3]
LM[3]
LM[3]
LU(30)
LU(30)
LU(30)

acl name
---------------------------

CORE
CORE
CORE
CORE
CORE
CORE

Control Plane Protection

LPTS: PIFIB ACL-Based Policers
RP/0/RSP0/CPU0:rasr9000-2w-b#show

lpts pifib hardware entry

type ipv4 start-index 12 num-entries 7 location 0/1/CPU0
Thu Apr

3 18:40:54.467 EDT

VRF ID
: 0x60000000
Destination IP
: any
Source IP
: [Link]
Is Fragment
: 0
Interface
: any
M/L/T/F
: 0/IPv4_STACK/0/BGP-known
DestNode
: 48
DestAddr
: 48
SID
: 7
L4 Protocol
: TCP
TCP flag byte
: any
Source port
: Port:28603
Destination Port : 179
Ct
: 0x612050
Accepted/Dropped : 5058/0
Lp/Sp
: 1/255
# of TCAM entries : 1
HPo/HAr/HBu/Cir/acl:
14876914/33000pps/33000ms/33000pps/CORE
.

VRF ID
: 0x60000000
Destination IP
: any
Source IP
: [Link]
Is Fragment
: 0
Interface
: any
M/L/T/F
: 1/IPv4_LISTENER/0/LDP-UDP
DestNode
: FGID 3
DestAddr
: 3
SID
: 7
L4 Protocol
: UDP
Source port
: Port:any
Destination Port : 646
Ct
: 0x612060
Accepted/Dropped : 16214/0
Lp/Sp
: 1/255
# of TCAM entries : 1
HPo/HAr/HBu/Cir/acl:
14876914/33000pps/33000ms/33000pps/CORE
State
: Entry in TCAM
Rsp/Rtp
: 16/30
---------------------------------------------------

Control Plane Protection

LPTS Excessive Flow Trap

Control Plane Protection

LPTS Excessive Flow Trap: Configuration
RP/0/RSP0/CPU0:rasr9000-2w-b#show running-config lpts punt excessive-flow-trap
Tue Mar 11 11:47:47.820 EDT
lpts punt excessive-flow-trap
penalty-rate arp 50
penalty-rate icmp 50
penalty-rate igmp 100
penalty-rate ip 100
penalty-timeout arp 5
penalty-timeout icmp 5
penalty-timeout igmp 2
penalty-timeout ip 4
non-subscriber-interfaces
!

Policing for-us from offending source instead of dropping for flow type from all peers
Penalizing bad actor sub-interface on major protocols: IP, IGMP, ICMP, ARP, DHCP,
PPP, PPPoE, L2TP. Potentially impacting several protocols from offending peer
Not enabled by default
Check if default penalty rates and timeouts are acceptable in your case

Control Plane Protection

LPTS Excessive Flow Trap: Default & Configured Penalties
RP/0/RSP0/CPU0:rasr9000-2w-b#show lpts punt excessive-flow-trap information
.
Police
Penalty
Rate (pps)
Timeout (mins)
Protocol
Default Config
Default Config
Punt Reasons
------------------------------------------------ARP
10
50
15
5
ARP
Reverse ARP
Dynamic ARP Inspection (DAI)
ICMP
.

10

50

15

ICMP

IGMP

10

100

15

IGMP
IGMP Snoop
MLD Snoop

IPv4/v6

10

100

15

IP Subscriber (IPSUB)
IPv4 options
IPv4 FIB
IPv4 TTL exceeded
IPv4 fragmentation needed
IPv4/v6 adjacency
IPV4/v6 unknown IFIB
UDP-known

Control Plane Protection

LPTS Excessive Flow Example: A VRRP Flooding
LC/0/0/CPU0:Mar 11 12:52:09.059 : flowtrap[187]: %OS-FLOWTRAP-4-BAD_ACTOR_INTF_DETECTED : Excessive VRRP
flow detected on interface TenGigE0/0/0/5.511. The interface will be penalty-policed at 10 pps for 15
minutes.
RP/0/RSP0/CPU0:rasr9000-2w-b#show lpts
.
FlowType
Policer Type
---------------------- ------- ------.
VRRP
148
Static
.

pifib hardware police location 0/0/CPU0

Cur. Rate Def. Rate Accepted
Dropped
TOS Value
---------- ---------- -------------------- -------------------- ---------1000

1000

804133

40681182

01234567

RP/0/RSP0/CPU0:rasr9000-2w-b#show lpts punt excessive-flow-trap all location 0/0/CPU0

Tue Mar 11 13:04:35.545 EDT
Interface: TenGigE0/0/0/5.511
Intf Handle: 0x04001740
Location: 0/0/CPU0
Protocol: ****
Punt Reason: VRRP
Penalty Rate: 10 pps
Penalty Timeout: 15 mins
Time Remaining: 8 mins 22 secs
LC/0/0/CPU0:Mar 11 13:24:33.899 : flowtrap[187]: %OS-FLOWTRAP-4-BAD_ACTOR_INTF_CLEARED : Interface
TenGigE0/0/0/5.511 cleared from penalty-policing by timeout.

Control Plane Protection

LPTS Excessive Flow Example: A VRRP Flooding
RP/0/RSP0/CPU0:rasr9000-2w-b#show vrrp
Tue Mar 11 13:07:23.623 EDT

IPv4 Virtual Routers:

A
|
|
Interface
vrID Prio A
Te0/0/0/5.500 100 100
Te0/0/0/5.501 101
90
Te0/0/0/5.502 102 100
Te0/0/0/5.503 103
90
Te0/0/0/5.504 104 100
Te0/0/0/5.505 105
90
Te0/0/0/5.506 106 100
Te0/0/0/5.507 107
90
Te0/0/0/5.508 108 100
Te0/0/0/5.509 109
90
Te0/0/0/5.510 110 100
Te0/0/0/5.511 111
90
Te0/0/0/5.512 112 100
Te0/0/0/5.513 113
90
Te0/0/0/5.514 114 100
Te0/0/0/5.515 115
90
Te0/0/0/5.516 116 100
Te0/0/0/5.517 117
90
.

indicates IP address owner

P indicates configured to preempt
|
P State
Master addr
VRouter addr
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Master
local
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]
P Master
local
[Link]
P Backup
[Link]
[Link]

Exception Punt

Traffic: Transit, For us, and Exceptions

Differentiate on ingress NP
RP

Line Card (LC)

Transit

CPU
LC-CPU

Look up, re-write,

forward

sRP

For us
Destined to RP, or
link local scope
Punt to RP or ingress
LC CPU

ucode

PIFIB
(TCAM, dynamic)

Ingress
NP

Exception
MTU failure, TTL
failure, etc. Should
have been transit
Punt to LC CPU

ucode

F
A
B
RI
C

CPU

Egress LC

Exceptions, & some Forus traffic: L2, BFD, ARP

For-us traffic processed by
LPTS: L3 control traffic,
management
Transit traffic

Control Plane Protection

LPTS exception punt

Handles transit exceptions, some protocols, and snooping

Exceptions are transit that needs special processing [examples: MTU failure, TTL
exhaustion]
Some protocols handled by LC CPU [BFD, ARP, CDP]
IGMP snooping

Punted to LC CPU

Exception is IGMP snooping, punted to RSP CPUs

Policers

Configured LC rate applied to LC, if not then a default rate applied

No global rate configuration option. But a pre-configuration per LC option
Enforced by each NPs microcode
More policers are added in newer releases

Control Plane Protection

Exception punt policers
RP/0/RSP0/CPU0:rasr9k-1y#show lpts pifib hardware static-police location 0/0/CPU0
Sun Dec 2 06:42:23.474 UTC
------------------------------------------------------------Node 0/0/CPU0:
------------------------------------------------------------Burst = 100ms for all flow types
------------------------------------------------------------Punt Reason
SID
Flow Rate Burst Rate Accepted
Dropped
----------------------- --------------- --------- --------- ----------- ---------PUNT_INVALID
NETIO_LOW
400
400
0
0
PUNT_ALL
NETIO_HI
1000
200
0
0
CDP
NETIO_CRUCIAL
50
50
11763
0
ARP
ARP
5000
5000
75
0
RARP
NETIO_CRUCIAL
1000
200
0
0
LOOP
NETIO_LOW
1000
200
33448
0
BUNDLE_PROTO_PUNT
LACP
1000
200
0
0
UNKNOWN_OSI
NETIO_LOW
1000
200
0
0
IGMP_SNOOP
NETIO_MED
4000
2000
0
0
DIAGS
DIAG
1000
200
11132
0
PUNT_NO_MATCH
NETIO_LOW
200
200
0
0
IPV4_TTL_ERROR
NETIO_LOW
500
500
0
0
IPV4_FRAG_NEEDED_PUNT
NETIO_LOW
10000
10000
0
0
IPV4_BFD
BFD
500000
500000
0
0
DROP_PACKET
NETIO_LOW
100
20
0
0
PUNT_ADJ
NETIO_LOW
300
300
3
0
IPV6_LINK_LOCAL
NETIO_HI
2000
2000
1000
0

Destination
----------Local
Local
Local
Local
Local
Local
Local
Local
0x0030 (0/RSP0/CPU0)
Local
Local
Local
Local
Local
Local
Local
Local

Control Plane Protection

Customize punt policer rates
RP/0/RSP0/CPU0:rasr9k-1y(config)#lpts punt police location 0/0/CPU0
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol cdp rate 50
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol arp rate 5000
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol ipv4 options rate 100
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# exception icmp rate 200
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# exception ipv4 ttl-error rate 500
.
RP/0/RSP0/CPU0:rasr9k-1y#show running-config lpts punt police location 0/0/CPU0
Sun Dec 2 07:05:30.358 UTC
lpts punt police location 0/0/CPU0
exception invalid rate 400
protocol cdp rate 50
protocol arp rate 5000
protocol ipv4 options rate 100
exception icmp rate 200
exception ipv4 ttl-error rate 500
exception ipv4 fragment rate 10000
exception adjacency rate 300
exception acl-deny rate 50
exception ipv6 ttl-error rate 500
exception ipv6 fragment rate 10000
exception mpls fragment rate 10000
exception mpls ttl-error rate 500
!

Control Plane Protection

Customize punt policer rates pre-configure
RP/0/RSP0/CPU0:rasr9k-1y(config)#lpts punt police location preconfigure 0/4/CPU0
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol cdp rate 50
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol arp rate 5000
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# protocol ipv4 options rate 100
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# exception icmp rate 200
RP/0/RSP0/CPU0:rasr9k-1(config-punt-policer-per-node)# exception ipv4 ttl-error rate 500
.
RP/0/RSP0/CPU0:rasr9k-1y#show running-config | begin lpts punt police location preconfigure
0/4/CPU0
Sun Dec 2 07:05:30.358 UTC
lpts punt police location 0/0/CPU0
exception invalid rate 400
protocol cdp rate 50
protocol arp rate 5000
protocol ipv4 options rate 100
exception icmp rate 200
exception ipv4 ttl-error rate 500
exception ipv4 fragment rate 10000
exception adjacency rate 300
exception acl-deny rate 50
exception ipv6 ttl-error rate 500
exception ipv6 fragment rate 10000
exception mpls fragment rate 10000
!

Control Plane Protection

Monitoring Hints

A TCL script to periodically check and log excessive drops:

[Link]
[Link][65755]: LPTS threshold (80%) exceeded for flow type Rawdefault on 0/2/0, 102.513333333% of 250 pps in last 60 seconds

To clear punt/exception Accepted/Dropped counters:

#clear controller np counters all location

ASR 9000v Satellite

Host-satellite operation: Control
Discovery

CPU

MAC-DA

MAC-SA

Control VID

ASR 9000v Satellite

Payload/FCS

CPU

ASR 9000 Host

Like CDP

Heartbeat
One per second

TCP control
connection

RP/0/RSP0/CPU0:rasr9000-2w-a#show nv satellite protocol

Ether 3 brief
Sat Dec 14 02:43:00.278 EST
Interface
Sat-ID Status
-------------- ------ -----------------------------BE3
100
Satellite Ready

discovery interface Bundle-

Discovered links
----------------------Te0/1/0/3, Te0/1/1/3

RP/0/RSP0/CPU0:rasr9000-2w-a#show tcp brief | include [Link]

Sat Dec 14 02:47:59.152 EST
0x1002e004 0x6000000d
0
0 [Link]:17514
[Link]:13680

ESTAB

RP/0/RSP0/CPU0:rasr9000-2w-a#show nv satellite protocol control satellite 100 brief

Sat Dec 14 02:48:36.020 EST
Sat-ID IP Address
Protocol state Channels
------ ------------ -------------- ----------------------------------100
[Link] Connected
Ctrl, If-Ext L1, If-Ext L2, X-link,
VICL, Soft Reset, Inventory,
EnvMon, Alarm, Platform
RP/0/RSP0/CPU0:rasr9000-2w-a# show nv satellite status satellite 100 brief
Sat Dec 14 02:59:56.752 EST
Sat-ID Type
IP Address
MAC address
State
------ -------- ------------ -------------- -------------------------------100
asr9000v [Link] 8478.ac01.349c Connected (Stable)

Management Traffic

Management
Management Plane Protocols

Simple Network Management Protocol (SNMP)

Telnet
SSH
SFTP
FTP
TFTP
Secure Copy Protocol (SCP)
TACACS+
RADIUS
NetFlow (also used by the Data Plane as that is where the traffic comes from)
Network Time Protocol (NTP)
Syslog

Management Traffic
Management interfaces: Out-of-band, in-band, and global

No communication permitted between inband and out-of-band

Management VRF is not necessary, but looks cleaner

Management Traffic
Out-of-Band: Virtual address, interfaces, and protocols
RP/0/RSP0/CPU0:rasr9k-1y#show running-config
.

vrf MGMT
address-family ipv4 unicast
!

rp mgmtethernet forwarding ! If LC<>Mgmt

forwarding is needed
!

ipv4 virtual address vrf MGMT [Link]

[Link]
ipv4 virtual address use-as-src-addr
!

interface Loopback1 ! If needed

vrf MGMT
ipv4 address [Link] [Link]
!

interface MgmtEth0/RSP0/CPU0/0
vrf MGMT
ipv4 address [Link] [Link]
!

interface MgmtEth0/RSP0/CPU0/1
shutdown
!

interface MgmtEth0/RSP1/CPU0/0
vrf MGMT
ipv4 address [Link] [Link]
!

!
key chain OSPF-MGMT
key 1
accept-lifetime 00:00:00 january 01 2012 23:59:59
december 31 2014

key-string password 153B382537

send-lifetime 00:00:00 january 01 2012 23:59:59
december 31 2014

cryptographic-algorithm HMAC-MD5
!
accept-tolerance 90000
!
router ospf OSPF
vrf MGMT
router-id [Link]
area 0
authentication message-digest keychain OSPF-MGMT
interface Loopback1
!
interface MgmtEth0/RSP0/CPU0/0
!
interface MgmtEth0/RSP1/CPU0/0
!
!
!

Management Traffic
Out-of-Band: Virtual address, interfaces, and protocols
RP/0/RSP0/CPU0:rasr9k-1y#show running-config
control-plane management-plane out-of-band
Wed Dec 5 00:45:07.132 UTC

control-plane
management-plane
out-of-band
vrf MGMT
interface MgmtEth0/RSP0/CPU0/0
allow SSH peer
address ipv4 [Link]/24
!

allow SNMP peer

address ipv4 [Link]
!
!

interface MgmtEth0/RSP1/CPU0/0
allow SSH peer
address ipv4 [Link]/24
!

allow SNMP peer

address ipv4 [Link]
!
!
!

RP/0/RSP0/CPU0:rasr9k-1y#show mgmt-plane
Wed Dec 5 00:46:26.162 UTC
Management Plane Protection
inband interfaces
---------------------interface - TenGigE0_0_0_2/
ssh configured peer v4 allowed - [Link]/24

outband interfaces
---------------------interface - MgmtEth0_RSP0_CPU0_0/
ssh configured peer v4 allowed snmp configured peer v4 allowed interface - MgmtEth0_RSP1_CPU0_0/
ssh configured peer v4 allowed snmp configured peer v4 allowed -

[Link]/24
[Link]
[Link]/24
[Link]

Management Traffic
In-band: If OOB is not available
RP/0/RSP0/CPU0:rasr9k-1y#show running-config
control-plane management-plane inband
Tue Dec 11 23:05:11.597 UTC
control-plane
management-plane
inband
interface TenGigE0/0/0/2
allow SSH peer
address ipv4 [Link]/24
!
!
!
!
!

RP/0/RSP0/CPU0:rasr9k-1y#show mgmt-plane
Wed Dec 5 00:46:26.162 UTC
Management Plane Protection
inband interfaces
---------------------interface - TenGigE0_0_0_2/
ssh configured peer v4 allowed - [Link]/24
outband interfaces
---------------------interface - MgmtEth0_RSP0_CPU0_0/
ssh configured peer v4 allowed snmp configured peer v4 allowed interface - MgmtEth0_RSP1_CPU0_0/
ssh configured peer v4 allowed snmp configured peer v4 allowed -

[Link]/24
[Link]
[Link]/24
[Link]

Management Protection
Authentication, authorization, and accounting
tacacs source-interface Loopback1 vrf MGMT
tacacs-server host [Link] port 49
key 7 13061E010803
!

taskgroup operation
task read bgp
task read isis
task write ospf
inherit taskgroup operator
!

taskgroup provisioning
inherit taskgroup netadmin
description PROVISIONING GROUP
!

usergroup PROVISIONING
taskgroup netadmin
taskgroup provisioning
!

aaa authentication login default local

aaa accounting exec default start-stop group
tacacs+ none
aaa authorization exec default group tacacs+
local none
aaa authorization commands default group tacacs+
none
aaa authentication login default group tacacs+
local

RP/0/RSP0/CPU0:rasr9k-1y(config)#do show aaa

taskgroup
Wed Dec 5 01:40:50.022 UTC
Task group 'operation'
Inherits from task group 'operator'
Task IDs included directly by this group:
Task:
bgp : READ
Task:
isis : READ
Task:
ospf : READ
WRITE
Task group 'operation' has the following combined
set
of task IDs (including all inherited groups):
Task:
basic-services : READ
WRITE
EXECUTE
DEBUG
Task:
bgp : READ
Task:
cdp : READ
Task:
diag : READ
Task:
ext-access : READ
EXECUTE
Task:
isis : READ
Task:
logging : READ
Task:
ospf : READ
WRITE
Task group 'provisioning'
Inherits from task group 'netadmin
.

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

Unicast Packet
Forwarding

Traffic: Transit, For us, and Exceptions

Differentiate on ingress NP
Transit

RP

Line Card (LC)

CPU

Look up, re-write,

forward

LC-CPU

sRP

For us
Destined to RP, or
link local scope
Punt to RP or ingress
LC CPU

ucode

PIFIB
(TCAM, dynamic)

Ingress
NP

Exception
MTU failure, TTL
failure, etc. Should
have been transit
Punt to LC CPU

ucode

F
A
B
RI
C

CPU

Egress LC

Exceptions, & some Forus traffic: L2, BFD, ARP

For-us traffic processed by
LPTS: L3 control traffic,
management
Transit traffic

Unicast Transit Frame Path

Physical > NP > FIA > Fabric > FIA > NP > Physical
3x 10G
3x10GE
SFP +

Typhoon

FIA

3x 10G
3x10GE
SFP +

Typhoon
FIA
3x 10G

Typhoon

Typhoon
3x 10G

3x10GE
SFP +

Typhoon

FIA

3x 10G
3x10GE
SFP +

Typhoon

Switch
Fabric

3x 10G
3x10GE
SFP +

FIA

Egress
Typhoon

100G

FIA

Ingress
Typhoon

100G

100GE
MAC/PHY

Typhoon

FIA

3x 10G
3x10GE
SFP +

100G

100GE
MAC/PHY

Switch Fabric ASIC

FIA

3x 10G
3x10GE
SFP +

Switch Fabric ASIC

3x10GE
SFP +

Ingress
Typhoon

Typhoon

A9K-24X10GE

Switch
Fabric

FIA

A9K-2X100GE

Egress
Typhoon

100G

Unicast Transit Frame Path

Forwarding

All frames take same path stages

Ingress physical => ingress NP => FIA => fabric => FIA => egress NP => egress
physical
Super-framing in fabric, and per super-frame load sharing

Two stage forwarding

Ingress NP: to which egress port, ingress encap (if tunneling) and ingress features
Egress NP: Adjacency, encap, and egress features

VOQ, and back-pressure signaling

Each FIA has a VOQ per each egress 10 Gbps port. More VOQs per 40 and 100
Gbps ports
Back pressure is signaled backwards from egress NP to ingress FIA for buffering

Unicast Two Stage Forwarding

By ingress NP and egress NP
First stage: lookup on ingress NPU
Egress NPU (or SFP: switch fabric port)

Second stage: lookup on egress NPU

Egress port and rewrite information

Fabric header and super-framing

3x 10G
3x10GE
SFP +

Typhoon

3x 10G
3x10GE
SFP +

Typhoon

FIA

3x 10G
3x10GE
SFP +

Typhoon

Switch
Fabric

Switch Fabric
ASIC

Typhoon

Switch Fabric ASIC

FIA

3x 10G
3x10GE
SFP +

FIA

Ingress
Typhoon

100G

100GE
MAC/PHY

FIA

Egress
Typhoon

100G

NP Feature Processing
Order of processing by ingress and egress NPs
Ingress NP

I/F
classification

Security ACL
classification

QOS
classification

Fwd lookup

*IFIB lookup

*IFIB action

QoS action

L2 rewrite

QOS policer
action

Security ACL
action

To fabric

From fabric
Egress NP

Security ACL
action
QoS+ policer
action

QOS
classification

Security ACL
classification

L2 rewrite

Fwd lookup

The NP FIB
From RP control plane to data plane NP
LDP

RSVP-TE

Static

LSD

BGP

OSPF

ISIS

EIGRP

RIB

RSP CPU

ARP
FIB [HW]

SW FIB

AIB

Adjacency
LC NPU

LC CPU

AIB: Adjacency Information Base

RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database

The NP FIB
RIB info: example
LDP

RSVP-TE

LSD

Static

BGP

OSPF

ISIS

EIGRP

RIB

RSP CPU

RP/0/RSP0/CPU0:asr#sh route [Link]/31

ARP

AIB

Routing entry for [Link]/31

Known via "isis isis1", distance 115, metric 20, type level-1
Installed Mar 2 17:58:12.251 for 00:00:47
FIB
Adjacency
SWDescriptor
FIB
Routing
Blocks
[Link], from [Link], via TenGigE0/1/0/3
LC NPU
Route metric is 20
No advertising protos.
LC CPU
AIB: Adjacency Information Base
RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database

The NP FIB
Line card adjacency
LDP

RSVP-TE

LSD

Static

BGP

OSPF

ISIS

EIGRP

RIB

RSP CPU

RP/0/RSP0/CPU0:asr#show adjacency summary location 0/1/CPU0

ARP

AIB

Adjacency table (version 26) has 19 adjacencies:

11 complete adjacencies
8 incomplete
adjacencies
FIB
Adjacency
SW
FIB
0 deleted adjacencies in quarantine list
8 adjacencies of type IPv4
LC NPU
8 complete adjacencies of type IPv4
LC CPU
0 incomplete
adjacencies
of typeInformation
IPv4
AIB: Adjacency
Base
0 deleted adjacencies
type Information
IPv4 in quarantine
RIB:ofRouting
Base
list
FIB: Forwarding Information Base
0 interface adjacencies
of type
IPv4
LSD: Label
Switch
Database
4 multicast adjacencies of type IPv4

The NP FIB
FIB entry in NP: example
RP/0/RSP0/CPU0:asr#sh cef
[Link] hardware
LDP
RSVP-TEingress loc 0/1/CPU0
Static
[Link]/31, version 1, internal 0x40000001 (0xb1d66c6c) [1],
0x0
(0xb1b4f758), 0x0 (0x0)
Updated Mar 2 17:58:11.987
local adjacency [Link]
Prefix Len 31, traffic index 0, precedence routine (0)
via [Link], TenGigE0/1/0/3,LSD
5 dependencies, weight 0, class 0
next hop [Link]
local adjacency
EZ:0 Leaf
============
Search ctrl-byte0:
0x3 ctrl-byte1:
0x8 ctrl-byte2:0x5
Leaf Action :
FORWARD
prefix length :
31
Search Control FlagsARP
:
match
:
1
valid:
1
done
:
0
ifib_lookup: SW
0 FIB
ext_lsp_array :
0
match_all_bit:
0
AIB
recursive
:
0
nonrecursive :
1
default_action:
1

Non Recursive Leaf:

------------------ldi ptr
: 10936 (0x2ab8)
rpf ptr :
0x0000

LC CPU
igp statsptr:0

BGP

OSPF

ISIS

EIGRP

RIB

RSP CPU

FIB

Adjacency
LC NPU

AIB: Adjacency Information Base

RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database

L3 Unicast Packet Journey

Mapping the port to NP and FIA

NP

! Example: Path from GigabitEthernet0/0/1/0 [Link] TO TenGigE0/4/0/20.6 [Link]

RP/0/RSP0/CPU0:rasr9k-1y#show controllers NP ports all location 0/0/CPU0

Fri Feb 22 15:57:32.307 UTC

Node: 0/0/CPU0:
---------------------------------------------------------------Map
NP Bridge Fia
Ports
-- ------ --- --------------------------------------------------0 -0
TenGigE0/0/0/0, TenGigE0/0/0/1, TenGigE0/0/0/2, TenGigE0/0/0/3
1 -1
GigabitEthernet0/0/1/0 - GigabitEthernet0/0/1/19
RP/0/RSP0/CPU0:rasr9k-1y#show controllers NP ports all location 0/4/CPU0
Fri Feb 22 15:55:22.370 UTC

Node: 0/4/CPU0:
---------------------------------------------------------------NP Bridge Fia
Ports
-- ------ --- --------------------------------------------------0 -0
TenGigE0/4/0/0, TenGigE0/4/0/1, TenGigE0/4/0/2
1 -0
TenGigE0/4/0/3, TenGigE0/4/0/4, TenGigE0/4/0/5
2 -1
TenGigE0/4/0/6, TenGigE0/4/0/7, TenGigE0/4/0/8
3 -1
TenGigE0/4/0/9, TenGigE0/4/0/10, TenGigE0/4/0/11
4 -2
TenGigE0/4/0/12, TenGigE0/4/0/13, TenGigE0/4/0/14
5 -2
TenGigE0/4/0/15, TenGigE0/4/0/16, TenGigE0/4/0/17
6 -3
TenGigE0/4/0/18, TenGigE0/4/0/19, TenGigE0/4/0/20
7 -3
TenGigE0/4/0/21, TenGigE0/4/0/22, TenGigE0/4/0/23

the port to NP
and FIA

L3 Unicast Packet Journey

The egress interface identifier

NP

RP/0/RSP0/CPU0:rasr9k-1y#show controllers pm interface tenGigE 0/4/0/20.6

Fri Feb 22 16:45:22.404 UTC
Get internal
Ifname(1): TenGigE0_4_0_20.6, ifh: 0xc001340 :
iftype
0x19
identifiers
egress_uidb_index 0x1d
ingress_uidb_index 0x1d
port_num
0x14
Interface handle: unique to
subslot_num
0x0
phy_port_num
0x14
logical sub-interface
channel_id
0x6
channel_map
0x0
lag_id
0x0
virtual_port_id
0x0
switch_fabric_port 0x136
Fabric port, shared between all port
in_tm_qid_fid0
0x0
in_tm_qid_fid1
0x0
sub-interfaces. The fabric destination.
in_qos_drop_base
0x0
out_tm_qid_fid0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
out_tm_qid_fid1
0x0
0x0
0x0
0x0
0x0
0x0
0x0
0x0
out_qos_drop_base 0x0
bandwidth
10000000 kbps
.

L3 Unicast Packet Journey

Ingress NP FIB
RP/0/RSP0/CPU0:rasr9k-1y#show cef ipv4 [Link]
hardware ingress location 0/0/CPU0
Fri Feb 22 17:40:35.887 UTC
[Link]/24, version 364, attached, connected,
internal 0xc0000c1 (ptr 0x8856b534) [1], 0x0
(0x873dde50), 0x0 (0x0)
Updated Feb 22 16:09:42.862
remote adjacency to TenGigE0/4/0/20.6
Prefix Len 24, traffic index 0, precedence
routine (0), priority 0
via TenGigE0/4/0/20.6, 2 dependencies, weight
0, class 0 [flags 0x8]
path-idx 0 [0x8a60a7bc 0x0]
remote adjacency
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0,
has_collapsed_ldi:0, collapse_bwalk_required:0,
ecdv2_marked:0
Leaf H/W Result:
Physical Result: 0x11dd0600 (LE)
Raw Data0: 0x91ad1000 00000001 360c0013
40000000
Raw Data1: 0x00000000 00000136 00180000
00000000
.

RX H/W Result on NP:1 [Adj ptr:0x3a (BE)]:

Raw Data0: 0x91000000 00000136 0c001340 00000000

adj_resolve_control_byte0
match: 1
valid: 1
iptunl_adj: 0
remote_rack: 0
adj_resolve_control_byte1
adj_down: 0
mgscp_en: 0
rx_lag_hash_en: 0
rx_lag_adj: 0
adj_resolve_control_byte2
rx_lag_adj: 0
rx_adj_null0: 0
rp_destined: 0
rx_punt: 0
rx_drop: 0
sfp/vqi
: 0x136
if_handle
: 0xc001340
.

L3 Unicast Packet Journey

Egress NP FIB
RP/0/RSP0/CPU0:rasr9k-1y#show cef ipv4 [Link]
hardware egress location 0/4/CPU0
Fri Feb 22 17:55:28.494 UTC
[Link]/32, version 0, internal 0x4080001 (ptr
0x8efc2704) [1], 0x0 (0x8e0f2210), 0x0 (0x0)
Updated Feb 22 16:13:35.351
local adjacency [Link]
Prefix Len 32, traffic index 0, Adjacencyprefix, precedence routine (0), priority 0
via [Link], TenGigE0/4/0/20.6, 3
dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0x91a2cef8 0x0]
next hop [Link]
local adjacency
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0,
has_collapsed_ldi:0, collapse_bwalk_required:0,
ecdv2_marked:0
Leaf H/W Result:
Physical Result: 0x11e80300 (LE)
Raw Data0: 0x91ad1000 8a030001 360c0013
40400000
.

TX H/W Result for NP:6 (index: 0x38a (BE)):

Raw Data0: 0x91080000 1d000000 dc050000 400b5f00
Raw Data1: 0x0000c006 01020000 00000000 00000000

adj_resolve_control_byte0
reserved: 0
egr_uidb_internal: 1
match: 1
valid: 1
iptunl_adj: 0
adj_resolve_control_byte1
tx_adj_null0: 0
tx_punt: 0
tx_drop: 0
default_action: 1
spare: 0
adj_resolve_control_byte2
spare: 0
spare_cb: 0
flags
gre_adj
: 0
uidb_index
: 0x1d00 (LE)
reserve_pad_word: 0
l3_mtu
: 1500
reserve_pad_1
: 0
adj_stats_index : 0x400b5f00
dest_mac
: 0x0000.c006.0102
ether reserved : 0000000000000000
.

L3 Unicast Packet Journey

GRE encap case
RP/0/RSP0/CPU0:rasr9k-1y#show cef vrf DEF ipv4
[Link] hardware ingress location 0/4/CPU0
Sat Feb 23 14:35:00.017 UTC
[Link]/24, version 1, attached, connected,
internal 0xc0000c1 (ptr 0x8e154de4) [1], 0x0
(0x8e0ec7c0), 0x0 (0x0)
Updated Feb 21 16:28:04.573
local adjacency point2point
Prefix Len 24, traffic index 0, precedence
routine (0), priority 0
via tunnel-ip25, 3 dependencies, weight 0,
class 0 [flags 0x8]
path-idx 0 [0x90fdd3b4 0x0]
local adjacency
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0,
has_collapsed_ldi:0, collapse_bwalk_required:0,
ecdv2_marked:0
Leaf H/W Result:

TX H/W Result for NP:6 (index: 0x28a (BE)):

.
adj_resolve_control_byte0
reserved: 0
egr_uidb_internal: 1
match: 1
valid: 1
iptunl_adj: 1
.
flags
gre_adj
: 1
uidb_index
: 0x1b00 (LE)
reserve_pad_word: 0
l3_mtu
: 1476
reserve_pad_1
: 0
adj_stats_index : 0x18005f00
GRE Adj
ip_src
: [Link]
tos
: 0
df
: 1
rsvd flag bits: 0
vrf_id
: 0

Physical Result: 0x11be0200 (LE)

Raw Data0: 0x11a50000 c9020000 00000000 00000000
Raw Data1: 0x00000000 00000000 00180000 0000a2ff

leaf_resolve_control_byte0
.

ip_dst
: [Link]
ttl
: 0xff
tos_reflect
: 1
encap_checksum: 0x40a3
reserved
: 0

L3 Unicast Packet Journey

GRE encap case: GRE adjacency
RP/0/RSP0/CPU0:rasr9k-1y#show cef vrf DEF
adjacency tunnel-ip 25 hardware ingress location
0/4/CPU0
Sat Feb 23 14:44:52.239 UTC
Display protocol is ipv4
Interface
Address
Type
Refcount
ti25
local

Prefix: [Link]/32
3

Adjacency: PT:0x8aa0c0c8 [Link]/32

Interface: ti25
GRE header:
0000004500400000a2fb2fff011414ac021414ac00080000
GRE tunnel adjacency
GRE tunnel info: 0x91b3b050 (0x1 3),
tos-propagate is set
Interface Type: 0x25, Base Flags:
0x2001 (0x90fdd3b4)
Nhinfo PT: 0x90fdd3b4, Idb PT:
0x8d8f8898, If Handle: 0x8000120
Dependent adj type: remote
(0x90fdd460)
Dependent adj intf: ti25
Ancestor If Handle: 0x0

TX H/W Result for NP:6 (index: 0x28a (BE)):

.
adj_resolve_control_byte0
reserved: 0
egr_uidb_internal:
1
match: 1
valid:
1
iptunl_adj: 1
.
flags
gre_adj
: 1

uidb_index
:
reserve_pad_word:
l3_mtu
:
reserve_pad_1
:
adj_stats_index :

:
:
:
:
:

0x1b00 (LE)
0
1476
0
0x18005f00

GRE Adj
ip_src
[Link]
tos
0xff
df
1
rsvd flag bits
0x40a3
vrf_id
0

: [Link]

ip_dst

: 0

ttl

: 1

tos_reflect

: 0

encap_checksum

: 0

reserved

L3 Unicast Packet Journey

IP to MPLS-TE case
RP/0/RSP0/CPU0:rasr9k-1y#show cef ipv4 [Link]
hardware ingress location 0/4/CPU0
Sat Feb 23 15:22:57.224 UTC
[Link]/24, version 259, internal 0x4004001
(ptr 0x8efba154) [1], 0x0 (0x8e0ece00), 0x440
(0x90dca470)
Updated Feb 22 11:03:15.593
Prefix Len 24, traffic index 0, precedence
routine (0), priority 3
via [Link], tunnel-te200, 5
dependencies, weight 0, class 0 [flags 0x0]
TE-NH H/W Result for 1st NP:0 (index: 0x3 (BE)):
.
tunnel_over_tunnel: 0
spare: 0
.
TE_local_label:
label: 16012
exp: 0
eos: 1
TE_tunnel_label:
label: 0
exp: 0
eos: 1
te_nh_stats_ptr: 0x880a5f
.

RX H/W Result for 1st NP:0 (index: 0x38 (BE)):

Raw Data0: 0x91000000 0000005c 00000640
00000000
adj_resolve_control_byte0
match: 1
valid: 1
iptunl_adj: 0
remote_rack: 0
adj_resolve_control_byte1
adj_down: 0
mgscp_en: 0
rx_lag_hash_en: 0
rx_lag_adj: 0
adj_resolve_control_byte2
rx_lag_adj: 0
rx_adj_null0: 0
rp_destined: 0
rx_punt: 0
rx_drop: 0
sfp/vqi
: 0x5c
if_handle
: 0x640
.

Multicast Packet
Replication

Multicast Transit Frame Path

Replication stages: per LC, per FIA, per NP, per interface

Multicast Transit Frame Path

Lookups and replications

Two MFIB lookups

Egress replication

Ingress NP tags frame with FGID and MGID, and ingress feature processing
FGID [Fabric Group ID]: to which LCs
MGID [Multicast Group ID]: Entry index to egress NPs
Egress NP replicates to egress interfaces, and egress feature processing

1st replication is at RSP switch fabric. Replicates to 1 copy per egress line card [FGID]
2nd replication at each egress LC switch fabric. Replicates to 1 copy per egress FIA [MGID]
3rd replication at each egress FIA. Replicates to 1 copy per egress NP [MGID]
4th replication at each egress NP. Replicates to 1 copy for each egress interface [MFIB]

Per flow load sharing

Ingress NP hashes to ingress FIA (like on a 100 Gbps interface)

FIA hashes to LC fabric links
LC fabric hashes to RSP fabric links
Similar per-flow load sharing over 2 links RSP fabric=>LC, LC fabric=>FIA, FIA=>NP

Multicast Fabric Group ID

FGID = destination card
Slot

Slot Mask

Slot

Slot Mask

Logical

Physical

Binary

Hex

Logical

Physical

Binary

Hex

LC7

1000000000

0x0200

LC3

0000100000

0x0020

LC6

0100000000

0x0100

LC2

0000010000

0x0010

LC5

0010000000

0x0080

LC1

0000001000

0x0008

LC4

0001000000

0x0040

LC0

0000000100

0x0004

RSP0

0000100000

0x0020

RSP1

0000000010

0x0002

RSP1

0000010000

0x0010

RSP0

0000000001

0x0001

LC3

0000001000

0x0008

LC2

0000000100

0x0004

LC1

0000000010

0x0002

LC0

0000000001

0x0001

9006

9010

Follows the sequence of slots in chassis

L3 Multicast Packet Journey

Reading the mRIB
RP/0/RSP0/CPU0:rasr9k-1y#show mrib route [Link] [Link] detail
Tue Feb 26 17:15:05.039 UTC
IP Multicast Routing Information Base
Entry flags: L - Domain-Local Source, E - External Source to the Domain,
.
([Link],[Link]) Ver: 0x5180 RPF nbr: [Link] Flags:,
PD: Slotmask: 0x41
MGID: 16903
0x41 = 0100 0001:
Up: 5d09h
LC0 & LC4 in 9010
Incoming Interface List
[physical slots 0, 6]
TenGigE0/0/0/2 Flags: A IC II LI, Up: 5d09h
Outgoing Interface List
TenGigE0/0/0/0 Flags: F IC NS II LI, Up: 4d08h
TenGigE0/0/0/1 Flags: F IC NS II LI, Up: 4d08h
TenGigE0/4/0/2.2 Flags: F NS LI, Up: 5d09h
TenGigE0/4/0/20.6 Flags: F IC NS II LI, Up: 00:59:25
GigabitEthernet0/0/1/19 Flags: F IC NS II LI, Up: 01:08:45

L3 Multicast Packet Journey

Reading the NP mFIB

NP

RP/0/RSP0/CPU0:rasr9k-1y#show mfib hardware route detail [Link] [Link] location 0/4/CPU0

Tue Feb 26 18:09:54.515 UTC
LC Type: Typhoon A9K-24x10GE-TR
.
Source: [Link]
Group: [Link]
Mask: 64 RPF Int: Te0/0/0/2
MGID: 16903
MLI: 5
Fabric Slotmask: 0x41
FGID: 0x41
Route Information
-----------------------------------------------------------------------NP B S DC PL PR PF DR RI
T OC
MF TR TE TD CD MI
Base
-----------------------------------------------------------------------0 F F F F F F F 0x640
0 1
F F F F F 0x0 0x5100d4
1 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
2 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
Outgoing interface
3 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
count per NP
4 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
5 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
6 F F F F F F F 0x640
1 1
F F F F F 0x0 0x5100d4
7 F F F F F F F 0x640
0 0
F F F F F 0x0 0x5100d4
-----------------------------------------------------------------------Software MGID Information
---------------------------------------------------MGID: 16903 Mask: 0x41 Old MGID: 0 Old Mask: 0x1
----------------------------------------------------

L3 Multicast Packet Journey

NP

Reading egress LC Fabric and FIA replication

FIA
NP

RP/0/RSP0/CPU0:rasr9k-1y#show controllers mgidprgm mgidindex 16903 location 0/0/CPU0

Tue Feb 26 17:35:10.026 UTC
Device
MGID-Bits
Client-Last-Modified
=======================================================
XBAR-0
FIA-0
FIA-1

11
10
10

MFIBV4
MFIBV4
MFIBV4

Fabric to

1st

MGID

&

2nd

FIA

FIA to 2nd NP

RP/0/RSP0/CPU0:rasr9k-1y#show controllers mgidprgm mgidindex 16903 location 0/4/CPU0

Tue Feb 26 17:35:15.417 UTC
Device
MGID-Bits
Client-Last-Modified
=======================================================
Fabric to 1st &
XBAR-0
1001
MFIBV4
FIA-0
1
MFIBV4
FIA-1
0
MFIBV4
None
FIA-2
0
MFIBV4
FIA-3
1
MFIBV4
FIA to 1st NP

4th FIA

Egress LC

L3 Multicast Packet Journey

Reading the hardware counters
RP/0/RSP0/CPU0:rasr9000-2w-a#show mfib vrf TRAFFIC hardware route statistics [Link]
[Link] location 0/1/CPU0
Thu Jan

9 22:09:04.997 EST

LC Type: Typhoon A9K-MOD80-SE

-------------------------------------------------------------------------Legend:
N:
NP ID
R:
Received
F:
Forwarded
P:
Punted to CPU
ID:
Ingress Drop
ED:
Egress Drop
Source: [Link] Group: [Link] Mask:64
------------------------------------------------------------------------NP
R(packets:bytes)/F(packets:bytes)/P(packets)/ID(packets)/ED(packets)
------------------------------------------------------------------------0
18326252:4948100612 / 0:0 / 0 / 0 / 0
1
0:0 / 18212304:4917314359 / 0 / 0 / 0
------------------------------------------------------------------------Interface Statistics:
------------------------------------------------------------------------C
Interface
F/P/D (packets:bytes)
------------------------------------------------------------------------1
Gi100/0/0/9
18212651:4917382603 / 0:0 / 0:0
-------------------------------------------------------------------------

Received on NP0
Forwarded on NP1

Forwarded on
interface [satellite]

L2 Frame Forwarding &

Flooding

L2 Frame Journey
L2 frame forwarding/flooding
RP/0/RSP0/CPU0:rasr9k-1y#show l2vpn forwarding
bridge-domain BRIDGES:DOMAIN-A hardware ingress
detail location 0/4/CPU0
Sun Feb 24 13:53:34.530 UTC

Bridge-domain name: BRIDGES:DOMAIN-A, id: 0,

state: up
MAC learning: enabled
MAC port down flush: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 2000, Action: limit, no flood,
Notification: syslog, trap
MAC limit reached: no
MAC Secure: enabled, Logging: enabled, Action: none
DHCPv4 snooping: profile not known on this node
Dynamic ARP Inspection: enabled, Logging: enabled
Dynamic ARP Inspection Address Validation:
IPv4 verification: enabled
Source MAC verification: enabled
Destination MAC verification: enabled
IP Source Guard: disabled, Logging: disabled
IGMP snooping: disabled, flooding: enabled

Bridge MTU: 1500 bytes

Number of bridge ports: 4
Number of MAC addresses: 2002
.

Bridge Domain: 0 NP 0
Flags: Virtual Table, Multicast Flooding, Learn
Enable, No Learn, Learn Drop
Num Members: 0, Learn Key: 0x00, Half Age: 5
fgid shg0: 0x0001, fgid shg1: 0x0041, fgid
shg2: 0x0041
PBB Core BD: 0, ISID: 0
Bridge Domain: 0 NP 1
Flags: Virtual Table, Multicast Flooding, Learn
Enable, No Learn, Learn Drop
Num Members: 0, Learn Key: 0x00, Half Age: 5
fgid shg0: 0x0001, fgid shg1: 0x0041, fgid
shg2: 0x0041
PBB Core BD: 0, ISID: 0
.
Bridge Domain: 0 NP 6
Flags: Virtual Table, Multicast Flooding, Learn
Enable, No Learn, Learn Drop
Num Members: 1, Learn Key: 0x00, Half Age: 5
fgid shg0: 0x0001, fgid shg1: 0x0041, fgid
shg2: 0x0041
PBB Core BD: 0, ISID: 0
Bridge Port 0
XID: 0x09b00001, Active
virtual
XID: 0x09b00001, Active
.

L2 Frame Journey
L2 frame forwarding/flooding
! CONTINUED
TenGigE0/4/0/20.101, state: oper up
Number of MAC: 0
Statistics:
packets: received 36731752, sent 14772099
bytes: received 2203905120, sent 886325940
Storm control drop counters:
packets: broadcast 0, multicast 0, unknown
unicast 2961034169
bytes: broadcast 0, multicast 0, unknown unicast
177662050140
Dynamic arp inspection drop counters:
packets: 0, bytes: 0
IP source guard drop counters:
packets: 0, bytes: 0
Platform Bridge Port context:
Ingress State: Bound
Flags: DAI, DAI-ipv4, DAI-src-MAC, DAI-dst-MAC, DAI-log,
MAC-SEC, MAC-SEC-log,
MAC-learn-disabled
MAC Security Actions: Drop, No Notify
Platform AC context:
Ingress AC: VPLS, State: Bound
Flags: Learn Limit - No Learn, Learn Limit - Drop, Storm
Control BCast,
Storm Control MCast, Storm Control UCast, Port
Level MAC Limit

XID: 0x09b00001, SHG: None

Ingress uIDB: 0x001c, Egress uIDB: 0x001c, NP: 6,
Port Learn Key: 0
.

NP6
Ingress uIDB:
Flags: DAI, DAI Notification, Dest MAC validation,
IP Addr Validation,
L2PT, L2, Source MAC validation, Status, Ext
Required, VLAN Ops,
VPLS
Stats Ptr: 0x000000, uIDB index: 0x001c, Wire Exp
Tag: 1
BVI Bridge Domain: 0, BVI Source XID: 0x00000000
VLAN1: 0, VLAN1 etype: 0x0000, VLAN2: 0, VLAN2
etype: 0x0000
L2 ACL Format: 0, L2 ACL ID: 0, IPV4 ACL ID: 0,
IPV6 ACL ID: 0
QOS ID: 0, QOS Format ID: 0
Local Switch dest XID: 0x09b00001
UIDB IF Handle: 0x0c000042, Source Port: 0, Num
VLANs: 0
Xconnect ID: 0x09b00001, NP: 6
Type: AC
Flags: Learn enable, Type 5, Learn limit no learn,
Learn limit drop,
Broadcast storm control, Multicast storm
control, Unknown unicast storm control,
VPLS
uIDB Index: 0x001c
Bridge Domain ID: 0, Stats Pointer: 0xf78122
Storm Control enabled for: Broadcast, Multicast,
Unknown Unicast, Pointer: 0x00001801
.

L2 MAC
MAC learning and synchronization
RP/0/RSP0/CPU0:rasr9k-1y#show l2vpn forwarding bridge-domain BRIDGES:DOMAIN-A mac-address
hardware ingress location 0/4/CPU0
Fri Feb 22 18:50:08.433 UTC
To Resynchronize MAC table from the Network Processors, use the command...
l2vpn resynchronize forwarding mac-address-table location <r/s/i>
Mac Address
Type
Learned from/Filtered on
LC learned Resync Age
-------------------------------------------------------------------------------0000.c001.0167 dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 18s
0000.c001.016b dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 11s
0000.c001.016c dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 9s
0000.c001.016d dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 18s
0000.c001.016e dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 20s
0000.c001.016f dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 8s
0000.c001.0171 dynamic Gi0/0/1/0.200
0/0/CPU0
0d 0h 0m 17s
0000.c001.0102 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 16s
0000.c001.0104 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 20s
0000.c001.0105 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 8s
0000.c001.0106 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 9s
0000.c001.0107 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 18s
0000.c001.0108 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 15s
0000.c001.0109 dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 3s
0000.c001.010a dynamic Te0/4/0/20.101
0/4/CPU0
0d 0h 0m 4s
.

Mapped to
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A

ASR 9000 Satellite

to/from Host

ASR 9000v Satellite

Traffic flow
MAC-DA

MAC-DA

MAC-SA

VLANs (OPT)

Payload

MAC-DA

MAC-SA

nV-tag

VLANs (OPT)

MAC-SA

VLANs (OPT)

Payload/FCS

ASR 9000v Satellite

ASR 9000 Host

nV tag identifies the satellite port traffic

No learning or switching on satellite

L2/3/4 and ingress/egress QoS done on host

Payload

ASR 9000v Satellite

Queueing: Satellite to host

Normal MQC QoS at

nv Ethernet ingress

P1: Satellite protocol

50mbps policed

Regular MQC, HQoS

P1+P2 + Normal

P2: control packet

1G policed

L2/L3 control packets

User data
Priority queue

Cos/IPP/EXP 5-7

User data
Normal queue

Cos/IPP/EXP 0-4

ASR 9000v

Implicit classification

P1 and P2 are strict priority

User data priority:normal 100:1 bandwidth

Fabric link

ASR 9000 Host

ASR 9000v Satellite

Queueing: Host to satellite
Priority propagation

P1: Satellite protocol

50 mbps policed
Non-blocking at
satellite

Shape the total bandwidth of nv Ethernet to the real

satellite access port link bandwidth: 10/100/1000MB
before send to fabric 10G port

Fabric link
ASR 9000v

nv Ethernet

Regular MQC, H-QoS

P1 +P2 + P3+ Normal

nv Ethernet

nv Ethernet
nv Ethernet
nv Ethernet

ASR 9000 Host

Regular MQC, H-QoS

P1+P2 + P3+ Normal

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

MPLS in the Data Plane

Main Label Uses

Forwarding
The destination for this label is
[FEC]. Usually a host address of an
LER
A path to a label destination is an
LSP
Intermediate nodes may not know
much about payload or the basis for
its forwarding
Ultimate destination may not need
the label [PHP]

Service

How to handle this payload [IP, L3VPN

VRF, L2VPN, PW, CEoP, control]
Significant to edge nodes. Forwarding
nodes along the path may not know
what it means

Forwarding

Without label

RP/0/RSP0/CPU0:rasr9000-2w-a#show cef ipv4 [Link]

.
remote adjacency to GigabitEthernet0/1/0/1
Prefix Len 32, traffic index 0, precedence routine (0), priority 1
via [Link], GigabitEthernet0/1/0/1, 4 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0x721f30e0 0x0]
next hop [Link]
remote adjacency

With label

IP in

RP/0/RSP0/CPU0:rasr9000-2w-a#show cef ipv4 [Link]

.
remote adjacency to GigabitEthernet0/1/0/1
Prefix Len 32, traffic index 0, precedence routine (0), priority 1
via [Link], GigabitEthernet0/1/0/1, 20 dependencies, weight 0, class 0 [flags
0x0]
path-idx 0 [0x723990b4 0x0]
MPLS in
MPLS out
next hop [Link]
remote adjacency
local label 111012
labels imposed {101000}

Label Operations
RP/0/0/CPU0:P101#show cef ipv4 [Link]/32
.
Prefix Len 32, traffic index 0, precedence n/a, priority 1
via [Link], tunnel-te181, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0xacc9d674 0x0]
next hop [Link]
local adjacency
Push
local label 101000
labels imposed {108000}

RP/0/0/CPU0:P101#show mpls forwarding

Thu Jun 6 09:16:22.581 EDT
Local Outgoing
Prefix
Label Label
or ID
------ ----------- -----------------101000 108000
[Link]/32
101001 102000
11210
101005 Pop
[Link]/32
101007 Exp-Null-v4 11211
.
101039 Unlabelled [Link]/32
101040 Unlabelled [Link]/32
.

Outgoing
Next Hop
Interface
------------ --------------tt181
[Link]
Gi0/0/0/0.112 [Link]
tt1111
[Link]
Gi0/0/0/1
[Link]
Gi0/0/0/0.112 [Link]
Gi0/0/0/0.112 [Link]

Bytes
Switched
-----------Swap
0
0
0
Pop top label
0

Unlabeled: Pop all to

0
bottom of stack
4591105

Aggregate Label

Bottom of stack label not sufficient for forwarding decision

Payload header has to be used for forwarding

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding

Mon Jun 17 21:05:46.166 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Label Label
or ID
Interface
------ ----------- ------------------ ------------ --------------Pop topmost & forward
.
111007 101000
[Link]/32
tt1111
[Link]
111014 Pop
PW([Link]:1) Gi0/1/0/3.1 point2point
111015 Aggregate
CUST-A: Per-VRF Aggr[V]
\
CUST-A
111016 Unlabelled [Link]/24[V] Gi0/1/0/3.200 [Link]

Pop all tags &

forward

Pop & lookup

Bytes
Switched
-----------375048
314906
6320
0

Aggregate Label vs. Non-Aggregate

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding vrf CUST-A detail
Mon Jun 17 21:31:10.474 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------111015 Aggregate
CUST-A: Per-VRF Aggr[V]
\
CUST-A
8240
Updated Jun 17 20:03:20.046
Path Flags: 0x10 [ ]
MAC/Encaps: 0/0, MTU: 0
Pop & lookup
Label Stack (Top -> Bottom): { }
Packets Switched: 65
111016 Unlabelled [Link]/24[V] Gi0/1/0/3.200 [Link]
Updated Jun 17 21:23:42.495
Version: 47, Priority: 3
MAC/Encaps: 18/18, MTU: 1500
Label Stack (Top -> Bottom): { Unlabelled }
Packets Switched: 0

Pop & forward

Aggregate Label: Example

RP/0/RSP0/CPU0:rasr9000-2w-a#show route vrf CUST-A ipv4 connected
Mon Jun 17 21:34:49.647 EDT
C

[Link]/24 is directly connected, 00:56:39, GigabitEthernet0/1/0/3.200

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding vrf CUST-A detail

Mon Jun 17 21:31:10.474 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------111015 Aggregate
CUST-A: Per-VRF Aggr[V]
\
CUST-A
8240
Updated Jun 17 20:03:20.046
Path Flags: 0x10 [ ]
Pop & lookup
MAC/Encaps: 0/0, MTU: 0
Label Stack (Top -> Bottom): { }
Packets Switched: 65
.

Non-Aggregate Label: Example

RP/0/RSP0/CPU0:rasr9000-2w-a#show route vrf CUST-A ipv4 static
Mon Jun 17 21:34:57.549 EDT
S

[Link]/24 [1/0] via [Link], 00:11:45

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding vrf CUST-A detail

Mon Jun 17 21:31:10.474 EDT
.

111016 Unlabelled [Link]/24[V] Gi0/1/0/3.200 [Link]

Updated Jun 17 21:23:42.495
Version: 47, Priority: 3
MAC/Encaps: 18/18, MTU: 1500
Label Stack (Top -> Bottom): { Unlabelled }
Packets Switched: 0

Pop & forward

Forwarding Labels

LDP: Label Distribution Protocol

Label Imposition (Push)

Label Swap

31

31

Label Swap
55

55

Label Disposition (Pop)

0

[Link]/32

To [Link]

CE

Use 77

PE

Use 31

Use 55

Use 60

Use 55

PE

Use 31

CE

Use 0

PE

CE

Use 90

CE
P

Binds and advertises labels for all IGP prefixes [Cisco default]

Multicast hellos for neighbor discovery. TCP 646 for session.

Hop by hop. No guarantee for end to end LSP.

PE

LDP: Control Plane: Neighbor Sessions

RP/0/0/CPU0:P101#show mpls ldp neighbor
Thu Jun

6 10:41:01.283 EDT

Peer LDP Identifier: [Link]:0

TCP connection: [Link]:31207 - [Link]:646; MD5 on
Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 180 sec)
Session Holdtime: 180 sec
State: Oper; Msgs sent/rcvd: 27745/27777; Downstream-Unsolicited
Up time: 2w2d
LDP Discovery Sources:
Targeted Hello ([Link] -> [Link], active)
Addresses bound to this peer:
[Link]
[Link]
[Link]
[Link]

[Link]

Peer LDP Identifier: [Link]:0

TCP connection: [Link]:35863 - [Link]:646; MD5 on
Graceful Restart: Yes (Reconnect Timeout: 120 sec, Recovery: 0 sec)
Session Holdtime: 180 sec
State: Oper; Msgs sent/rcvd: 3024/3020; Downstream-Unsolicited
Up time: 1d19h
LDP Discovery Sources:
Targeted Hello ([Link] -> [Link], active)
GigabitEthernet0/0/0/1
Addresses bound to this peer:
[Link]
[Link]
[Link]
[Link]

[Link]

My potential
next hops on
this neighbor

LDP: Control Plane: Label Binding

RP/0/0/CPU0:P101#show mpls ldp bindings
.
[Link]/32, rev 161
Local binding: label: 101005
Remote bindings: (1 peers)
Peer
Label
-----------------------[Link]:0
108009
[Link]/32, rev 116
Local binding: label: 101041
Remote bindings: (2 peers)
Peer
Label
-----------------------[Link]:0
108038
[Link]:0
111006
[Link]/32, rev 117
Local binding: label: 101039
Remote bindings: (2 peers)
Peer
Label
-----------------------[Link]:0
108031
[Link]:0
111007

[Link]/32, rev 118

Local binding: label: 101040
Remote bindings: (2 peers)
Peer
Label
-----------------------[Link]:0
108032
[Link]:0
111008
[Link]/32, rev 119
Local binding: label: 101042
Remote bindings: (2 peers)
Peer
Label
-----------------------[Link]:0
108033
Use the one [Link]:0
111009
matching
IGP
[Link]/32, rev 120
Local binding: label: 101043
route, if any
Remote bindings: (2 peers)
Peer
Label
-----------------------[Link]:0
108034
[Link]:0
111010
.

LDP: Forwarding: FIB and LFIB

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef ipv4 [Link]/32

IPv4 in

Wed Mar 19 12:25:01.496 EDT

[Link]/32, version 272, internal 0x4004001 0x0 (ptr 0x7238643c) [1], 0x0 (0x71635290),
0x450 (0x71e26460)
Updated Mar 19 12:23:44.913
remote adjacency to GigabitEthernet0/1/0/1
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via [Link], GigabitEthernet0/1/0/1, 20 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x0 [0x719ea954 0x0]
next hop [Link]
remote adjacency
local label 188017
labels imposed {108002}
RP/0/RSP0/CPU0:rasr9000-2w-b#show mpls forwarding labels 188017 detail
Wed Mar 19 12:25:05.202 EDT

Local Outgoing
Prefix
Outgoing
Next Hop
Label Label
or ID
Interface
------ ----------- ------------------ ------------ --------------188017 108002
[Link]/32
Gi0/1/0/1
[Link]
Updated Mar 19 12:23:44.913
Version: 272, Priority: 3
MAC/Encaps: 14/18, MTU: 1386
Label Stack (Top -> Bottom): { 108002 }
NHID: 0x5
Packets Switched: 19

Bytes
Switched
-----------1558

MPLS in

LDP: Forwarding: In the Forwarding Plane

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef ipv4
[Link]/32 hardware ingress location 0/1/CPU0
.

local adjacency [Link]

Prefix Len 32, traffic index 0, precedence n/a,
priority 3
via [Link], GigabitEthernet0/1/0/1, 21
dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 NHID 0x5 [0x8b15d134 0x0]
next hop [Link]
local adjacency
local label 188017
labels imposed {108002}
.

NR-LDI H/W Result for path 0 [index: 0x34c7 (BE),

common to all NPs]:
.

output_label: 108002
label_msb: 0x1a5e
exp: 0x0

label_lsb: 0x2
eos: 0x1

RX H/W Result for 1st NP:0 (index: 0x3b (BE)):

.

if_handle

: 0x3e0

TX H/W Result for NP:0 (index: 0x33d3 (BE)):

.

uidb_index
l3_mtu
adj_stats_index
dest_mac
.

:
:
:
:

0x900 (LE)
1386
0x381f61
0x000c.29f4.90c6

RP/0/RSP0/CPU0:rasr9000-2w-b#show mpls forwarding labels

188017 hardware ingress location 0/1/CPU0
Wed Mar 19 13:01:00.202 EDT
Local Outgoing Prefix
Outgoing Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ------ --------------- ------------ -----------188017 108002 [Link]/32 Gi0/1/0/1 [Link] N/A
.
NR-LDI H/W Result for path 0 [index: 0x34c7 (BE), common
to all NPs]:
.
output_label: 108002
label_msb: 0x1a5e
label_lsb: 0x2
exp: 0x0
eos: 0x1
.
RX H/W Result for 1st NP:0 (index: 0x3b (BE)):
.
if_handle
: 0x3e0
.
TX H/W Result for NP:0 (index: 0x33d3 (BE)):
.
uidb_index
: 0x900 (LE)
l3_mtu
: 1386
adj_stats_index : 0x381f61
dest_mac
: 0x000c.29f4.90c6
ether reserved : 0000000000000000
.

TE: RSVP
RP/0/0/CPU0:P104#show rsvp neighbors
Mon Jun 10 08:38:07.069 EDT
Global Neighbor: [Link]
Interface Neighbor
Interface
-------------------- -----------[Link]
GigabitEthernet0/0/0/0.124
Global Neighbor: [Link]
Interface Neighbor
Interface
-------------------- -----------[Link]
GigabitEthernet0/0/0/0.134
Global Neighbor: [Link]
Interface Neighbor
Interface
-------------------- -----------[Link]
GigabitEthernet0/0/0/0.145
Global Neighbor: [Link]
Interface Neighbor
Interface
-------------------- -----------[Link]
GigabitEthernet0/0/0/0.146
Global Neighbor: [Link]
Interface Neighbor
Interface
-------------------- -----------[Link]
GigabitEthernet0/0/0/0.1424

TE: RSVP
RP/0/0/CPU0:P104#show rsvp interface
Mon Jun 10 08:38:12.129 EDT
*: RDM: Default I/F B/W % : 75% [default] (max resv/bc0), 0% [default] (bc1)
Interface
MaxBW (bps) MaxFlow (bps) Allocated (bps)
MaxSub (bps)
----------- ------------ ------------- -------------------- ------------Gi0/0/0/0.124
38G
38G
3G ( 7%)
0
Gi0/0/0/0.134
38G
38G
2G ( 5%)
0
tt10452
0
0
0 ( 0%)
0
Gi0/0/0/0.145
38G
38G
0 ( 0%)
0
Gi0/0/0/0.146
38G
38G
5G ( 13%)
0
tt10454
0
0
0 ( 0%)
0
Gi0/0/0/0.1424
9500M
9500M
0 ( 0%)
0
tt10456
0
0
0 ( 0%)
0
tt10457
0
0
0 ( 0%)
0
tt10459
0
0
0 ( 0%)
0
tt10460
0
0
0 ( 0%)
0

FRR backup
tunnels at
headend

TE: Links
RP/0/0/CPU0:P104#show mpls traffic-eng link-management summary
Mon Jun 10 11:47:53.059 EDT
System Information::
Links Count
Flooding System
IGP Areas Count

: 5 (Maximum Links Supported 500)

: enabled
: 1

IGP Areas
---------IGP Area[1]:: IS-IS ISIS level 2
Flooding Protocol
: IS-IS
Flooding Status
: flooded
Periodic Flooding
: enabled (every 180 seconds)
Flooded Links
: 5
IGP System ID
: 0101.0010.4001
MPLS TE Router ID
: [Link]
IGP Neighbors
: 5

TE: Topology View [At Headend]

RP/0/0/CPU0:PE135#show mpls traffic-eng topology summary
Mon Jun 10 11:50:35.198 EDT
My_System_id: 0101.0113.5001.00 (IS-IS ISIS level-2)
My_BC_Model_Type: RDM
Signalling error holddown: 10 sec Global Link Generation 5292918
IS-IS ISIS level 2
Local System Id:
TE router ID configured:
in use:

0101.0113.5001
[Link]
[Link]

IGP Id: 0101.0010.8001.00, MPLS TE Id: [Link] Router Node

4 links
.
IGP Id: 0101.0111.2001.00, MPLS TE Id: [Link] Router Node (Overloaded)
2 links

.
IGP Id: 0101.0010.8001.05, Network Node
2 links
.
Total: 43 nodes (16 router, 27 network), 108 links
Grand Total: 43 nodes (16 router, 27 network)

108 links

TE: The Label Exchange

R9

R8
R3
R4

R2

Pop

R5

R1

Label 32
Label 49
Label 17

R6

R7
Label 22

Setup: Path (ERO = R1->R2->R6->R7->R4->R9)

Reply: Resv communicates labels & reserves bandwidth on each link

TE: The Path [The Unidirectional Tunnel]

R9

R8
R3
R4

R2

Pop

R5

R1

Label 32
Label 49
Label 17

R6

R7

Label 22
CSPF calculation & signaling initiated by headend based on its MPLS TE topology view.

Creates a P2P [or P2MP, MP2MP] unidirectional tunnel.

Signaling and periodic refreshing of state done using RSVP.

TE: Control Plane View

RP/0/0/CPU0:P104#show mpls traffic-eng tunnels tabular
Mon Jun 10 14:48:32.737 EDT
Tunnel
LSP
Destination
Source
Tun
FRR LSP Path
Name
ID
Address
Address State State Role Prot
----------------- ----- --------------- --------------- ------ ------ ---- ----*tunnel-te10452
7
[Link]
[Link]
up Inact Head Inact
*tunnel-te10454
2
[Link]
[Link]
up Inact Head Inact
.
*tunnel-te10460
2
[Link]
[Link]
up Inact Head Inact
P101_t181
25
[Link]
[Link]
up Inact Mid
P108_t811
25
[Link]
[Link]
up Inact Mid
autob_P101_t10150
24
[Link]
[Link]
up Inact Mid
autob_P101_t10152
25
[Link]
[Link]
up Inact Mid
.
autob_P108_t10868
13
[Link]
[Link]
up Inact Tail
autom_PE112_t1121
16
[Link]
[Link]
up Ready Mid
autom_PE124_t1241
16
[Link]
[Link]
up Ready Mid
autob_PE124_t1245
27
[Link]
[Link]
up Inact Mid
.
autom_rasr9000-2w
11
[Link]
[Link]
up Ready Mid
autom_rasr9000-2w
11
[Link]
[Link]
up Ready Mid
* = automatically created backup tunnel
+ = automatically created mesh tunnel

TE: Control Plane View

RP/0/0/CPU0:P104#show mpls traffic-eng tunnels brief
Mon Jun 10 15:03:38.965 EDT
TUNNEL NAME
*tunnel-te10452
*tunnel-te10454
*tunnel-te10456
*tunnel-te10457
*tunnel-te10459
*tunnel-te10460
P101_t181
P108_t811
autob_P101_t10150_
autob_P101_t10152_

DESTINATION
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]

STATUS
up
up
up
up
up
up
up
up
up
up

STATE
up
up
up
up
up
up
up
up
up
up

.
autom_rasr9000-2w10.101.112.1
up
autom_rasr9000-2w10.101.124.1
up
autom_rasr9000-2w10.101.125.1
up
autom_rasr9000-2w10.101.135.1
up
autom_rasr9000-2w10.101.137.1
up
* = automatically created backup tunnel
Displayed 6 (of 6) heads, 33 (of 33) midpoints, 6 (of 6) tails
Displayed 6 up, 0 down, 0 recovering, 0 recovered heads

up
up
up
up
up

TE: Control Plane View: Tunnel Headend

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls traffic-eng tunnels 11106 detail
Tue Jun 11 00:04:29.172 EDT
Name: tunnel-te11106 Destination: [Link] (auto-tunnel mesh)
Status:
Admin:
up Oper:
up
Path: valid
Signalling: connected
path option 10, type dynamic (Basis for Setup, path weight 22230)
G-PID: 0x0800 (derived from egress interface properties)
Bandwidth Requested: 1000000 kbps CT0
Creation Time: Fri May 31 16:37:30 2013 (1w3d ago)
Config Parameters:
Bandwidth: 1000000 kbps (CT0) Priority: 3 3 Affinity: 0x0/0xffff
Metric Type: TE (default)
.
Current LSP Info:
Instance: 2, Signaling Area: IS-IS ISIS level-2
Uptime: 1w3d (since Fri May 31 16:37:30 EDT 2013)
Outgoing Interface: GigabitEthernet0/1/0/1, Outgoing Label: 101048
Router-IDs: local
[Link]
downstream [Link]
Soft Preemption: None
.

TE: Control Plane View: Tunnel Midpoint

RP/0/0/CPU0:P104#show mpls traffic-eng tunnels 10152
Mon Jun 10 15:01:13.715 EDT
LSP Tunnel [Link] 10152 [25] is signalled, connection is up
Tunnel Name: autob_P101_t10152_Gi0_0_0_0.113_10.100.103.1 Tunnel Role: Mid
InLabel: GigabitEthernet0/0/0/0.124, 104054
OutLabel: GigabitEthernet0/0/0/0.145, 105016
Signalling Info:
Src [Link] Dst [Link], Tun ID 10152, Tun Inst 25, Ext ID [Link]
Router-IDs: upstream
[Link]
local
[Link]
downstream [Link]
Bandwidth: 0 kbps (CT0) Priority: 6 6 DSTE-class: no match
Soft Preemption: None
Path Info:
Incoming Address: [Link]
Incoming:
Explicit Route:
Strict, [Link]
Strict, [Link]
Strict, [Link]
Strict, [Link]
.

TE: Forwarding: Headend Forwarding Methods

Auto-route announce [IGP]

Forwarding adjacency [IGP]

Policy-based (& class-based) routing

Static routes

Pseudo-wire tunnel selection

TE: Forwarding: Headend

RP/0/0/CPU0:PE135#show mpls forwarding tunnels
Tue Jun 11 07:44:26.151 EDT
Tunnel
Outgoing
Outgoing
Next Hop
Bytes
Name
Label
Interface
Switched
-------- ----------- ------------ --------------- -----------tt13501 103049
Gi0/0/0/0.1335 [Link]
0
tt13502 103051
Gi0/0/0/0.1335 [Link]
0
tt13503 103048
Gi0/0/0/0.1335 [Link]
105560
tt13504 103047
Gi0/0/0/0.1335 [Link]
2600
tt13505 103037
Gi0/0/0/0.1335 [Link]
0
.
RP/0/0/CPU0:PE135#show mpls forwarding tunnels 13503 detail
Tue Jun 11 07:45:18.917 EDT
Tunnel
Outgoing
Outgoing
Next Hop
Bytes
Name
Label
Interface
Switched
-------- ----------- ------------ --------------- -----------tt13503 103048
Gi0/0/0/0.1335 [Link]
105560
Updated May 31 07:34:51.047
Version: 401, Priority: 2
MAC/Encaps: 18/22, MTU: 4456
Label Stack (Top -> Bottom): { 103048 }
Local Label: 135001
Packets Switched: 1015

TE: Hardware Forwarding: Headend

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef ipv4 [Link]/32 hardware ingress location 0/1/CPU0
Tue Jun 11 02:36:41.139 EDT
[Link]/32, version 183, internal 0x4000001 (ptr 0x885cd0d8) [1], 0x0 (0x87741040), 0x0
(0x0)
Updated Jun 11 02:19:01.772
Prefix Len 32, traffic index 0, precedence routine (0), priority 1
via [Link], tunnel-te18804, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0x8a77f2d8 0x0]
next hop [Link]
local adjacency
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:
Physical Result: 0x11a00200 (LE)
Raw Data0: 0x11850000 b9020000 00000000 00000000
Raw Data1: 0x00000000 00000000 00200000 0000a2ff
leaf_resolve_control_byte0
reserved: 0
ifib_lookup: 0
txadj_internal: 0
rec_fs: 0
.

match: 1

valid: 1

TE: Hardware Forwarding: Headend Cont.

TE-NH H/W Result for 1st NP:0 (index: 0x5 (BE)):

Raw Data0: 0x5100002d e6311a5e b170115f 0000008a

Raw Data1: 0x02000000 16000000 00000000 00000000
cb0
spare: 0
default_action:
backup_indication: 0
match:
rsvd: 0
valid:
cb1
spare_cb: 0
tp_path_ss:
te_nh_incomplete: 0
tunnel_over_tunnel: 0
spare:
cb2
spare: 0
te_nh_incomplete:
spare_cb: 0
TE_local_label:
label: 188003
exp: 0
eos:
TE_tunnel_label:
label: 108011
exp: 0
eos:
te_nh_stats_ptr: 0x70115f
merge_point_label:
.

1
1
1
0
0
0

1
1

TE: Hardware Forwarding: Headend Cont.

RX H/W Result for 1st NP:0 (index: 0x16 (BE)):

Raw Data0: 0x91000000 00000088 06000200 00000000

adj_resolve_control_byte0
match: 1
valid: 1
iptunl_adj: 0
remote_rack: 0
adj_resolve_control_byte1
adj_down: 0
rx_lag_hash_en: 0
rx_lag_adj: 0
adj_resolve_control_byte2
rx_lag_adj: 0
rp_destined: 0
rx_drop: 0
sfp/vqi
: 0x88
if_handle
: 0x6000200

mgscp_en: 0

rx_adj_null0: 0
rx_punt: 0

Egress interface

.
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers pm location 0/1/CPU0 | begin 6000200
Tue Jun 11 05:19:29.503 EDT
Ifname(2): GigabitEthernet0_1_0_1, ifh: 0x6000200 :
.

TE: Forwarding: Midpoint

RP/0/0/CPU0:P104#show mpls forwarding
Tue Jun 11 07:04:03.137 EDT
Local Outgoing
Prefix
Label Label
or ID
------ ----------- -----------------104003 Exp-Null-v4 10254
104004 Exp-Null-v4 10357
104005 Exp-Null-v4 12450
104010 106032
10352
.
104028 Exp-Null-v4 10750
104042 106003
181
104044 106004
12410
104046 Exp-Null-v4 10260
104047 105058
10257
104048 105007
10259
104049 106017
11210
Tunnel ID
104050 Exp-Null-v4 10153
104052 105059
10150
104054 105016
10152
104059 Exp-Null-v4 10196
104061 102021
18801
104062 102044
811
.

Outgoing
Next Hop
Bytes
Interface
Switched
------------ --------------- -----------Gi0/0/0/0.1424 [Link]
0
Gi0/0/0/0.1424 [Link]
0
Gi0/0/0/0.134 [Link]
0
Gi0/0/0/0.146 [Link]
0
Gi0/0/0/0.134 [Link]
Gi0/0/0/0.146 [Link]
Gi0/0/0/0.146 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.146 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.145 [Link]
Gi0/0/0/0.1424 [Link]
Gi0/0/0/0.124 [Link]
Gi0/0/0/0.124 [Link]

0
10747212
0
0
0
0
0
684820
1108
1392
0
0
1057172746

TE: Hardware Forwarding: Midpoint

RP/0/0/CPU0:P104#show mpls forwarding
Tue Jun 11 11:31:38.647 EDT
Local Outgoing
Prefix
Label Label
or ID
------ ----------- -----------------104062 102044
811
.

labels 104062 hardware ingress location 0/0/CPU0

Outgoing
Next Hop
Bytes
Interface
Switched
------------ --------------- -----------Gi0/0/0/0.124 [Link]
N/A

Labels in Labels: LDP in TE

RP/0/RSP0/CPU0:rasr9000-2w-a#traceroute [Link] source [Link]
Tue Jun 11 20:49:15.907 EDT
Type escape sequence to abort.
Tracing the route to [Link]
1
2
3
4
5
6

TE label

[Link] [MPLS: Label 101055 Exp 0] 4 msec 3 msec 2 msec

[Link] [MPLS: Label 101055 Exp 0] 2 msec 3 msec 2 msec
[Link] [MPLS: Label 103097 Exp 0] 1 msec 2 msec 2 msec
[Link] [MPLS: Label 105060 Exp 0] 2 msec 2 msec 2 msec
[Link] [MPLS: Label 107068 Exp 0] 2 msec 2 msec 2 msec
[Link] 2 msec * 1 msec

TE tunnel

RP/0/RSP0/CPU0:rasr9000-2w-a#traceroute [Link] source [Link]

Tue Jun 11 20:56:32.972 EDT
Type escape sequence to abort.
Tracing the route to [Link]
1
2
3
4
5

LDP explicit null

[Link] [MPLS: Labels 101055/0 Exp 0] 3 msec 3 msec 2 msec

[Link] [MPLS: Labels 103097/0 Exp 0] 1 msec 2 msec 1 msec
[Link] [MPLS: Labels 105060/0 Exp 0] 1 msec 2 msec 2 msec
[Link] [MPLS: Labels 107068/0 Exp 0] 2 msec 2 msec 2 msec
[Link] 2 msec * 3 msec

LDP in TE tunnel

Labels in Labels: LDP in TE with NNHOP FRR

RP/0/RSP0/CPU0:rasr9000-2w-a#traceroute [Link] source [Link]
.
1 [Link] [MPLS: Labels 101055/0 Exp 0] 3 msec 3 msec 3 msec
2 [Link] [MPLS: Labels 103097/0 Exp 0] 3 msec 3 msec 2 msec
3 [Link] [MPLS: Labels 104010/107068/0 Exp 0] 3 msec 2 msec 3 msec
4 [Link] [MPLS: Labels 106032/107068/0 Exp 0] 4 msec 4 msec 2 msec
5 [Link] [MPLS: Labels 0/107068/0 Exp 0] 3 msec 3 msec 3 msec
6 [Link] 3 msec * 2 msec

P104
104010

107068

106032

107068

LDP in TE tunnel
and FRR active
= 3 labels

P106
107068

107068

105060

PE111

101055

P101

103097

P103
PLR

P105

P107
MP

PE178

Labels in Labels: TE NNHOP PLR

RP/0/0/CPU0:P103#show mpls forwarding labels 103097 hardware ingress detail
Tue Jun 11 12:51:07.075 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------103097 105052
11111
Gi0/0/0/0.135 [Link]
N/A
Updated Jun 11 12:17:41.262
Path Flags: 0x400 [ BKUP-IDX:1 (0xacde6f2c) ]
Version: 1598, Priority: 2
MAC/Encaps: 18/22, MTU: 4456
Label Stack (Top -> Bottom): { 105052 }
Packets Switched: 0
.
RP/0/0/CPU0:P103#show mpls forwarding labels 103097 hardware ingress detail
Tue Jun 11 12:51:31.414 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------103097 107068
11111
tt10352
[Link]
N/A
Updated Jun 11 12:51:26.135
Version: 1675, Priority: 2
MAC/Encaps: 18/26, MTU: 4456
Label Stack (Top -> Bottom): { 104010 107068 }
.

FRR Ready

FRR Active

Labels in Labels: LDP in TE at Headend

RP/0/RSP0/CPU0:rasr9000-2w-a#show cef ipv4 [Link]/32 hardware egress location 0/1/CPU0
Tue Jun 11 21:41:12.866 EDT
[Link]/32, version 285, internal 0x4004001 (ptr 0x8854bcf8) [1], 0x0 (0x876e74a0), 0x450
(0x89cb6110)
Updated Jun 11 21:12:35.330
Prefix Len 32, traffic index 0, precedence routine (0), priority 1
via [Link], tunnel-te11111, 3 dependencies, weight 0, class 0 [flags 0x0]
path-idx 0 [0x8a7a27d8 0x0]
next hop [Link]
local adjacency
LDP label
local label 111011
labels imposed {0}
.
TE-NH H/W Result for 1st NP:0 (index: 0x34 (BE)):
Raw Data0: 0x5100001b 1ac118ab f138145f 0000008a
Raw Data1: 0x02000000 16000000 00000000 00000000
.

TE label [topmost]

TE_tunnel_label:
label: 101055
exp: 0
te_nh_stats_ptr: 0x38145f
merge_point_label:
label: 0
.

eos: 1

Labels in Labels: More Forwarding Labels

Flow Label

Flow Aware Transport Pseudo-Wire [FAT PW] for VPWS and VPLS
[Link]
onfiguration/guide/[Link]#wp1339194
Used for forwarding hashing, but it is at bottom of stack.

Service Labels

The Service Exchange

Native Service

Native Service

MPLS tunnel

2. Service Discovery

1. Setup Local
Service
Properties

PE2

PE1

MPLS / IP

3. Service Signaling

Discovery: Manual or BGP

Signaling: LDP or BGP

LDP Signaling: PW Example

RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn xconnect detail
.
PW: neighbor [Link], PW ID 1, state is up ( established )
PW class ONE, XC ID 0xc0000001
Encapsulation MPLS, protocol LDP
Source address [Link]
PW type Ethernet, control word enabled, interworking none
PW backup disable delay 0 sec
Sequencing not set
Load Balance Hashing: pw-label
Flow Label flags configured (Tx=1,Rx=1), negotiated (Tx=1,Rx=1)
PW Status TLV in use
Advertised
MPLS
Local
------------ -----------------------------Label
111014
Group ID
0x6000180
Interface
GigabitEthernet0/1/0/3.1
MTU
1504
Control word enabled
PW type
Ethernet
VCCV CV type 0x2
(LSP ping verification)
.

Received
Remote
----------------------------188014
0x6000180
GigabitEthernet0/1/0/3.1
1504
enabled
Ethernet
0x2
(LSP ping verification)

Forwarding: AC to PW
RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn forwarding interface g0/1/0/3.1 hardware ingress detail
location 0/1/CPU0
Mon Jun 17 23:18:54.890 EDT
Local interface: GigabitEthernet0/1/0/3.1, Xconnect id: 0xc40001, Status: up
Segment 1
AC, GigabitEthernet0/1/0/3.1, status: Bound
Statistics:
packets: received 2809, sent 2810
bytes: received 330634, sent 386882
packets dropped: PLU 0, tail 0
bytes dropped: PLU 0, tail 0
Segment 2
MPLS, Destination address: [Link], pw-id: 1, status: Bound
Pseudowire label: 188014
Control word enabled
Load-Balance-Type: pw-label
Flow Label flag: Tx=1
Statistics:
packets: received 2810, sent 2809
bytes: received 386882, sent 330634
packets dropped: PLU 0, tail 0, out of order 0
bytes dropped: PLU 0, tail 0, out of order 0
Platform AC context:
Ingress AC: AToM, State: Bound
.

Forwarding: AC to PW Cont.
RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn forwarding interface g0/1/0/3.1 hardware ingress detail
location 0/1/CPU0
.
Platform AC context:
Ingress AC: AToM, State: Bound
Flags: Remote is PW
.
Platform PW context:
Ingress PW: AToM, State: Bound
XID: 0xc0008000, bridge: 0, MAC limit: 0, l2vpn ldi index: 0x0002, vc label: 188014,
nr_ldi_hash: 0x68, r_ldi_hash: 0xb3, lag_hash: 0xf4, SHG: None
Flags: Control Word, Flow Label imposition
NP0
Xconnect ID: 0xc0008000, NP: 0
Type: Pseudowire (with control word)
Flags: Learn enable, Type 5, Local replication, Flow Label imposition
VC label hash, nR-LDI Hash: 0x68, R-LDI Hash: 0xb6, LAG Hash: 0xf4,
VC output label: 0x2de6e (188014), LDI: 0x0002, stats ptr: 0x00000000
Split Horizon Group: None
.

Forwarding: PW to AC
RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding labels 111014 hardware ingress detail location
0/1/CPU0
Mon Jun 17 23:58:30.490 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------111014 Pop
PW([Link]:1) Gi0/1/0/3.1 point2point
N/A
Updated Jun 15 01:29:06.149
Path Flags: 0x8 [ ]
PW Flow Label: Enabled
MAC/Encaps: 0/0, MTU: 0
Label Stack (Top -> Bottom): { }
Packets Switched: 0
LEAF - HAL pd context :
sub-type : MPLS_VPWS, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_mar
Leaf H/W Result:
Raw Data0: 0x51009400 01004004 00000000 00000000
Raw Data1: 0x00000000 00000000 00002013 5f000000
cb0
vpn_special: 0
vc_label_vpws: 1
vc_label_vpls: 0
match: 1
.

BGP Signaling: L3 IPv4 VPN

RP/0/RSP0/CPU0:rasr9000-2w-a#show bgp vpnv4 unicast labels
Tue Jun 18 01:25:27.965 EDT
BGP router identifier [Link], local AS number 65001
BGP generic scan interval 60 secs
BGP table state: Active
Table ID: 0x0
RD version: 348768
BGP main routing table version 40
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network
Next Hop
Rcvd Label
Local Label
Route Distinguisher: 65001:1 (default for vrf CUST-A)
*> [Link]/24
[Link]
nolabel
111015
*>i172.20.201.0/24
[Link]
188015
nolabel
* i
[Link]
188015
nolabel
*> [Link]/24
[Link]
nolabel
111016
*>i172.20.211.0/24
[Link]
188016
nolabel
* i
[Link]
188016
nolabel
Processed 4 prefixes, 6 paths

Received

Advertised

Forwarding: IPv4 to VPNv4

RP/0/RSP0/CPU0:rasr9000-2w-a#show cef vrf CUST-A ipv4 [Link]/24 hardware ingress location
0/1/CPU0
Tue Jun 18 01:45:27.771 EDT
[Link]/24, version 50, internal 0x14004001 (ptr 0x87935564) [1], 0x0 (0x0), 0x410
(0x89c84170)
Updated Jun 18 01:25:21.070
Prefix Len 24, traffic index 0, precedence routine (0), priority 3
via [Link], 5 dependencies, recursive [flags 0x6010]
path-idx 0 [0x89d4cb84 0x0]
LDP label
next hop VRF - 'default', table - 0xe0000000
next hop [Link] via 111007/0/21
next hop [Link]/32 tt1111
labels imposed {101000 188016}
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:

VPNv4 label

Physical Result: 0x11ba0200 (LE)

Raw Data0: 0x51924000 2de70100 00000000 00000000
Raw Data1: 0x0b000000 00000000 00180000 0000a2ff
leaf_resolve_control_byte0
reserved: 0
ifib_lookup: 0
.

match: 1

valid: 1

Forwarding: VPNv4 to IPv4

RP/0/RSP0/CPU0:rasr9000-2w-a#show

mpls forwarding labels 111016 detail hardware ingress location 0/1/CPU0

Tue Jun 18 02:08:02.870 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------111016 Unlabelled [Link]/24[V] Gi0/1/0/3.200 [Link]
N/A
Updated Jun 17 21:23:42.087
Version: 47, Priority: 3
MAC/Encaps: 18/18, MTU: 1500
Label Stack (Top -> Bottom): { Unlabelled }
Packets Switched: 0
LEAF - HAL pd context :
sub-type : MPLS_VPN, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_mark
Leaf H/W Result:
Raw Data0: 0x11020900 00000000 00000000 00000000
Raw Data1: 0x99000000 00000000 10000000 0000a2ff
cb0
vpn_special: 0
vc_label_vpws: 0
vc_label_vpls: 0
match: 1
rsvd: 0
valid: 1
.

Forwarding: VPNv4 Aggregate to IPv4

RP/0/RSP0/CPU0:rasr9000-2w-a#show mpls forwarding labels 111015 detail hardware ingress location
0/1/CPU0
Tue Jun 18 02:06:14.191 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Bytes
Label Label
or ID
Interface
Switched
------ ----------- ------------------ ------------ --------------- -----------111015 Aggregate
CUST-A: Per-VRF Aggr[V]
\
CUST-A
N/A
Updated Jun 17 20:03:19.525
Path Flags: 0x10 [ ]
MAC/Encaps: 0/0, MTU: 0
Label Stack (Top -> Bottom): { }
Packets Switched: 0
LEAF - HAL pd context :
sub-type : MPLS_DEAG, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_mar
Leaf H/W Result:
Raw Data0: 0x910008ff 00000000 00000000 00000000
Raw Data1: 0x00000000 00000000 10000015 5f000000
cb0
vpn_special: 1
vc_label_vpws: 0
vc_label_vpls: 0
match: 1
.

Forwarding: Load Sharing To Core Bundle

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef vrf TRAFFIC ipv4 [Link] hardware ingress location
0/0/CPU0
Tue Dec 10 10:14:29.843 EST

[Link]/24, version 41, internal 0x14004001 (ptr 0x8dd2b964) [1], 0x0 (0x0), 0x410 (0x90d929b0)
Updated Dec 5 08:06:31.568
Prefix Len 24, traffic index 0, precedence routine (0), priority 3
via [Link], 7 dependencies, recursive [flags 0x6010]
path-idx 0 [0x90e5ab08 0x0]
next hop VRF - 'default', table - 0xe0000000
next hop [Link] via 188002/0/21
next hop [Link]/32 tt180
labels imposed {ImplNull 111018}
LEAF - HAL pd context :
sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0, collapse_bwalk_required:0, ecdv2_marked:0
Leaf H/W Result:
Physical Result: 0x117a0300 (LE)
.

Other fields:
leaf_ptr: 0xc4bc05(LE)
urpf_ptr: 0
NextHopPrefix:label:eos=188002:0

bgp_next_hop: 0xa656f01

Please use show cef or show mpls forwarding command again

with nexthop prefix specified for nexthop hardware details

Forwarding: Load Sharing To Core Bundle

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef ipv4 [Link] hardware ingress detail location 0/0/CPU0
| include if_handle
Tue Dec 10 11:02:58.582 EST

if_handle

: 0x2d320

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers pm location 0/0/CPU0 | include 2d320

Tue Dec 10 11:03:07.283 EST
Ifname(2): Bundle-Ether1, ifh: 0x2d320 :
parent_bundle_ifh 0x2d320

RP/0/RSP0/CPU0:rasr9000-2w-b#show bundle Bundle-Ether 1

Tue Dec 10 11:03:14.257 EST

Bundle-Ether1
Status:
Up
Local links <active/standby/configured>:
5 / 0 / 5
Local bandwidth <effective/available>:
50000000 (50000000) kbps
MAC address (source):
10f3.110b.161b (Chassis pool)
Port
Device
State
Port ID
B/W, kbps
-------------------- --------------- ----------- -------------- ---------Te0/0/0/5
Local
Active
0x8000, 0x0001
10000000
Te0/1/1/0
Local
Active
0x8000, 0x0003
10000000
Te0/1/1/1
Local
Active
0x8000, 0x0004
10000000
Te0/1/1/2
Local
Active
0x8000, 0x0002
10000000
Te0/1/1/3
Local
Active
0x8000, 0x0005
10000000
.

Link
Link
Link
Link
Link

is
is
is
is
is

Active
Active
Active
Active
Active

Forwarding: Load Sharing To Core Bundle

RP/0/RSP0/CPU0:rasr9000-2w-b#bundle-hash Bundle-Ether 1 location 0/0/CPU0
Tue Dec 10 11:17:15.224 EST
Calculate Bundle-Hash for L2 or L3 or sub-int based: 2/3/4 [3]:
Enter traffic type (1.IPv4-inbound, [Link]-inbound, 3:IPv6-inbound): [1]:
Single SA/DA pair or range: S/R [S]:
Enter source IPv4 address [[Link]]: [Link]
Enter destination IPv4 address [[Link]]: [Link]
Compute destination address set for all members? [y/n]: n
Enter L4 protocol ID. (Enter 0 to skip L4 data) [0]:
Invalid protocol. L4 data skipped.
Link hashed [hash_val:3] to is TenGigE0/1/1/3 ICL () LON 4 ifh 0x6000680
Another? [y]: n
RP/0/RSP0/CPU0:rasr9000-2w-b#

Forwarding: Load Sharing To Multiple CEs

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef vrf TRAFFIC ipv4 [Link] hardware ingress location 0/1/CPU0
Tue Dec 10 12:13:20.504 EST

[Link]/24, version 33, internal 0x4000001 (ptr 0x877d8564) [1], 0x0 (0x0), 0x0 (0x0)
Updated Dec 5 08:06:32.256
Prefix Len 24, traffic index 0, precedence routine (0), priority 3
via [Link], 3 dependencies, recursive [flags 0x0]
path-idx 0 [0x877d8964 0x0]
next hop [Link] via [Link]/32
via [Link], 3 dependencies, recursive [flags 0x0]
path-idx 1 [0x877d8f64 0x0]
next hop [Link] via [Link]/32
via [Link], 3 dependencies, recursive [flags 0x0]
path-idx 2 [0x877d87e4 0x0]
next hop [Link] via [Link]/32
NextHopPrefix:[Link]/32
Please use show cef or show mpls forwarding command again
with nexthop prefix specified for nexthop hardware details
NextHopPrefix:[Link]/32
Please use show cef or show mpls forwarding command again
with nexthop prefix specified for nexthop hardware details
NextHopPrefix:[Link]/32
Please use show cef or show mpls forwarding command again
with nexthop prefix specified for nexthop hardware details
.

Forwarding: Load Sharing To Multiple CEs

RP/0/RSP0/CPU0:rasr9000-2w-b#show cef vrf TRAFFIC ipv4 exact-route [Link] [Link] hardware
ingress location 0/0/CPU0
Tue Dec 10 12:17:36.699 EST

[Link]/24, version 33, internal 0x4000001 (ptr 0x8dd31064) [1], 0x0 (0x0), 0x0 (0x0)
Updated Dec 5 08:06:31.504
local adjacency [Link]
Prefix Len 24, traffic index 0, precedence routine (0), priority 3
via TenGigE0/0/0/0
via [Link], 3 dependencies, recursive [flags 0x0]
path-idx 0 [0x8dd29564 0x0]
next hop [Link] via [Link]/32
RP/0/RSP0/CPU0:rasr9000-2w-b#show cef adjacency tenGigE 0/0/0/0 [Link] hardware egress detail
location 0/0/CPU0
Tue Dec 10 12:23:14.902 EST

Display protocol is ipv4

Interface
Address
Type
Refcount
Te0/0/0/0
Prefix: [Link]/32
local
5
Adjacency: PT:0x8a7742e8 [Link]/32
Interface: Te0/0/0/0
MAC: 02.c0.00.00.f3.[Link].[Link].00
Interface Type: 0x1e, Base Flags: 0x1 (0x91c7ad58)
Nhinfo PT: 0x91c7ad58, Idb PT: 0x8d18a318, If Handle: 0x40000c0
Dependent adj type: remote (0x90fd7c70)
.

BGP Signaling: VPLS Bridge

RP/0/RSP0/CPU0:rasr9000-2w-a#show bgp l2vpn vpls
Tue Jun 18 18:59:28.339 EDT
BGP router identifier [Link], local AS number 65001
BGP generic scan interval 60 secs
BGP table state: Active
Table ID: 0x0
RD version: 412008
BGP main routing table version 35
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best
i - internal, r RIB-failure, S stale
Origin codes: i - IGP, e - EGP, ? - incomplete
Network
Next Hop
Rcvd Label
Local Label
Route Distinguisher: 65001:100 (default for vrf BRIDGES:BR-A)
*> 111:110/32
[Link]
nolabel
111030
*> 111:130/32
[Link]
nolabel
111090
*> 111:180/32
[Link]
nolabel
111060
.
*>i188:110/32
[Link]
188060
nolabel
* i
[Link]
188060
nolabel
*>i188:130/32
[Link]
188090
nolabel
* i
[Link]
188090
nolabel
Received
.
Processed 13 prefixes, 23 paths

Advertised

Forwarding: VPLS Bridge: EFP to VFI

RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn forwarding bridge-domain BRIDGES:BR-A hardware ingress
detail location 0/1/CPU0
Tue Jun 18 21:18:34.152 EDT
Bridge-domain name: BRIDGES:BR-A, id: 0, state: up
MAC learning: enabled
MAC port down flush: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
MAC Secure: disabled, Logging: disabled
DHCPv4 snooping: profile not known on this node
Dynamic ARP Inspection: disabled, Logging: disabled
IP Source Guard: disabled, Logging: disabled
IGMP snooping: disabled, flooding: enabled
Bridge MTU: 1500 bytes
Number of bridge ports: 5
Number of MAC addresses: 2
Multi-spanning tree instance: 0
Platform bridge context:
.

Lots of information:
All EFPs, all PWs,
all labels!

Forwarding: VPLS Bridge: EFP to VFI

RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn forwarding bridge-domain BRIDGES:BR-A mac-address
hardware ingress detail location 0/1/CPU0
Tue Jun 18 21:10:27.472 EDT
To Resynchronize MAC table from the Network Processors, use the command...
l2vpn resynchronize forwarding mac-address-table location <r/s/i>
Mac Address
Type
Learned from/Filtered on
LC learned Resync Age
Mapped to
-------------------------------------------------------------------------------0022.9088.2ac0 dynamic Gi0/1/0/3.300
0/1/CPU0
0d 0h 0m 5s
N/A
0022.55e6.ae20 dynamic ([Link], 300)
0/1/CPU0
0d 0h 0m 4s
N/A
.

Dest MAC

PW

Forwarding: VPLS Bridge: EFP to VFI

RP/0/RSP0/CPU0:rasr9000-2w-a#show l2vpn forwarding neighbor [Link] 300 hardware egress
detail location 0/1/CPU0
Tue Jun 18 21:21:47.126 EDT
Xconnect id: 0xc0000007, Status: up
Segment 1
MPLS, Destination address: [Link], pw-id: 300, status: Bound
Pseudowire label: 188061
Control word disabled
Statistics:
packets: received 376162, sent 376005
bytes: received 51910302, sent 44368536
PW label
packets dropped: PLU 0, tail 0, out of order 0
bytes dropped: PLU 0, tail 0, out of order 0
Segment 2
Bridge id: 0, Split horizon group id: 1
Storm control: disabled
MAC learning: enabled
MAC port down flush: enabled
Flooding:
Broadcast & Multicast: enabled
Unknown unicast: enabled
MAC aging time: 300 s, Type: inactivity
MAC limit: 4000, Action: none, Notification: syslog
MAC limit reached: no
.

Forwarding: VPLS Bridge: VFI to EFP

RP/0/RSP0/CPU0:rasr9000-2w-b#show mpls forwarding
Tue Jun 18 06:37:43.199 EDT
Local Outgoing
Prefix
Outgoing
Next Hop
Label Label
or ID
Interface
------ ----------- ------------------ ------------ --------------188000 Pop
[Link]/32
tt18800
[Link]
188001 Pop
[Link]/32
tt18801
[Link]
188002 Pop
[Link]/32
tt18802
[Link]
188003 Pop
[Link]/32
tt18803
[Link]
188004 Pop
[Link]/32
tt18804
[Link]
188005 Pop
[Link]/32
tt18805
[Link]
188013 108009
[Link]/32
tt8881
[Link]
188014 Pop
PW([Link]:1) Gi0/1/0/3.1 point2point
188015 Aggregate
CUST-A: Per-VRF Aggr[V]
\
CUST-A
188016 Unlabelled [Link]/24[V] Gi0/1/0/3.200 [Link]
188061 Pop
PW([Link]:300)
\
BD=0
point2point
188075 Pop
PW([Link]:300)
\
BD=0
point2point
188097 Pop
PW([Link]:300)
\
BD=0
point2point

Bytes
Switched
-----------0
0
300
0
300
0
219343578
319362
49800
0

100172000
0
0

Pop label

Forwarding: VPLS Bridge: VFI to EFP

RP/0/RSP0/CPU0:rasr9000-2w-b#show l2vpn forwarding bridge-domain BRIDGES:BR-A mac-address
hardware egress location 0/1/CPU0
Tue Jun 18 06:44:04.464 EDT
To Resynchronize MAC table from the Network Processors, use the command...
l2vpn resynchronize forwarding mac-address-table location <r/s/i>
Mac Address
Type
Learned from/Filtered on
LC learned Resync Age
Mapped to
-------------------------------------------------------------------------------0022.55e6.ae20 dynamic Gi0/1/0/3.300
0/1/CPU0
0d 0h 0m 4s
N/A
0022.9088.2ac0 dynamic ([Link], 300)
0/1/CPU0
0d 0h 0m 5s
N/A
.

Dest MAC

Agenda

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

Diagnostics &
Troubleshooting: System

Background Diagnostics
RSP default diagnostics
RP/0/RSP0/CPU0:rasr9000-2w-a#admin show diagnostic content location 0/RSP0/CPU0
Wed Dec 11 19:44:32.957 EST

RP 0/RSP0/CPU0:
Diagnostics test suite attributes:
M/C/* - Minimal bootup level test / Complete bootup level test / NA
B/O/* - Basic ondemand test / not Ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
Every minute or
X/* - Not a health monitoring test / NA
every 5 seconds
F/* - Fixed monitoring interval test / NA
E/* - Always enabled monitoring test / NA
A/I - Monitoring is active / Monitoring is inactive

ID
====
1)
2)
3)
4)
5)
6)
7)
8)

Test Name
==================================
CPUCtrlScratchRegister ---------->
ClkCtrlScratchRegister ---------->
ZenJfScratchRegister ------------>
FabSwitchIdRegister ------------->
SrspStandbyEobcHeartbeat -------->
SrspActiveEobcHeartbeat --------->
FabricLoopback ------------------>
PuntFabricDataPath -------------->

Test Interval
ThreAttributes
(day hh:mm:[Link] shold)
============ ================= =====
***N****A
000 00:01:00.000 1
***N****A
000 00:01:00.000 1
***N****A
000 00:01:00.000 1
*B*N****A
000 00:01:00.000 1
*B*NS***A
000 00:00:05.000 3
*B*NS***A
000 00:00:05.000 3
MB*N****A
000 00:01:00.000 3
*B*N****A
000 00:01:00.000 3

Error threshold
(consecutive)

Background Diagnostics
Test example: PuntFabricDataPath

Looping the path between RP CPU and each NP

Background Diagnostics
LC default diagnostics
RP/0/RSP0/CPU0:rasr9000-2w-a#admin show diagnostic content location 0/0/CPU0
Wed Dec 11 20:32:08.842 EST

A9K-24x10GE-SE 0/0/CPU0:
Diagnostics test suite attributes:
M/C/* - Minimal bootup level test / Complete bootup level test / NA
B/O/* - Basic ondemand test / not Ondemand test / NA
P/V/* - Per port test / Per device test / NA
D/N/* - Disruptive test / Non-disruptive test / NA
S/* - Only applicable to standby unit / NA
X/* - Not a health monitoring test / NA
F/* - Fixed monitoring interval test / NA
E/* - Always enabled monitoring test / NA
A/I - Monitoring is active / Monitoring is inactive
ID
====
1)
2)
3)
4)
5)
6)

Test Name
==================================
CPUCtrlScratchRegister ---------->
PHYCtrlScratchRegister ---------->
PortCtrlScratchRegister --------->
FIAScratchRegister -------------->
LcEobcHeartbeat ----------------->
NPULoopback --------------------->

Test Interval
ThreAttributes
(day hh:mm:[Link] shold)
============ ================= =====
*B*N****A
000 00:01:00.000 1
*B*N****A
000 00:01:00.000 1
*B*N****A
000 00:01:00.000 1
*B*N****A
000 00:01:00.000 1
*B*N****A
000 00:00:05.000 3
*B*N****A
000 00:01:00.000 3

Background Diagnostics
Reading the results
RP/0/RSP0/CPU0:rasr9000-2w-a#admin show diagnostic result location
0/RSP1/CPU0 detail
Wed Dec 11 20:35:26.998 EST

) SrspActiveEobcHeartbeat ---------> .
Error code ------------------>
Total run count ------------->
Last test execution time ---->
First test failure time ----->
Last test failure time ------>
Last test pass time --------->
Total failure count --------->
Consecutive failure count --->

Current bootup diagnostic level for RP 0/RSP1/CPU0: minimal

RP 0/RSP1/CPU0:

Overall diagnostic result: PASS

Diagnostic level at card bootup: minimal
Test results: (. = Pass, F = Fail, U = Untested)
_____________________________________________________________________
1

) CPUCtrlScratchRegister ----------> .

Error code ------------------>

Total run count ------------->
Last test execution time ---->
First test failure time ----->
Last test failure time ------>
Last test pass time --------->
Total failure count --------->
Consecutive failure count --->

) ClkCtrlScratchRegister ----------> .
Error code ------------------>
Total run count ------------->
Last test execution time ---->
First test failure time ----->
Last test failure time ------>
Last test pass time --------->
Total failure count --------->
Consecutive failure count --->

________________________________________________________________________
7

0 (DIAG_SUCCESS)
31553
Wed Dec 11 20:35:08 2013
n/a
n/a
Wed Dec 11 20:35:08 2013
0
0

) FabricLoopback ------------------> .
Error code ------------------>
Total run count ------------->
Last test execution time ---->
First test failure time ----->
Last test failure time ------>
Last test pass time --------->
Total failure count --------->
Consecutive failure count --->

0 (DIAG_SUCCESS)
31553
Wed Dec 11 20:35:08 2013
n/a
n/a
Wed Dec 11 20:35:08 2013
0
0

_____________________________________________________________________

0 (DIAG_SUCCESS)
378621
Wed Dec 11 20:35:25 2013
n/a
n/a
Wed Dec 11 20:35:25 2013
0
0

0 (DIAG_SUCCESS)
31552
Wed Dec 11 20:35:08 2013
n/a
n/a
Wed Dec 11 20:35:08 2013
0
0

________________________________________________________________________
8

) PuntFabricDataPath --------------> .
Error code ------------------>
Total run count ------------->
Last test execution time ---->
First test failure time ----->
Last test failure time ------>
Last test pass time --------->
Total failure count --------->
Consecutive failure count --->

0 (DIAG_SUCCESS)
31552
Wed Dec 11 20:35:08 2013
n/a
n/a
Wed Dec 11 20:35:08 2013
0
0

________________________________________________________________________

Background Diagnostics
Errors and clears
Set: threshold
exceeded
Path to LC 2 NP 0
failed in this example

RP/0/RSP0/CPU0:Feb 5 05:05:44.051 :
pfm_node_rp[354]:%PLATFORMDIAGS3PUNT_FABRIC_DATA_PATH_FAILED :
Set|online_diag_rsp[237686]|System Punt/Fabric/data Path
Test(0x2000004)|failure threshold is 3, (slot, NP)failed:
(0/2/CPU0, 0)

Clear: test previously

failing, now passed
Indication of
transient fault. Keep
watching

show pfm location

all shows platform
errors reported

RP/0/RSP0/CPU0:Feb 5 05:05:46.051 :
pfm_node_rp[354]:%PLATFORMDIAGS3PUNT_FABRIC_DATA_PATH_FAILED :
Clear|online_diag_rsp[237686]|System Punt/Fabric/data Path
Test(0x2000004)|failure threshold is 3, (slot, NP)failed:
(0/2/CPU0, 0)

Troubleshooting:
Forwarding Path

Monitor Interface
See interface stats in almost real time
RP/0/RSP0/CPU0:rasr9000-2w-b#monitor interface tenGigE 0/1/1/1
rasr9000-2w-b

Monitor Time: 00:00:22

SysUptime: 501:59:18

TenGigE0/1/1/1 is up, line protocol is up

Encapsulation ARPA
Traffic Stats:(2 second rates)
Input Packets:
2495245669613
Input pps:
7441113
Input Bytes:
164703177204108
Input Kbps (rate):
3928857
Output Packets:
3017277633655
Output pps:
6626897
Output Bytes:
205177835436607
Output Kbps (rate):
3605031
Errors Stats:
Input Total:
Input CRC:
Input Frame:
Input Overrun:
Output Total:
Output Underrun:

1
0
0
0
0
0

Quit='q', Freeze='f', Thaw='t', Clear='c', Interface='i',

Next='n', Prev='p'
Brief='b', Detail='d', Protocol(IPv4/IPv6)='r'

Delta
14890408
982758522
( 39%)
13261227
901762428
( 36%)
0
0
0
0
0
0

The Physical
Checking on port physical: SFP/XFP, levels

NP

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers TenGigE 0/0/0/0 phy

Mon Dec

9 13:53:37.848 EST

SFP EEPROM port: 0

Xcvr Type: SFP
Xcvr Code: SFP-10G-SR
Encoding: 64B66B
Bit Rate: 10300 Mbps
Link Reach 50u fiber: 80 meter
Link Reach 62.5u fiber: 20 meter
Vendor Name: CISCO-FINISAR
Vendor OUI: 00.90.65
Vendor Part Number: FTLX8571D3BCL-C2 (rev.: A
Laser wavelength: 850 nm (fraction: 0.00 nm)
Optional SFP Signal: Rate Sel, LOS
Vendor Serial Number: FNS164018G7
Date Code (yy/mm/dd): 12/10/06 lot code:

Thresholds:
Alarm High
Warning High
Temperature:
+75.000 C
+70.000 C
Voltage:
3.630 Volt
3.465 Volt
Bias:
11.800 mAmps
10.800 mAmps
Transmit Power: 1.479 mW (1.70 dBm)
0.741 mW (-1.30 dBm)
Receive Power: 1.585 mW (2.00 dBm)
0.794 mW (-1.00 dBm)
Temperature: 26.684
Voltage: 3.301 Volt
Tx Bias: 7.612 mAmps
Tx Power: 0.613 mW (-2.13 dBm)
Rx Power: 0.567 mW (-2.46 dBm)
Oper. Status/Control:

Warning Low
+0.000 C
3.135 Volt
5.000 mAmps
0.186 mW (-7.30 dBm)
0.102 mW (-9.90 dBm)

Alarm Low
-5.000 C
2.970 Volt
4.000 mAmps
0.074 mW (-11.30 dBm)
0.041 mW (-13.90 dBm)

The Physical
Reading the controller counters: In, out, invalid, unicast, mcast, frame sizes
RP/0/RSP0/CPU0:rasr9k-1y#show controllers TenGigE0/4/0/20
stats
Sun Feb 24 14:44:18.899 UTC
Statistics for interface TenGigE0/4/0/20 (cached values):
Ingress:
Input total bytes
Input good bytes

= 3081227904920
= 3081227904920

Input total packets

= 23220024479
Input 802.1Q frames
=0
Input pause frames
=0
Input pkts 64 bytes
= 7143534733
Input pkts 65-127 bytes = 2888766549
Input pkts 128-255 bytes = 13124923916
Input pkts 256-511 bytes = 62799261
Input pkts 512-1023 bytes = 0
Input pkts 1024-1518 bytes = 0
Input pkts 1519-Max bytes = 0
Input good pkts
Input unicast pkts
Input multicast pkts
Input broadcast pkts
Input drop overrun
.

= 23220024479
= 23220023458
= 62
= 959
=0

Egress:
Output total bytes
Output good bytes

= 1345771624
= 1345771624

Output total packets

= 21895707
Output 802.1Q frames
=0
Output pause frames
=0
Output pkts 64 bytes
= 21665536
Output pkts 65-127 bytes = 21179
Output pkts 128-255 bytes = 168767
Output pkts 256-511 bytes = 40225
Output pkts 512-1023 bytes = 0
Output pkts 1024-1518 bytes = 0
Output pkts 1519-Max bytes = 0
Output good pkts
Output unicast pkts
Output multicast pkts
Output broadcast pkts
Output drop underrun
Output drop abort
Output drop other
Output error other

= 21895707
= 21870499
= 25195
= 13
=0
=0
=0
=0

Interface Programming in Hardware

Example L3 VLAN sub-interface
RP/0/RSP0/CPU0:rasr9000-2w-b#show ethernet hardware
interface TenGigE 0/0/0/2.200 location 0/0/CPU0
Tue Jan 21 21:45:18.351 EST
-----------------------------------------------------------------------------

Physical port
Interface name
Ifhandle
Parent Ifhandle
Status
TCAM entries
TCAM entry type
Channel ID
PI policy validity
NP port
NP map (previous)
idb pointer
Admin mode
Interface state
Interface type
Physical)
tunn_ovrd_mode
tunneling ethertype set
Ingress UIDB index
Egress UIDB index

:
:
:
:
:
:
:
:
:
:
:
:
:
:
:

2
TenGigE0/0/0/2.200
0x040012C0
0x04000140
Init|Mem|Alloc|TShm|Pgm
1
Single tag exact
0
0x0
4
0x0 (0x0)
0x5002b570
1 (Up)
1 (Up)
3 (L3 Sub-if over

TCAM 0 address: 0x23880

TCAM 1 address: 0x0
TCAM 2 address: 0x0
.

........ TCAM entry 0 (uncompressed logical)

................................
mask=0xFFFF

value=0x0004(4)

Validity Bits:
validity1:
validity2:
validity3:
isid_valid:

mask=1
mask=1
mask=0
mask=0

value=1
value=0
value=0
value=0

Tag 1
:
ethertype:
VLAN id :

mask=0xFFFF
mask=0x0FFF

value=0x8100
value=0x00C8(200)

Tag 2
:
ethertype:
VLAN id :

mask=0x0000
mask=0x0000

value=0x0000
value=0x0000(0)

: QnQ Child /w no parent

Source MAC
:
mask :
value :

0000.0000.0000
0000.0000.0000

: 29
: 29

........ TCAM entry 0 (2nd gen physical)

....................................

-----------------------------------------------------------------------------

TCAM key status: 0x404

index: 0

Port Number

TCAM mask:
FC FF 00 00 FF FF
FF 00 00 BB BB BB
TCAM value:
01 00 00 81 00 00
00 04 00 00 00 00

FF

FF

FF

FF

FF

FF

00

F0

00

00

00

00

00

00

C8

00

Unicast Transit Frame Path

Physical > NP > FIA > Fabric > FIA > NP > Physical
3x 10G
3x10GE
SFP +

Typhoon

FIA

3x 10G
3x10GE
SFP +

Typhoon
FIA
3x 10G

Typhoon

Typhoon
3x 10G

3x10GE
SFP +

Typhoon

FIA

3x 10G
3x10GE
SFP +

Typhoon

Switch
Fabric

3x 10G
3x10GE
SFP +

FIA

Egress
Typhoon

100G

FIA

Ingress
Typhoon

100G

100GE
MAC/PHY

Typhoon

FIA

3x 10G
3x10GE
SFP +

100G

100GE
MAC/PHY

Switch Fabric ASIC

FIA

3x 10G
3x10GE
SFP +

Switch Fabric ASIC

3x10GE
SFP +

Ingress
Typhoon

Typhoon

Switch
Fabric

FIA

Egress
Typhoon

100G

The Internal Path

Mapping the port to NP and FIA

NP

! Example: Path from GigabitEthernet0/0/1/0 [Link] TO TenGigE0/4/0/20.6 [Link]

RP/0/RSP0/CPU0:rasr9k-1y#show controllers NP ports all location 0/0/CPU0
Fri Feb 22 15:57:32.307 UTC
Node: 0/0/CPU0:
---------------------------------------------------------------Map the port to NP
NP Bridge Fia
Ports
and FIA
-- ------ --- --------------------------------------------------0 -0
TenGigE0/0/0/0, TenGigE0/0/0/1, TenGigE0/0/0/2, TenGigE0/0/0/3
1 -1
GigabitEthernet0/0/1/0 - GigabitEthernet0/0/1/19
RP/0/RSP0/CPU0:rasr9k-1y#show controllers NP ports all location 0/4/CPU0
Fri Feb 22 15:55:22.370 UTC
Node: 0/4/CPU0:
---------------------------------------------------------------NP Bridge Fia
Ports
-- ------ --- --------------------------------------------------0 -0
TenGigE0/4/0/0, TenGigE0/4/0/1, TenGigE0/4/0/2
1 -0
TenGigE0/4/0/3, TenGigE0/4/0/4, TenGigE0/4/0/5
2 -1
TenGigE0/4/0/6, TenGigE0/4/0/7, TenGigE0/4/0/8
3 -1
TenGigE0/4/0/9, TenGigE0/4/0/10, TenGigE0/4/0/11
4 -2
TenGigE0/4/0/12, TenGigE0/4/0/13, TenGigE0/4/0/14
5 -2
TenGigE0/4/0/15, TenGigE0/4/0/16, TenGigE0/4/0/17
6 -3
TenGigE0/4/0/18, TenGigE0/4/0/19, TenGigE0/4/0/20
7 -3
TenGigE0/4/0/21, TenGigE0/4/0/22, TenGigE0/4/0/23

Inside NP
Processing pipelines

Parse Search

Resolve Modify

TM
Queueing
Scheduling

Inside Typhoon NP
Reading pipeline counters

Parse

Search

Resolve

Modify

TM
Queueing
Scheduling

RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers NP counters np0 location 0/1/CPU0

Wed Nov 27 21:09:07.635 EST

Node: 0/1/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v2

List of NP counters:
[Link]

Read 64 non-zero NP counters:

Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------16 MDF_TX_LC_CPU
6722114
10
17 MDF_TX_WIRE
1826039
3
21 MDF_TX_FABRIC
1635541
2
29 PARSE_FAB_RECEIVE_CNT
1837406
3
33 PARSE_INTR_RECEIVE_CNT
5083364
7
37 PARSE_INJ_RECEIVE_CNT
1228130
2
.

499
502
541
584
604

RSV_ING_L2_SMAC_MISS
RSV_ING_L2_LEARN
RSV_REFRESH_FROM_NOTIFY_CNT
RSV_L2BC_BVI
RESOLVE_REMOTE_RACK_PREP_CNT

708

LRN_PERIODIC_AGING_DELETE_ENTRY

774

ARP

848
852
900
902
904

PUNT_ADJ
PUNT_ACL_DENY
PUNT_STATISTICS
PUNT_DIAGS_RSP_ACT
PUNT_DIAGS_RSP_STBY

60
60
62
2
5539915

0
0
0
0
8

60

119

2
161
5083356
11419
11427

0
0
7
0
0

Typhoon Counters and Rates

Example: Ingress NP, no drops

Parse

Search

Resolve

Modify

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0

Mon Dec 9 15:16:34.889 EST
Node: 0/0/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v2
Read 59 non-zero NP counters:
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------16 MDF_TX_LC_CPU
10255120
8
17 MDF_TX_WIRE
6382883323428
1
21 MDF_TX_FABRIC
8903307706961
31250074
29 PARSE_FAB_RECEIVE_CNT
6382883151049
0
33 PARSE_INTR_RECEIVE_CNT
8653828
8
37 PARSE_INJ_RECEIVE_CNT
744943
1
41 PARSE_ENET_RECEIVE_CNT
8910925981070
31250074
45 PARSE_TM_LOOP_RECEIVE_CNT
8035316
5
49 PARSE_TOP_LOOP_RECEIVE_CNT
61
0
57 PARSE_ING_DISCARD
2344591
0
195 PRS_HEALTH_MON
8035316
5
204 INTR_FRAME_TYPE_7
8653827
8
214 DBG_PRS_EP_L_PRS_VPLS_PW_IMPOSE
10
0
233 PARSE_RSP_INJ_FAB_CNT
70634
0
235 PARSE_RSP_INJ_DIAGS_CNT
55255
0
236 PARSE_EGR_INJ_PKT_TYP_UNKNOWN
66847
0
237 PARSE_EGR_INJ_PKT_TYP_IPV4
3787
0
246 PARSE_LC_INJ_FAB_CNT
101092
0
.

To FIA

From Phy

TM
Queueing
Scheduling

Typhoon Counters and Rates

NP drops, rate and direction

Parse

Search

Resolve

Modify

TM
Queueing
Scheduling

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0

Tue Dec 10 14:18:39.195 EST

Node: 0/0/CPU0:
---------------------------------------------------------------Show global stats counters for NP0, revision v2
Read 59 non-zero NP counters:
Offset Counter
FrameValue
Rate (pps)
------------------------------------------------------------------------------16 MDF_TX_LC_CPU
11004363
9
17 MDF_TX_WIRE
8712222364719
29761820
21 MDF_TX_FABRIC
11063035007386
27714366
29 PARSE_FAB_RECEIVE_CNT
8712222113330
29761820
33 PARSE_INTR_RECEIVE_CNT
9401470
9
37 PARSE_INJ_RECEIVE_CNT
832185
1
41 PARSE_ENET_RECEIVE_CNT
11070653296959
27714366
45 PARSE_TM_LOOP_RECEIVE_CNT
8437075
5

To egress
To fabric
From fabric

359
367
368
369
370
373

PARSE_MAC_NOTIFY_RCVD
PARSE_FAST_DISCARD_LOW_PRIORITY_DROP_0
PARSE_FAST_DISCARD_LOW_PRIORITY_DROP_1
PARSE_FAST_DISCARD_LOW_PRIORITY_DROP_2
PARSE_FAST_DISCARD_LOW_PRIORITY_DROP_3
DBG_RSV_EP_L_RSV_ING_L3_IFIB

183
106211394050
106210662138
106211061617
106211474043
3707021673

0
883832
883856
883943
883922
0

830
831

PUNT_NO_MATCH
PUNT_NO_MATCH_EXCD

4746
464963896

0
0

849
852
853

PUNT_ADJ_EXCD
PUNT_ACL_DENY
PUNT_ACL_DENY_EXCD

273406
1479378
1163570900

0
0
0

From interface
NP catching up

Typhoon Counters and Rates

TM drops

Parse

Search

Resolve

Modify

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP tm counters np1 location 0/0/CPU0

Tue Dec 10 14:40:47.210 EST
Node: 0/0/CPU0:
---------------------------------------------------------------==== TM Counters (NP 1 TM 0) ====
TM Counters:
xmt paks: 897837659243, xmt bytes: 62718673698431
drop paks: 29447137293, drop_bytes: 2002405351616
RP/0/RSP0/CPU0:rasr9000-2w-b#
RP/0/RSP0/CPU0:rasr9000-2w-b#
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP tm counters np1 location 0/0/CPU0
Tue Dec 10 14:40:49.816 EST
Node: 0/0/CPU0:
---------------------------------------------------------------==== TM Counters (NP 1 TM 0) ====
TM Counters:
xmt paks: 897909308598, xmt bytes: 62723686013270
drop paks: 29466027670, drop_bytes: 2003689898884

TM
Queueing
Scheduling

FIA Counters
FIA counts, drops and direction
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers fabric
fia instance 0 stats location 0/0/CPU0
Tue Dec 10 14:49:58.704 EST
********** FIA-0 **********
Category: count-0
From Unicast Xbar[0]
From Unicast Xbar[1]
From Unicast Xbar[2]
From Unicast Xbar[3]
From MultiCast Xbar[0]
From MultiCast Xbar[1]
From MultiCast Xbar[2]
From MultiCast Xbar[3]
To Unicast Xbar[0]
To Unicast Xbar[1]
To Unicast Xbar[2]
To Unicast Xbar[3]
To MultiCast Xbar[0]
To MultiCast Xbar[1]
To MultiCast Xbar[2]
To MultiCast Xbar[3]
To Line Interface[0]
To Line Interface[1]
From Line Interface[0]
From Line Interface[1]
Ingress drop:
Egress drop:
Total drop:

733461306331
733460650405
0
0
233068
0
0
0
933450146675
932066610046
0
0
451799
0
0
0
8759312354291
457138023968
11117127781061
489302108080
97191712670
0
97191712670

3x10GE
SFP +
3x10GE
SFP +

3x 10G

NP

FIA

3x 10G

NP

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers fabric fia

instance 0 drops ingress location 0/0/CPU0
Tue Dec 10 15:33:37.655 EST

********** FIA-0 **********

Category: in_drop-0
From Spaui Drop-0
accpt tbl-0
ctl len-0
short pkt-0
max pkt len-0
min pkt len-0
From Spaui Drop-1
accpt tbl-1
Back pressure
ctl len-1
from egress NP
short pkt-1
max pkt len-1
min pkt len-1
Tail drp
Vqi drp
Header parsing drp
pw to ni drp
ni from pw drp
sp0 crc err
sp0 bad align
sp0 bad code
sp0 align fail
sp0 prot err
sp1 crc err
sp1 bad align
.

0
0
0
0
0
0
0
0
0
0
0
0
125787328841
0
0
0
0
0
0
0
3
0
0
0

Troubleshooting: Packet
Capture

Packet Capture: Problem Packets

Example: incrementing drops

Parse

Search

Resolve

Modify

TM
Queueing
Scheduling

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0 | include DROP

Sat Jan 18 18:46:52.618 EST
.
370 RSV_DROP_XID_NO_MATCH
209680463
0
404 RSV_ING_VPWS_ERR_DROP
3719838164404
11160601
411 RSV_L2_SHG_DROP
27390624
0
1171 MDF_PUNT_POLICE_DROP
7924962278163
23809032
1178 MODIFY_PUNT_REASON_MISS_DROP
1
0
1246 VIRTUAL_IF_GENERIC_INPUT_DROP
1
0

RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0 | include DROP

Sat Jan 18 18:46:56.297 EST
incrementing
.
370 RSV_DROP_XID_NO_MATCH
209680463
0
404 RSV_ING_VPWS_ERR_DROP
3719879236984
11161027
411 RSV_L2_SHG_DROP
27390624
0
1171 MDF_PUNT_POLICE_DROP
7925049898728
23809936
1178 MODIFY_PUNT_REASON_MISS_DROP
1
0
Rate [PPS] or
1246 VIRTUAL_IF_GENERIC_INPUT_DROP
1
0
increments from
last command run

Packet Capture: Problem Packets

Example: incrementing drops

Parse

Search

Resolve

Modify

TM
Queueing
Scheduling

RP/0/RSP0/CPU0:rasr9000-2w-b#monitor np counter RSV_ING_VPWS_ERR_DROP np0 count 3 location

0/0/CPU0
Sat Jan 18 19:02:36.386 EST

Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~50ms traffic outage. Links will stay Up.
Proceed y/n [y] >

Alert!
Captured
are
dropped

Alert!
Traffic
loss

Packet Capture: Problem Packets

Example: incrementing drops

Parse

Search

Resolve

Modify

TM
Queueing
Scheduling

RP/0/RSP0/CPU0:rasr9000-2w-b#monitor np counter RSV_ING_VPWS_ERR_DROP np0 count 3 location

0/0/CPU0
Sat Jan 18 19:02:36.386 EST

Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~50ms traffic outage. Links will stay Up.
Ignore
Proceed y/n [y] >
[internal]
Monitor RSV_ING_VPWS_ERR_DROP on NP0 ... (Ctrl-C to quit)

Alert!
Captured
are
dropped

Sat Jan 18 19:02:44 2014 -- NP0 packet

From
0000:
0010:
0020:
0030:
0040:
0050:
0060:
0070:
0080:
0090:

TenGigE0/0/0/0: 157 byte packet,

00 00 02 01 61 90 00 00 c0 02 01
08 00 45 00 00 8b 00 00 00 00 40
01 01 c0 01 01 02 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
10 f3 11 05 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00

bytes[0-3] invalid!
02 81 00 00 0a
....a...@.......
3d f8 30 c0 01
..E.......@=x0@.
00 00 00 00 00
..@.............
00 00 00 00 00
................
00 00 00 00 00
.s..............
UP to 300 B
00 00 00 00 00
................
No CRC
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00
.............

Alert!
Traffic
loss

Packet Capture: Problem Packets

Example: incrementing drops
From
0000:
0010:
0020:
0030:
0040:
0050:
0060:
0070:
0080:
0090:
00a0:
00b0:
00c0:
00d0:
00e0:

TenGigE0/0/0/0: 234 byte packet,

00 00 02 01 61 90 00 00 c0 02 01
08 00 45 00 00 d8 00 00 00 00 40
01 01 c0 01 01 02 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
10 f3 11 05 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00

Parse

Search

bytes[0-3] invalid!
02 81 00 00 0a
....a...@.......
3d f7 e3 c0 01
..E..X....@=wc@.
00 00 00 00 00
..@.............
00 00 00 00 00
................
00 00 00 00 00
.s..............
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
..........

(count 3 of 3)
Cleanup: Confirm NP reset now (~50ms traffic outage).
Ready? [y] >
RP/0/RSP0/CPU0:rasr9000-2w-b#

Resolve

Modify

TM
Queueing
Scheduling

Alert!
Captured
are
dropped

Alert!
Traffic
loss

Packet Capture: Problem Packets

Decoding
00
08
01
00
10
00
00
00
00
00

00
00
01
00
f3
00
00
00
00
00

02
45
c0
00
11
00
00
00
00
00

01
00
01
00
05
00
00
00
00
00

61
00
01
00
00
00
00
00
00
00

Parse
90
8b
02
00
00
00
00
00
00
00

00
00
00
00
00
00
00
00
00
00

00
00
00
00
00
00
00
00
00
00

c0
00
00
00
00
00
00
00
00
00

02
00
00
00
00
00
00
00
00
00

01
40
00
00
00
00
00
00
00
00

02
3d
00
00
00
00
00
00
00
00

81
f8
00
00
00
00
00
00
00
00

00
30
00
00
00
00
00
00
00

00
c0
00
00
00
00
00
00
00

0a
01
00
00
00
00
00
00
00

Search

Resolve

Modify

TM
Queueing
Scheduling

Use a
decoder

Packet Capture: Transit Packets

Example: IPv4 L3VPN ingress
RP/0/RSP0/CPU0:rasr9000-2w-b#show running-config ipv4 access-list CAPTURE
Sat Jan 18 20:13:35.941 EST

ipv4 access-list CAPTURE

10 permit ipv4 [Link]/24 [Link]/24 capture
20 permit ipv4 any any
!
RP/0/RSP0/CPU0:rasr9000-2w-b#show running-config interface TenGigE 0/0/0/2

Count in NP
Let all else go!

Sat Jan 18 20:13:50.654 EST

interface TenGigE0/0/0/2
vrf TRAFFIC
Apply to transit
ipv4 address [Link] [Link]
ipv4 access-group CAPTURE ingress
!
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0 | include
ACL_CAPTURE_NO_SPAN
Sat Jan 18 20:14:26.109 EST

477 ACL_CAPTURE_NO_SPAN
6802507
38003
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0 | include
ACL_CAPTURE_NO_SPAN

NP ACL capture

Sat Jan 18 20:14:28.819 EST

477 ACL_CAPTURE_NO_SPAN
6905417
38002
counter
RP/0/RSP0/CPU0:rasr9000-2w-b#show controllers NP counters np0 location 0/0/CPU0 | include
incrementing
ACL_CAPTURE_NO_SPAN
Sat Jan 18 20:14:34.597 EST

477

ACL_CAPTURE_NO_SPAN

7124969

37991

Packet Capture: Transit Packets

Example: IPv4 L3VPN ingress
RP/0/RSP0/CPU0:rasr9000-2w-b#monitor np counter ACL_CAPTURE_NO_SPAN np0 count 3 location 0/0/CPU0
Sat Jan 18 20:31:53.311 EST

Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~50ms traffic outage. Links will stay Up.
Proceed y/n [y] >

Alert!
Captured
are
dropped

Alert!
Traffic
loss

Packet Capture: Transit Packets

Example: IPv4 L3VPN ingress
RP/0/RSP0/CPU0:rasr9000-2w-b#monitor np counter ACL_CAPTURE_NO_SPAN np0 count 3 location 0/0/CPU0
Sat Jan 18 20:31:53.311 EST

Warning: Every packet captured will be dropped! If you use the 'count'
option to capture multiple protocol packets, this could disrupt
protocol sessions (eg, OSPF session flap). So if capturing protocol
packets, capture only 1 at a time.
Warning: A mandatory NP reset will be done after monitor to clean up.
This will cause ~50ms traffic outage. Links will stay Up.
Proceed y/n [y] >
Monitor ACL_CAPTURE_NO_SPAN on NP0 ... (Ctrl-C to quit)
Those 3 packets
Sat Jan 18 20:32:34 2014 -- NP0 packet

From
0000:
0010:
0020:
0030:
0040:
0050:
0060:
0070:
0080:
0090:
.

TenGigE0/0/0/2: 250 byte packet,

00 11 0b 00 61 92 00 00 c0 04 01
00 ec 00 00 00 00 40 3d a8 08 c0
06 5d 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
10 f3 11 05 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00

Alert!
Captured
are
dropped

are dropped!

bytes[0-3] invalid!
02 08 00 45 60
....a...@.....E`
04 01 02 0a 0a
.l....@=(.@.....
00 00 00 00 00
.]..............
00 00 00 00 00
................
00 00 00 00 00
.s..............
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................

Alert!
Traffic
loss

Packet Capture: Transit Packets

Example: IPv4 L3VPN ingress
.
(count 2 of 3)
Sat Jan 18 20:32:36 2014 -- NP0 packet

From
0000:
0010:
0020:
0030:
0040:
0050:
0060:
0070:
0080:
0090:
00a0:
00b0:
00c0:
00d0:

TenGigE0/0/0/2: 220 byte packet,

00 11 0b 00 61 92 00 00 c0 04 01
00 ce 00 00 00 00 40 3d a8 bc c0
06 27 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
10 f3 11 05 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00
(count 3 of 3)

bytes[0-3] invalid!
02 08 00 45 00
....a...@.....E.
04 01 02 0a 0a
.N....@=(<@.....
00 00 00 00 00
.'..............
00 00 00 00 00
................
00 00 00 00 00
.s..............
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00 00 00 00 00
................
00
............

Cleanup: Confirm NP reset now (~50ms traffic outage).

Ready? [y] >
RP/0/RSP0/CPU0:rasr9000-2w-b#

Those 3 packets
were dropped!

Alert!
Captured
are
dropped

Alert!
Traffic
loss

Packet Capture: Transit Packets

Decoding the packet
00
00
06
00
10
00
00
00
00
00
00
00
00
00
00
00

11
ec
5d
00
f3
00
00
00
00
00
00
00
00
00
00
00

0b
00
00
00
11
00
00
00
00
00
00
00
00
00
00
00

00
00
00
00
05
00
00
00
00
00
00
00
00
00
00
00

61
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00

92
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00

00
40
00
00
00
00
00
00
00
00
00
00
00
00
00
00

00
3d
00
00
00
00
00
00
00
00
00
00
00
00
00
00

c0
a8
00
00
00
00
00
00
00
00
00
00
00
00
00
00

04
08
00
00
00
00
00
00
00
00
00
00
00
00
00
00

01
c0
00
00
00
00
00
00
00
00
00
00
00
00
00

02
04
00
00
00
00
00
00
00
00
00
00
00
00
00

08
01
00
00
00
00
00
00
00
00
00
00
00
00
00

00
02
00
00
00
00
00
00
00
00
00
00
00
00
00

45
0a
00
00
00
00
00
00
00
00
00
00
00
00
00

60
0a
00
00
00
00
00
00
00
00
00
00
00
00
00

Use a
decoder

We have covered

System architecture overview

Control and exception traffic

Transit frame forwarding

MPLS forwarding

Troubleshooting

Call to action

Read the material (more details and cases)

Practice and verify

Any potential issues in your deployments? Diagnose

Ask if and when you need more info

Participate in the My Favorite Speaker Contest

Promote Your Favorite Speaker and You Could Be a Winner

Promote your favorite speaker through Twitter and you could win $200 of Cisco
Press products (@CiscoPress)

Send a tweet and include

Your favorite speakers Twitter handle @MikeMikhail
Two hashtags: #CLUS #MyFavoriteSpeaker

You can submit an entry for more than one of your favorite speakers

Dont forget to follow @CiscoLive and @CiscoPress

View the official rules at [Link]

Complete Your Online Session Evaluation

Give us your feedback to be

entered into a Daily Survey
Drawing. A daily winner
will receive a $750 Amazon
gift card.

Complete your session surveys

though the Cisco Live mobile
app or your computer on
Cisco Live Connect.
Dont forget: Cisco Live sessions will be available
for viewing on-demand after the event at
[Link]/Online

Continue Your Education

Demos in the Cisco campus

Walk-in Self-Paced Labs

Table Topics

Meet the Engineer 1:1 meetings

Related sessions

Thank you

Glossary
1R2C
2R3C
802.1Q
AAA
AAL5
AC
ACE
ACL
ADJ
ARP
ASIC
ATM
B
bc
Bcast
BD
be
BFD
BGP
BPDU
CAM
CBWFQ
CDP
CEF
cir

One rate tow color

Two rate 3 color
An IEEE [Institute of Electrical and Electronics Engineers] standard
Authentication, Authorization, and Accounting
ATM Adaptation Layer 5
Attachment Circuit
Access Control Entry
Access Control List
Adjacency
Address Resolution Protocol
Application-Specific Integrated Circuit
Asynchronous Transfer Mode
Byte
Burst committed
Broadcast
Bridge Domain
Burst excess
Bidirectional Forwarding Detection
Border Gateway Protocol
Bridge Protocol Data Unit
Content Addressable Memory
Class-Based Weighted Fair Queuing
Cisco Discovery Protocol
Cisco Express Forwarding
Committed information rate

CLNS
CoS
CoPP
CPU
CRC
CSC
DBUS
dCEF
DB
DDR
DFC
DoS
DRAM
DSCP
DTP
DWDM
EFP
EIGRP
ELAM
EOBC
EoMPLS
eq
ES+
FIB
FPD

Connectionless Network Service

Class of Service
Control Plane Policing
Central Processing Unit
Cyclic Redundancy Check
Carrier Supporting Carrier
Data bus
Distributed Cisco Express Forwarding
Database
Double Data Rate
Distributed Forwarding Card
Denial of Service
Dynamic Random Access Memory
Differentiated Services Code Point
Dynamic Trunking Protocol
Dense Wavelength Division Multiplexing
Ethernet Flow Point
enhanced Internal Gateway Routing Protocol
Embedded Logic Analyzer Module
Ethernet Out of Band Channel
Ethernet over Multiprotocol Label Switching
Equal
Ethernet Services Plus
Forwarding Informatin Base
Field Programmable Device

Glossary
FPGA
Field Programmable Gate Array
FW
Firmware
Gbits
Gigabits
Gbps
Gigabit per second
GByte
Gigabyte
GE
Gigabit Ethernet
GHz
Gigahertz
GRE
Generic routing Encapsulation
HA
High Availability
HbH
Hop by Hop
HSRP
Hot Standby Router Protocol
H/W
Hardware
H-QoS
Hierarchical Quality of Service
ICMP
Internet Control Message Protocol
ID
Identity
IDS
Intrusion Detection system
IFIB
Internal FIB [Forwarding Information Base]
IOS
Internet Operating system
IP
Internet Protocol
IPCP
IP [Internet Protocol] Control Protocol [Part of PPP]
IPSec
Internet Protocol Security
IPv4
Internet Protocol version 4
IPv6
Internet Protocol version 6
IPV6CP
IPv6 [Internet Protocol version 6] control Protocol [Part of PPP}
Intermediate System - Intermediate System [Internal gateway routing protocol]
ISIS

L2
L2PT
L2VPN
L3
L4
LACP
LAN
LC
LCDBUS

LCP
LCRBUS

LDP
LFI
LFIB
LLQ
LPTS
MAC
Mbps
MByte
Mcast
MET
MHz
MIB
MIPS
Mod

Layer 2 [OSI Open Systems Interconnection] OR Leval 2

Layer 2 Protcol Tunneling
Layer 2 VPN [Virtual Private Network]
Layer 3 [OSI Open Systems Interconnection] or Level 3
Layer 4 [OSI Open Systems Interconnection] or Level 4
Link Aggregation Control Protocol
Local Area Network
Line Card
Line Card Data Bus
Link Control Protocol [Part of PPP]
Line Card Results Bus
Label Distribution Protocol
Link Fragment Interleave
Label Forwarding Information Base
Low Latency Queue
Local Packet Transport Services
Media Access Control
Megabits per second
Megabyte
Multicast
Multicast Expansion Table
Megahertz
Management Information Base
Multiprocessor without Interlock Pipeline Stages
Modulo

Glossary
Mpps
MPLS
MPLS-TP

MPP
MQC
MSDP
MSFC
MTU
MUX
NAT
ND
NP
NPU
NSF
NTP
NVRAM

OSM
OSPF
PA
PAgP
PDU
PFC
PIFIB
PoP
POS

Megapackets per second

Multiprotocol Label Switching
Multiprotocol Label Switching - Transport Profile
Management Plane Protection
Modular Quality of service Command line interface
Multicast Source Discovery Protocol
Multilayer Switch Feature Card
Maximum Transmission Unit
Multiplexer
Network Address Translation
Neighbor Discovery [protocol]
Network Processor
Network Processor Unit
Non-Stop forwarding
Network Time Protocol
Non-Volatile Random Access Memory
Optical Services Module
Open Shortest Path First [protocol]
Port Adapter
Port Aggregation Protocol
Protocol Data Unit
Policy Feature Card
Pre-IFIB [Internal Forwarding Information Base]
Point of Presence
Packet Over sonet

PPP
PPS
PSIRT
PW
QoS
RADIUS

RARP
RBUS
RIP
RJ45
RP
RPF
RSP
RSVP
RTBH
SCP
SDRAM

SFP
Sh
SIP
SNMP
SP
SPA
SRAM
SSH

Point to Point Protocol

Packets Per Second
Product Security Incident Reponse Team [Cisco]
Pseudo-Wire
Quality of Service
Remote Authentication Dial In Service [protocol]
Reverse ARP [Address Resolution Protocol]
Results bus
Routing Information Protocol
An 8 wire wiring standard
Routing Processor
Reverse Path Forwarding
Routing and Switching Processor OR Route Switch Processor [Cisco]
Resource reservation protocol
Remote Triggered Black Holing
Secure Copy
Synchronous Dynamic Random Access Memory
Small Form-factor Pluggable
Shaper
Shared Port Adapter Interface Processor [Cisco]
Simple Network Management Protocol
Service Provider OR Switching Processor
Shared Port Adapter
Static Random Access Memory
Secure Shell [protocol]

Glossary
SSO
SSRAM
SUP
SW
TAC
TACACS

TCAM
TCB
TCL
TCP
TDM
TTL
tx
uC
UDLD
uRPF
VACL
VLAN
VPLS
VPN
VRF
VRRP
VTP
VTY
WAN

Stateful Switch Over

Synchronous Static Random Access Memory
Supervisor [Cisco]
Switching
Technical Assistance Center [Cisco]
Terminal access Control Access-Control System [protocol]
Tertiary Content Addressable Memory
Transmission Control Block
Tool Command Language
Transmossion Control Protocol
Time Division Multiplexing
Time To Live
Transmit
Microcontroller
Unidirectional Link Detection
Unicast Reverse Path Forwarding
VLAN [Virtual Local Access Network] Access control List
Virtual Local Access Network
Virtual Provate LAN [Local Access Network] Service
Virtual Private Network
Virtual Routing and Forwarding
Virtual Router Redundancy Protocol
Virtual Trunking Protocol
Virtual Terminal line
Wide Area Network

WFQ
WRR
XML

Weighted Fair Queuing [Cisco]

Weighted Round Robbin
Extensible Markup Language

R&S Related Cisco Education Offerings

Course

Description

Cisco Certification

CCIE R&S Advanced Workshops (CIERS-1 &

CIERS-2) plus
Self Assessments, Workbooks & Labs

Expert level trainings including: instructor led workshops, self

assessments, practice labs and CCIE Lab Builder to prepare candidates
for the CCIE R&S practical exam.

CCIE Routing & Switching

Implementing Cisco IP Routing v2.0

Implementing Cisco IP Switched
Networks V2.0
Troubleshooting and Maintaining
Cisco IP Networks v2.0

Professional level instructor led trainings to prepare candidates for the

CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in
self study eLearning formats with Cisco Learning Labs.

CCNP Routing & Switching

Interconnecting Cisco Networking Devices:

Part 2 (or combined)

Configure, implement and troubleshoot local and wide-area IPv4 and IPv6
networks. Also available in self study eLearning format with Cisco Learning
Lab.

CCNA Routing & Switching

Interconnecting Cisco Networking Devices:

Part 1

Installation, configuration, and basic support of a branch network. Also

available in self study eLearning format with Cisco Learning Lab.

CCENT Routing & Switching

For more details, please visit: [Link]

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@[Link]

Design Cisco Education Offerings

Course

Description

Cisco Certification

Designing Cisco Network Service Architectures

(ARCH)

Provides learner with the ability to perform conceptual, intermediate, and

detailed design of a network infrastructure that supports desired capacity,
performance, availability required for converged Enterprise network
services and applications.

CCDP (Design Professional)

Designing for Cisco Internetwork Solutions

(DESGN)

Instructor led training focused on fundamental design methodologies used

to determine requirements for network performance, security, voice, and
wireless solutions. Prepares candidates for the CCDA certification exam.

CCDA (Design Associate)

For more details, please visit: [Link]

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@[Link]

Service Provider Cisco Education Offerings

Course
Deploying Cisco Service Provider Network Routing
(SPROUTE) & Advanced (SPADVROUTE)
Implementing Cisco Service Provider Next-Generation
Core Network Services (SPCORE)

Description
SPROUTE covers the implementation of routing protocols (OSPF, IS-IS, BGP),
route manipulations, and HA routing features; SPADVROUTE covers advanced
routing topics in BGP, multicast services including PIM-SM, and IPv6;

Cisco Certification
CCNP Service Provider

SPCORE covers network services, including MPLS-LDP, MPLS traffic engineering,

QoS mechanisms, and transport technologies;

Edge Network Services (SPEDGE)

SPEDGE covers network services, including MPLS Layer 3 VPNs, Layer 2 VPNs,
and Carrier Ethernet services; all within SP IP NGN environments.

Building Cisco Service Provider Next-Generation

Networks, Part 1&2 (SPNGN1), (SPNGN2)

The two courses introduce networking technologies and solutions, including OSI
and TCP/IP models, IPv4/v6, switching, routing, transport types, security, network
management, and Cisco OS (IOS and IOS XR).

CCNA Service Provider

Implementing Cisco Service Provider Mobility UMTS

Networks (SPUMTS);
Implementing Cisco Service Provider Mobility CDMA
Networks (SPCDMA);
Implementing Cisco Service Provider Mobility LTE
Networks (SPLTE)

The three courses (SPUMTS, SPCDMA, SPLTE) cover knowledge and skills
required to understand products, technologies, and architectures that are found in
Universal Mobile Telecommunications Systems (UMTS) and Code Division Multiple
Access (CDMA) packet core networks, plus their migration to Long-Term Evolution
(LTE) Evolved Packet Systems (EPS), including Evolved Packet Core (EPC) and
Radio Access Networks (RANs).

Cisco Service Provider Mobility

CDMA to LTE Specialist;
Cisco Service Provider Mobility UMTS
to LTE Specialist

Implementing and Maintaining Cisco Technologies

Using IOS XR (IMTXR)

Service Provider/Enterprise engineers to implement, verification-test, and optimize

core/edge technologies in a Cisco IOS XR environment.

Cisco IOS XR Specialist

For more details, please visit: [Link]

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@[Link]

Network Programmability Cisco Education Offerings

Course

Description

Cisco Certification

Integrating Business Applications with Network

Programmability (NIPBA);
Integrating Business Applications with Network
Programmability for Cisco ACI (NPIBAACI)

Learn networking concepts, and how to deploy and troubleshoot

programmable network architectures with these self-paced courses.

Cisco Business Application

Engineer Specialist Certification

Developing with Cisco Network Programmability

(NPDEV);
Developing with Cisco Network Programmability
for Cisco ACI (NPDEVACI)

Learn how to build applications for network environments and effectively

bridge the gap between IT professionals and software developers.

Cisco Network Programmability

Developer Specialist Certification

Designing with Cisco Network Programmability

(NPDES);
Designing with Cisco Network Programmability
for Cisco ACI (NPDESACI)

Learn how to expand your skill set from traditional IT infrastructure to

application integration through programmability.

Cisco Network Programmability

Design Specialist Certification

Implementing Cisco Network Programmability

(NPENG);
Implementing Cisco Network Programmability
for Cisco ACI (NPENGACI)

Learn how to implement and troubleshoot open IT infrastructure

technologies.

Cisco Network Programmability

Engineer Specialist Certification

For more details, please visit: [Link]

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@[Link]