ACI Multipod Configuration
and
Common Issues
Presenter: Edi Wibowo
Panelist: Linda Wang, John Meng and Stephanie Souvleris
Feb 2018
‘Wisdom is not a product of schooling but of the lifelong attempt to acquire it.’
Albert Einstein
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multipod Solution
Agenda
•
• Configuration Overview
• Common Issues
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Acronyms
VXLAN Virtual Extensible LAN
BUM Layer 2 Broadcast, Unknown unicast, and Multicast
APIC Application Policy Infrastructure Controller
PIM Protocol-Independent Multicast is a multicast routing
protocols for Internet Protocol networks
MP-BGP Multi Protocol Border Gateway Protocol
EVPN Ethernet VPN - MAC addresses to be treated as
routes in the BGP table.
ETEP External Tunnel End Point (Data Plane Forwarding)
CP-ETEP Control Plane Tunnel End Point (MP-BGP Neighbors)
MTU Maximum Transmission Unit
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
VXLAN
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multipod Topology
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multipod Benefits
• Single APIC Cluster/Single Domain
A single APIC controller cluster representing the single point of
management and policy definition for the entire network, independently
from the number of separate ACI fabrics (Pods) compounding it.
• Active/Active
Data Centers are deployed in multiple Pods, so to offer the freedom of
deploying the various application components across separate Pods.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Endpoint Learning Models of VXLAN
• Flood-and-Learn (Local Leaf to Local Leaf): Inside ACI Fabric POD
In this model, end-host information learning and VTEP discovery are both data-plane
based, with no control protocol to distribute end-host reachability information among
VTEPs.
• MP-BGP EVPN (Local Spine to Remote Spine): Between PODs via IPN
It provides control-plane learning for end hosts behind remote VTEPs. It uses a unified
control plane for both Layer 2 and Layer 3 forwarding in a VXLAN overlay network.
Each route carries the BD-VNID and/or VRF-VNID in the label field of EVPN route.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MP-BGP EVPN Benefits
• It uses the very well known MP-BGP VPN technology to support scalable multitenant
VXLAN overlay networks.
• The EVPN address family carries both Layer 2 and Layer 3 reachability information, thus
providing integrated bridging and routing in VXLAN overlay networks.
• It reduces network flooding through protocol-based host MAC/IP route distribution
• It provides optimal forwarding for east-west and north-south traffic and supports
workload mobility with the distributed anycast gateway function.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multipod Solution
Agenda
•
• Configuration Overview
• Common Issues
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Configuration Steps
1. On IPN routers: Configure DHCP relay addresses pointing to APICs on POD-1 (Seed
POD)
2. On IPN routers: Configure PIM sparse-mode bidir to build multicast groups for
overlay/tenant’s BUM data traffic
3. On IPN routers and Spines: Configure OSPF
4. On Spines: Configure OSPF and ISIS redistribution
5. On Spines: Configure MP-BGP EVPN
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Inter-Pod Network (IPN) Routers
• Fully connected IP network
• 40G/100G ports to connect to spines in all PODs
• IP Multicast – PIM Bidir support – Standalone N9K, N3548 etc
• OSPF protocol for inter-POD reachability
• DHCP Relay to APICs
• Use infra-vlan 4 as sub-interface encap vlan between IPN and spine
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
IPN Configuration
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
OSPF and MP-BGP Configuration on Spines
MP-BGP EVPN Interfaces (CP-ETEPs)
OSPF Interfaces
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MP-BGP EVPN Full Mesh or RR
External Tunnel End Point (ETEP)
Redistribution into ISIS from OSPF
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
MP-BGP EVPN Route Reflectors (RRs)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Verifying OSPF and BGP (1)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
External Tunnel End Point (ETEP)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Verifying OSPF and BGP (2)
OSPF session used
for CP-ETEPs reachability
CP-ETEPs used
for MP-BGP session
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Verifying OSPF and BGP (3)
dp2-p1-s1# show bgp l2vpn evpn vrf overlay-1
BGP routing table information for VRF overlay-1, address family L2VPN EVPNBGP table version
is 578, local router ID is 192.168.1.101Status: s-suppressed, x-deleted, S-stale, d-
dampened, h-history, *-valid, >-bestPath type: i-internal, e-external, c-confed, l-local,
a-aggregate, r-redist, I-injectedOrigin codes: i - IGP, e - EGP, ? - incomplete, | -
multipath, & - backup
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: [Link]:65003 (L2VNI1)
*>l[2]:[0]:[14680073]:[48]:[dcce.c15b.1e46]:[0]:[[Link]]/216
[Link] 100 32768
i*>i[2]:[0]:[14712846]:[48]:[dcce.c15b.1e47]:[0]:[[Link]]/216
[Link] 100 0 ETEPs used as next-hops
i*>l[2]:[0]:[14843893]:[48]:[000c.29dd.0164]:[0]:[[Link]]/216 In data traffic forwarding
[Link] 100 32768
i*>i[2]:[0]:[14843893]:[48]:[003a.7d4e.640c]:[0]:[[Link]]/216
[Link] 100 0 i
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Endpoint Route Distribution
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multipod Building Blocks
OSPF OSPF
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Multicast Groups
• Tenant’s BUM traffic is flooded to a multicast group so that other PODs receive the
traffic. The group is advertised to other PODs through BGP EVPN type-6 routes.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
ACI Multipod Solution
Agenda
•
• Configuration Overview
• Common Issues
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #1: Phantom RP (1)
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #1: Phantom RP
• Loopback address uses different lengths.
• IGP problem: RP address is not in the routing table of all IPNs.
• Loopback must be “ip ospf network-type point-to-point”
By default OSPF will advertise this route to loopback as /32 (most specific
route to that loopback). To override this we have to change the network type to
point-to-point. After this OSPF will advertise the address to loopback as /30 or
/29.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #2: L3 MTU
Since VXLAN data-plane traffic is MAC-in-IP encapsulation, the IPN must ensure to be
able to support an increased MTU on its physical connections, in order to avoid the need
for fragmentation and reassembly. The requirement is to increase to 9150 bytes the
supported MTU on all the Layer 3 interfaces of the IPN devices.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #3: Remote APICs
Cluster configuration ...
Enter the fabric name [ACI Fabric1 #1]: dp2-fabric
Enter the fabric ID (1-128) [1]:
Enter the number of controllers in the fabric (1-9) [3]: 5
Enter the POD ID (1-9): 2 POD ID 2 TEP Pool of POD 1
Enter the controller ID (1-3) [1]:
Enter the controller name [apic1]: dp2-apic4
Enter address pool for TEP addresses [[Link]/16]: [Link]/16
Note: The infra VLAN ID should not be used elsewhere in your environment and
should not overlap with any other reserved VLANs on other platforms.
Enter the VLAN ID for infra network (2-4094): 3967
Enter address pool for BD multicast addresses (GIPO) [[Link]/15]:
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #4: Known Bugs (1)
CSCuz28088
Symptom:
Fabric discovery will fail if ospf auth configured in the spine which is connected to pod-1 via IPN.
Conditions:
Fabric discovery will fail if ospf auth configured in the spine which is connected to pod-1 via IPN.
Workaround:
Disable ospf auth before fabric bringup/spine reboot.
Further Problem Description:
Fabric discovery will fail if ospf auth configured in the spine which is connected to pod-1 via IPN.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Common Issue #4: Known Bugs (2)
CSCvh29461
Symptom:
BGP between spine switches in different pods in a multipod environment goes down and is unable to be re-
established. After upgrading to a Cisco APIC 3.1 release, BGP between spine switches in different pods goes down if
a QoS CoS translation policy is enabled.
Conditions:
In a multipod environment with spine switches that have generation EX line cards:
1. Upgrade to 3.1(1i)
2. Enable a DSCP CoS translation policy under the infra tenant.
3. Within this policy, set the control plane policy to cs7.
Additionally, if the IPN is marking BGP traffic to DSCP 56, 59, 60, 61, 62, or 63, this condition can be hit as well.
The BGP traffic will not be classified as BGP traffic internally and as a result the spines will not see additional
messages received from the peers in the remote pod.
Workaround:
Disable the QoS cos-translation policy or mark / set control-plane traffic to something besides CS7
[Link]
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Thank You
Post your Questions at Community Support
[Link]
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Reference
• [Link]
virtualization/application-centric-infrastructure/[Link]
• [Link]
virtualization/application-centric-infrastructure/[Link]
• [Link]
x/L3_config/b_Cisco_APIC_Layer_3_Configuration_Guide/b_Cisco_APIC_Layer_3_Co
nfiguration_Guide_chapter_010011.html
• [Link]
d_QoS.html
• Building Data Centers with VXLAN BGP EVPN: A Cisco NX-OS Perspective by David
Jansen; Shyam Kapadia; Lukas Krattiger
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Confidential