In this post, we will examine a corner case about EVPN-VXLAN of Nexus9k devices. The customer has a specific design between two cities for Active/Passive DC network redundancy. Fortigate firewalls are gateways for each DC and they are reachable over IPSEC-SDWAN with radiolink and fiber circuits. Hence, we will configure EVPN-VXLAN over this IPSEC-SDWAN path to provide active/passive DC by extending VLANs. However, there are active-passive firewalls and this causes a problem in EVPN fabric which we will explain below.
Requirement
- The customer needs to build their active/passive DC, hence we need a Layer 2 extension to migrate VM machines among DCs.
- Since there are some budget consideration, layer 2 physical connection is not ready. We have IPSEC-SDWAN connection between sites. We should use it.
- Fortigate supports VXLAN. However, all vlans are under zone configuration. Fortigate needs to be removed from zone to configure vxlan. This is critical for policy management and will take long time to deploy.
- Customer has 4 idle Nexus devices, we decided to use them efficiently.
Design
- Fortigate devices on site1 and site2 are connected SDWAN-IPSEC through 2 redundant links.
- Nexus devices are in VPC on each site domain. These devices will be VTEP.
- VTEPs will have loopback0 to establish EVPN.These loopback0 are reachable through fortigate sdwan. We will configure multihop bgp.
Problem
As you can see from the topology, facing passive firewall will not get traffic. Leaf1 learns destination mac address from site2 via EVPN-VXLAN. But,Leaf2 is not getting those mac addresses through the VPC peer-link. In order to install mac addresses to leaf2, it needs an EVPN neighborship. This neighborship will reach through VPC peerlink to site 2 VPC leafs.
When any traffic comes to leaf2, leaf2 will see the destination mac address is learned from vxlan interface. To forward traffic, leaf2 will use vpc-peerlink since there is only this out. But leaf2 encapsulates VXLAN the packets it leaf2 learns from EVPN. This causes blackhole problem. Encapsulated packets are not forwarded through VPC peer-link
Solution
Cisco nexus suggest using infra-vlans for routing over vpc peer-link in such scenarios. Since there is IBGP between VPC pairs, you should enable infra-vlans to enable forward VXLAN traffic forwarding. If this command does not exist, some servers will have reachability issue. I detected this problem by rebooting leaf2. Later I only made shutdown of leaf2 ports which are facing downstream. This was workaround. Finally I found the article below and applied it in order to fix the problem permanently.
system nve infra-vlans 3960