VXLAN routing recommendations from Cumulous Networks

VXLAN routing recommendations from Cumulous Networks, which offers switch software (but not client software).

https://cumulusnetworks.com/blog/vxlan-designs-part-1/

VXLAN routing is the process in which a VTEP receives a VXLAN packet destined to itself, removes the VXLAN header and then performs a layer 3 route lookup on the inner decapsulated packet. Since the VTEP has to perform two sets of lookups, first on the encapsulated VXLAN traffic then on the decapsulated inner packet, it requires special hardware ASIC to perform both lookups in a single pass all in hardware.

A drawback of this solution is east-west traffic. Any inter-vxlan routed traffic has to trombone all the way to the exit leaf before making its way back through VXLAN to the destination. If you had two servers connected to the same TOR but on two different VXLAN, that traffic would have to move all the way through your CLOS network. This could cause an issue with scaling an environment that is primarily east-west traffic.

https://cumulusnetworks.com/blog/vxlan-designs-part-2/

the next logical place to put a gateway in VXLAN designs is to distribute them all on the top of rack (TOR), also known as the leaf. This TOR acts as a VTEP in the VXLAN solution. Its primary purpose is to encapsulate and decapsulate traffic. This solution is also colloquially known as Anycast Gateway VXLAN Routing. Anycast Gateway VXLAN Routing can only be performed on ASICs that support routing in and out of tunnels.

It shouldn’t be surprising that their recommendations are to do routing in the network hardware, rather than the VM host. They are a vendor of software for network hardware, not the compute hosts, after all.