Integrating Cumulus VX with VMware NSX using VMware ESXi

  • 5
  • Article
  • Updated 12 months ago
  • (Edited)

This article provides step-by-step instructions on how to setup a Cumulus VX virtual machines to integrate with the VMware NSX management framework and act as VTEP gateways. In this setup, all nodes are configured as VMs within a vSphere ESXi hypervisor.

Please refer to the schematic diagram for the setup below:

  • Configure the management nodes.

    • Create the NSX management node VMs (the NSX Controller, NSX Manager and NSX Service Node) in ESXi, which are used to establish a management network.

    • Create a datapath connection from the service node to one of the four nodes in 2L2S topology so that the service node has layer 3 connectivity with the network.

    • Alternatively, the NSX nodes can all be managed in band on the data network without a separate management network. To do this, the NSX nodes must be connected and reachable on the the layer 3 data network.

  • For this setup, each connection must behave like a point-to-point connection. However, ESXi by default adds each network adapter to a shared virtual bridge (VM network). In order to create point-to-point connection, you need to create a separate port group for each set of endpoints in the setup. To configure port groups, please refer to this community article (which is also needed if you want to configure unnumbered OSPF/BGP with 2L2S topology in step 2):

  • Additionally, by default promiscuous mode is disabled for vSwitch on ESXi, which prevents ARP replies from passing through when they are not learned locally, as is the case in this setting. In order to let all traffic pass through, you must enable promiscuous mode on the port group that connects leaf1 to host1 and leaf2 to host2. To enable promiscuous mode on a particular port group, go to ESXi Server and choose Configuration > Networking > Properties. Select the port group ("VX" in this case) and click Edit. On the Security tab, check the box to enable Promiscuous Mode:

  • Once enabled, you can verify that Promiscuous Mode status on the Ports tab appears as Accepted:

  • In this example, you can add leaf1 and leaf2 as two VTEP-enabled gateways, using VXLAN as the Transport Type. Also, add swp3 on leaf1 and swp3 on leaf2 as two gateway services. Then, create a logical network with the VXLAN transport type and give it a VNI. Finally, create two logical switch ports, using each gateway service you just created and add them to the logical switch.

  • Once all the configuration is complete, assign IP addresses to host1 and host2. The hosts' IP address space is independent of the underlying physical network since you created a layer 2 logical overlay network over the two-leaf, two-spine physical network using VXLAN; thus the two hosts will be able to ping each other using VXLAN tunneling. You can check ARP information on each host to see that the other host’s ARP is resolved and the two hosts are on the same layer 2 network.

Caution: The current Cumulus VX 2.5.3 image has a resource leak in the VTEP daemon that runs on the switch. You must restart the daemon every hour (using "service openvswitch-vtep restart"). This will be fixed in the next Cumulus VX release.   

Photo of Dev

Dev, Alum

  • 150 Points 100 badge 2x thumb

Posted 3 years ago

  • 5
Photo of Neel


  • 92 Points 75 badge 2x thumb
none of the diagram work, could you check on that pls