This article provides step-by-step instructions on how to setup a Cumulus VX virtual machines to integrate with the VMware NSX management framework and act as VTEP gateways. In this setup, all nodes are configured as VMs within a vSphere ESXi hypervisor.
Please refer to the schematic diagram for the setup below:
First, create a two-leaf, two-spine (2L2S) Clos topology with Cumulus VX VMs. Make sure you download the VMware OVA image.
Configure OSPF as the routing protocol so that the each VM can ping all of the others. This constitutes the "physical" network framework for the NSX setup. For information about configuring the VMs, read this article: http://docs.cumulusnetworks.com/display/VX/Using+Cumulus+VX+with+VMware+vSphere+-+ESXi+5.5
Configure the management nodes.
Create the NSX management node VMs (the NSX Controller, NSX Manager and NSX Service Node) in ESXi, which are used to establish a management network.
Create a datapath connection from the service node to one of the four nodes in 2L2S topology so that the service node has layer 3 connectivity with the network.
Alternatively, the NSX nodes can all be managed in band on the data network without a separate management network. To do this, the NSX nodes must be connected and reachable on the the layer 3 data network.
Create two host VMs, host1 connected to leaf1 and host2 connected to leaf2 respectively. Configure host1 and host2 to act as bare metal logical endpoints connected to the VTEPs configured in leaf1 and leaf2 respectively. For more information on configuring the VTEPs, read:
For this setup, each connection must behave like a point-to-point connection. However, ESXi by default adds each network adapter to a shared virtual bridge (VM network). In order to create point-to-point connection, you need to create a separate port group for each set of endpoints in the setup. To configure port groups, please refer to this community article (which is also needed if you want to configure unnumbered OSPF/BGP with 2L2S topology in step 2): https://community.cumulusnetworks.com/cumulus/topics/un-numbered-ospf-bgp-setup-on-vmware-esxi-with-cumulus-vx
Additionally, by default promiscuous mode is disabled for vSwitch on ESXi, which prevents ARP replies from passing through when they are not learned locally, as is the case in this setting. In order to let all traffic pass through, you must enable promiscuous mode on the port group that connects leaf1 to host1 and leaf2 to host2. To enable promiscuous mode on a particular port group, go to ESXi Server and choose Configuration > Networking > Properties. Select the port group ("VX" in this case) and click Edit. On the Security tab, check the box to enable Promiscuous Mode:
Once enabled, you can verify that Promiscuous Mode status on the Ports tab appears as Accepted:
Once the basic connectivity is set up, configure the VTEPs and NSX. For details, read: http://docs.cumulusnetworks.com/display/DOCS/Integrating+with+VMware+NSX.
In this example, you can add leaf1 and leaf2 as two VTEP-enabled gateways, using VXLAN as the Transport Type. Also, add swp3 on leaf1 and swp3 on leaf2 as two gateway services. Then, create a logical network with the VXLAN transport type and give it a VNI. Finally, create two logical switch ports, using each gateway service you just created and add them to the logical switch.
Once all the configuration is complete, assign IP addresses to host1 and host2. The hosts' IP address space is independent of the underlying physical network since you created a layer 2 logical overlay network over the two-leaf, two-spine physical network using VXLAN; thus the two hosts will be able to ping each other using VXLAN tunneling. You can check ARP information on each host to see that the other host’s ARP is resolved and the two hosts are on the same layer 2 network.
Caution: The current Cumulus VX 2.5.3 image has a resource leak in the VTEP daemon that runs on the switch. You must restart the daemon every hour (using "service openvswitch-vtep restart"). This will be fixed in the next Cumulus VX release.