x86 server

  • 1
  • Question
  • Updated 1 year ago
Can I run Cumulus Linux on a dual processor x86 server with six 6-port 10G ethernet nics? This is for collapsing server & switch into a single device for running processor intensive apps such as IPS, FW, LB, and other network functions.
Photo of Charles McCane

Charles McCane

  • 80 Points 75 badge 2x thumb

Posted 1 year ago

  • 1
Photo of Pete B

Pete B, Official Rep

  • 2,798 Points 2k badge 2x thumb
It might be possible, but it would be really tricky. 

You'd need to have ONIE on the server so you can install Cumulus Linux. Otherwise, the installation could be problematic.

If you do manage to install it, you won't get the hardware acceleration from the switching ASIC, since you won't have one in the server.

And in any event, it won't be a supported configuration, so you'd be on your own if you run into trouble. 
Photo of Jason Guy

Jason Guy, Employee

  • 1,712 Points 1k badge 2x thumb
I was thinking about this for a few days now. What specifically is needed on the host? We do have a supported package for routing on the host. This will turn your host into a router, Other than that, the bridging and other Debian packages are still available on the host. So I suppose some more information about the network design is needed to help solve this.
Photo of Charles McCane

Charles McCane

  • 80 Points 75 badge 2x thumb
Thanks for responding. This use case is for a cloud stack where there is a need to maximize revenue RU's. In such a stack there's a high value placed on the data plane ToR as you'd expect to see not more than a couple of RU's dedicated to switching. Increasing the value of the switch RU's might come from packaging in network vNF's onto the same 1RU device that's serving as the data plane ToR. I see value from an ODM switch/server combo that would be a server on the back but with a 36 port switch fascia on the front. If a 1RU SuperMicro Twin can fit two dual processor servers into 1RU, then it should be possible to put a switch and a server into the same 1RU form factor. I have a use case where I believe that an entire rack of Gigamon taps and security appliances could be collapsed into a custom 1RU ODM switch/server combo. With the right design, the real challenges could be pushed into software. Linux, with the ability to run multiple vNF's, would be perfect. Everything on the server software stack would be DPDK optimized. Not sure if there would need to be a separate OS for switching and another for processing but it would be compelling to have one overall system image that could control both sides.
Photo of Jason Guy

Jason Guy, Employee

  • 1,712 Points 1k badge 2x thumb
Hi Charles, I see what you mean now. This is a good idea. Seems like you are describing a hyper-converged switch/server platform. Cumulus linux can run on any x86 platform, and is simply debian, so it could be a server today. DPDK and SRIOV are fairly complicated and can be taxing on the server. So a true switching engine attached to the server would be excellent.

The current limitation is the way the switching ASIC is connected to the system, currently a PCIEx4 device. I am not sure if there is a device currently being produced, but the ODMs would need to create a device that has additional ethernet data connections into the ASIC. Perhaps 40G-KR connections internally connected to a PCI gen3 device? This would provide real backplane data connections into the switching engine (as far as I understand it).

Do you know of a switching platform being produced that has this design? If so, getting ONIE on that is doable, and running Cumulus Linux is certainly possible. If not, I would inquire with the ODMs and see if this server/switch design is in the works.