Link from Cumulus to Switch without cumulus dont run more than 3 hours on QSFP+ port, why? 10GbE SFP+ runs. any idea?!

  • 1
  • Problem
  • Updated 11 months ago
Cumulus Switch connected with QSFP+ / 40GbE Link goes offline after 2-3 hours, why? Link works with 10GbE but we will use the 40GbE Ports. Any idea? From Cumulus to Cumulus, it works, but not to the Switch without cumulus. We tried it with few FW versions but nothing helps... Also Reflashed the Gbics with diffrent Brand Firmware...
Photo of Andreas Bamert

Andreas Bamert

  • 82 Points 75 badge 2x thumb

Posted 11 months ago

  • 1
Photo of Sean Cavanaugh

Sean Cavanaugh, Alum

  • 3,360 Points 3k badge 2x thumb
Hey Andreas,

If you have real hardware feel free to open a support ticket with Cumulus global support services:

That being said.

goes offline after 2-3 hours
That is too ambiguous, what does "offline" mean here?  When you do a ip link show is LOWER_UP on the 40Gb port?  Do you have a diagram so we can reference the right ports and ask questions?  Do you have config, model numbers, etc?
Photo of Andreas Bamert

Andreas Bamert

  • 82 Points 75 badge 2x thumb
Hi Forum, our admin is in contact with cumulus specialists... But since now we havent got a solution. So i tought i ask that forum, if someone has the same problem. i am little smarter now... Link goes down, from up do down. the Physical link. We use a MLNX SN2700 100GbE Switch with cumulus 3.2.1. limited one port to QSFP+ 40GbE, and connected that one to a quanta T3048-LY2 Switch.
I tested that szenarion with direct attached cables, link is up. So i think that must be a Gbic Problem. We have a distance about 30 meters at clients racks. MPO costs to much, so we deside to do a LC-LC connection, unfortunately we have no gbics for that problem, no mellanox FW for Gbics. Has someone an idea to use QSFP+ with LC-LC cables? i try it with cisco firmware, that works but only for 2-3 hours, than the link is down. now i thougt that the gbics are the problem, we use gbics which allow us to change the firmware on it... tomorrow i start few tests with diffrent firmwares, if someone has a smart idea, many thanks from my side...
Photo of Erik Auerswald

Erik Auerswald

  • 270 Points 250 badge 2x thumb
Hi Andreas,

to use LC-LC (two fibers) with QSFP+ you need the correct kind of QSFP+ module that uses multiple wavelengths to transmit 4 data streams over one fiber. The branding on the modules is just used to white-list vendor supplied modules, it does not change the functionality.

Link-down after a couple of hours might be related to temperature (overheating QSFP+ modules) or too many errors on the link resulting in an error disabled port on one side. It might be something else as well, of course.

You might be able to use active optical cables (AOC) for a distance of about 30m, which should be significantly less expensive than using above mentioned QSFP+ modules.

Edit: See the PacketPushers blog post Why Compatibility And Support Don’t Justify $1000 Optics for an interesting perspective on the matter.