Running modern SaltStack (2018.3.0) on Cumulus

  • 1
  • Question
  • Updated 2 weeks ago
Cumulus Documentation states that the ability to bind a minion to a specific port or IP is only present in recent versions, e.g. https://docs.saltstack.com/en/latest/ref/configuration/minion.html#source-interface-name.  Upon running the bootstrap script, I'm able to get it installed with -X -d options, e.g.:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 14.0px 'Liberation Mono'; color: #ffffff; background-color: #000000; background-color: rgba(0, 0, 0, 0.68)} span.s1 {font-variant-ligatures: no-common-ligatures}
sudo sh bootstrap-salt.sh -X -d git v2018.3.0

But the minion will fail to launch because python-tornado is too old (from initial searches).  I haven't found a way around this yet, but in our environment, we need to bind to eth0/mgmt vrf, as we're running eBGP unnumbered, and we have issues communicating with the Salt Master.  I'm just wondering:

- is there a known fix to bind the interface/IP on the salt-minion that comes with Cumulus?
- is there another repo I can install from with newer packages known to work?
- any luck with early-access repos (don't really want to enable for other reasons)

I'm open to any suggestions - it seems best to just run Ansible at this point?  I really can't believe that Salt only recently added the binding options for the minion.  Am I missing something obvious?
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
  • done w/Salt on Cumulus ;)

Posted 2 weeks ago

  • 1
Photo of Pete Lumbis

Pete Lumbis, Employee

  • 684 Points 500 badge 2x thumb
I'm a little dense here, is the problem with the config or with the minion install? What version of Cumulus Linux are you using?

I was able to install the minion after adding the global debian repo to the apt sources.

-Pete
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
v3.5.3 - I was missing the second repo from the https://support.cumulusnetworks.com/hc/en-us/articles/215248118-Configuring-SaltStack-on-Cumulus-Linux doc.  That's definitely the path of least resistance, thanks!  (not having it gets you 2014.something).
(Edited)
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
Okay, everything's going well.  I still haven't looked into where this fits into the mgmt VRF doc/style (https://docs.cumulusnetworks.com/display/DOCS/Management+VRF ), but for now, I just included a line for eth0 in the minion configuration file.
(Edited)
Photo of Pete Lumbis

Pete Lumbis, Employee

  • 684 Points 500 badge 2x thumb
let me know how the testing goes. We'd love to have a KB about running the salt minion in a vrf. If you are binding to eth0 then it should only do route lookups in the mgmt vrf and operate as expected. "Should" being the keyword :)
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
I'd like to do some massive testing with a VX setup to determine exactly what the issues are.  Disabling IPv6 completely solved the issue, even though I was using interface + IP(v4) from the minion configs.  The issues were with the spines (at least in our configs).

Some notes for myself/others:
- hostname/dns/fqdn resolution
- disabling IPv6 on master vs. troubleshooting minion -> master comms
- referencing 'mgmt' vs. 'eth0' (works for ping) - also, config files, but is there a difference?

In this case, the mgmt VRFs are /30'ed off of an edge router (new setup), and bounce right back into the leaf (where the salt server is).  I'll get a more realistic demo in VX (and go from there).

Edit: seeing this in the logs:
[WARNING ] Unable to connect to the Master using a specific source IP / port
[WARNING ] Consider upgrading to pyzmq >= 16.0.1 and libzmq >= 4.1.6

Also, noticed this issue in the Saltstack repo: https://github.com/saltstack/salt/issues/38744
(Edited)
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
https://docs.saltstack.com/en/latest/topics/troubleshooting/minion.html  really helped, and running:
salt-minion -l debug
never has any issues - I can keep running commands and they never occasionally fail like when it's running as a daemon.  I'll keep looking into it tomorrow, and try on VX when I have time.
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
Okay, assuming you're running a Cumulus Validated Design for eBGP unnumbered, the salt-minion will respond on the `lo` address of 10.0.0.21 or 22 (at least for me).  I can't get it to bind, which pointed me to this link for the minion 

https://docs.saltstack.com/en/latest/ref/configuration/minion.html#source-interface-name

Warning
This option requires modern version of the underlying libraries used by the selected transport:
  • zeromq requires pyzmq >= 16.0.1 and libzmq >= 4.1.6
  • tcp requires tornado >= 4.5


On the Cumulus side:

cumulus@sw-spine1:mgmt-vrf:~$ sudo apt-cache policy libzmq3
libzmq3:
  Installed: 4.0.5+dfsg-3
  Candidate: 4.0.5+dfsg-3
  Version table:
 *** 4.0.5+dfsg-3 0
        500 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
     4.0.5+dfsg-2+deb8u1 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
cumulus@sw-spine1:mgmt-vrf:~$ sudo apt-cache policy python-zmq
python-zmq:
  Installed: 14.4.0-1
  Candidate: 14.4.0-1
  Version table:
 *** 14.4.0-1 0
        500 http://ftp.us.debian.org/debian/ jessie/main amd64 Packages
        100 /var/lib/dpkg/status
     14.4.0-1 0
        500 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main amd64 Packages



Salt on Cumulus always seems to want to use the `lo` IP for communication in this scenario, which has intermittent success.  Ironically, it works 100% of the time when I run the minions manually in debug mode.  I'll tcpdump that to see what the difference is.

Any ideas?
Photo of jd

jd

  • 132 Points 100 badge 2x thumb
To summarize:

  • the minions can't set source interface or IP because of older zmq/tornado packages
  • when the minion daemon is running manually (with/without debug mode), the minions use the FQDN IP as source, which is the eth0/mgmt-vrp in this case (which has zero issues)
  • for now, just going to run with -d, instead of using default init scripts (still not sure what environment change is doing this) - both are running as root, although I'm launching the daemon manually with sudo
[WARNING ] Unable to connect to the Master using a specific source IP / port
[WARNING ] Consider upgrading to pyzmq >= 16.0.1 and libzmq >= 4.1.6