Need help with Cumulus VX: Host multi-Chassis LAG(CLAG)

  • 1
  • Problem
  • Updated 3 years ago
  • (Edited)
Originally posted here: https://community.gns3.com/people/hemantsharma/blog/2015/10/24/cumulus-vx-host-multi-chassis-lagclag

Added the VRR config as a comment

PROBLEM: The CLAG comes up and leaf-1 can ping VRR IP address on switch-1. But when I either shutdown/reboot that switch OR shutdown all its interfaces, the Ping STOPS working.

Please help!!

cumulus@switch-1$ sudo ifquery -a
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

auto swp1
iface swp1

auto swp2
iface swp2

auto swp3
iface swp3

auto peerlink
iface peerlink
        bond-slaves swp1 swp2
        bond-mode 802.3ad
        bond-miimon 100
        bond-use-carrier 1
        bond-lacp-rate 1
        bond-min-links 1
        bond-xmit-hash-policy layer3+4
 
auto peerlink.4094
iface peerlink.4094
        address 169.254.1.1/30
        clagd-enable yes
        clagd-peer-ip 169.254.1.2
        clagd-sys-mac 44:38:39:FF:40:94
        clagd-args --vm
 
auto bond0
iface bond0
        bond-slaves swp3
        bond-mode 802.3ad
        bond-miimon 100
        bond-use-carrier 1
        bond-lacp-rate 1
        bond-min-links 1
        bond-xmit-hash-policy layer3+4
        clag-id 1
 
auto bond0.10
iface bond0.10
 
auto peerlink.10
iface peerlink.10
 
auto br10
iface br10
        bridge-ports bond0.10 peerlink.10
        address 1.1.1.1/24
        address-virtual 44:38:39:00:00:10 1.1.1.254/24
 
cumulus@switch-1$
 
 
 
cumulus@switch-2$ sudo ifquery -a
auto lo
iface lo inet loopback
 
auto eth0
iface eth0 inet dhcp
 
auto swp1
iface swp1
 
auto swp2
iface swp2
 
auto swp3
iface swp3
 
auto peerlink
iface peerlink

        bond-slaves swp1 swp2
        bond-mode 802.3ad
        bond-miimon 100
        bond-use-carrier 1
        bond-lacp-rate 1
        bond-min-links 1
        bond-xmit-hash-policy layer3+4
 
auto peerlink.4094
iface peerlink.4094
        address 169.254.1.2/30
        clagd-enable yes
        clagd-peer-ip 169.254.1.1
        clagd-sys-mac 44:38:39:FF:40:94
        clagd-args --vm
 
auto bond0
iface bond0
        bond-slaves swp3
        bond-mode 802.3ad
        bond-miimon 100
        bond-use-carrier 1
        bond-lacp-rate 1
        bond-min-links 1
        bond-xmit-hash-policy layer3+4
        clag-id 1
 
auto bond0.10
iface bond0.10
 
auto peerlink.10
iface peerlink.10
 
auto br10
iface br10
        bridge-ports bond0.10 peerlink.10
        address 1.1.1.2/24
        address-virtual 44:38:39:00:00:10 1.1.1.254/24
 
cumulus@switch-2$
 
 
cumulus@leaf-1$ sudo ifquery -a
auto lo
iface lo inet loopback
 
auto eth0
iface eth0 inet dhcp
 
auto swp1
iface swp1
 
auto swp2
iface swp2
 
auto spine1-2
iface spine1-2
        bond-slaves swp1 swp2
        bond-mode 802.3ad
        bond-miimon 100
        bond-use-carrier 1
        bond-lacp-rate 1
        bond-min-links 1
        bond-xmit-hash-policy layer3+4
 
auto spine1-2.10
iface spine1-2.10
 
auto br10
iface br10
        bridge-ports spine1-2.10
        address 1.1.1.100/24
 
cumulus@leaf-1$
 
cumulus@switch-1$ sudo cat /etc/ifplugd/action.d/ifupdown
#!/bin/sh
#set -e
 
#case "$2" in
#up)
#       /sbin/ifup $1
#       ;;
#down)
#       /sbin/ifdown $1
#       ;;
#esac
 
 
#!/bin/sh
set -e
case "$2" in
up)
        clagrole=$(clagctl | grep "Our Priority" | awk '{print $8}')
        if [ "$clagrole" = "secondary" ]
        then
                #List all the interfaces below to bring up when clag peerbond comes up.
            for interface in swp3 bond0
                do
                echo "bringing up : $interface"
                ip link set $interface up
            done
        fi
    ;;
down)
        clagrole=$(clagctl | grep "Our Priority" | awk '{print $8}')
        if [ "$clagrole" = "secondary" ]
        then
                #List all the interfaces below to bring down when clag peerbond goes down.
            for interface in swp3 bond0
            do
                echo "bringing down : $interface"
                ip link set $interface down
            done
        fi
    ;;
esac
 
cumulus@switch-1$ sudo cat /etc/default/ifplugd
# This file may be changed either manually or by running dpkg-reconfigure.
#
# N.B.: dpkg-reconfigure deletes everything from this file except for
# the assignments to variables INTERFACES, HOTPLUG_INTERFACES, ARGS and
# SUSPEND_ACTION.  When run it uses the current values of those variables
# as their default values, thus preserving the administrator's changes.
#
# This file is sourced by both the init script /etc/init.d/ifplugd and
# the udev script /lib/udev/ifplugd.agent to give default values.
# The init script starts ifplugd for all interfaces listed in
# INTERFACES, and the udev script starts ifplugd for all interfaces
# listed in HOTPLUG_INTERFACES. The special value all starts one
# ifplugd for all interfaces being present.
INTERFACES="peerlink"
HOTPLUG_INTERFACES=""
ARGS="-q -f -u0 -d10 -w -I"
SUSPEND_ACTION="stop"
cumulus@switch-1$
 
Same ifplugd config on the other switch
 
 
 
OUTPUT
 
cumulus@switch-1$ sudo clagctl
The peer is alive
     Our Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
    Peer Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.2
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94
 
cumulus@switch-1$
 
cumulus@switch-1$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:89:f5:2d brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master peerlink state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:42:28:ce brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master peerlink state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:42:28:ce brd ff:ff:ff:ff:ff:ff
5: swp3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:0d:f9:d7 brd ff:ff:ff:ff:ff:ff
6: swp4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:5f:0e:f1 brd ff:ff:ff:ff:ff:ff
7: swp5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:cb:ab:e4 brd ff:ff:ff:ff:ff:ff
30: peerlink: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:42:28:ce brd ff:ff:ff:ff:ff:ff
31: peerlink.4094@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:42:28:ce brd ff:ff:ff:ff:ff:ff
32: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT
    link/ether 08:00:27:0d:f9:d7 brd ff:ff:ff:ff:ff:ff
33: bond0.10@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br10 state UP mode DEFAULT
    link/ether 08:00:27:0d:f9:d7 brd ff:ff:ff:ff:ff:ff
34: peerlink.10@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br10 state UP mode DEFAULT
    link/ether 08:00:27:42:28:ce brd ff:ff:ff:ff:ff:ff
35: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:0d:f9:d7 brd ff:ff:ff:ff:ff:ff
36: br10-v0@br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 44:38:39:00:00:10 brd ff:ff:ff:ff:ff:ff
cumulus@switch-1$
 
cumulus@switch-2$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:38:d1:e9 brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master peerlink state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:ad:87:f4 brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master peerlink state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:ad:87:f4 brd ff:ff:ff:ff:ff:ff
5: swp3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:02:5a:c5 brd ff:ff:ff:ff:ff:ff
6: swp4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:69:1c:5a brd ff:ff:ff:ff:ff:ff
7: swp5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:f8:0e:02 brd ff:ff:ff:ff:ff:ff
39: peerlink: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:ad:87:f4 brd ff:ff:ff:ff:ff:ff
40: peerlink.4094@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:ad:87:f4 brd ff:ff:ff:ff:ff:ff
41: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT
    link/ether 08:00:27:02:5a:c5 brd ff:ff:ff:ff:ff:ff
42: bond0.10@bond0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br10 state LOWERLAYERDOWN mode DEFAULT
    link/ether 08:00:27:02:5a:c5 brd ff:ff:ff:ff:ff:ff
43: peerlink.10@peerlink: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br10 state UP mode DEFAULT
    link/ether 08:00:27:ad:87:f4 brd ff:ff:ff:ff:ff:ff
44: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:02:5a:c5 brd ff:ff:ff:ff:ff:ff
45: br10-v0@br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 44:38:39:00:00:10 brd ff:ff:ff:ff:ff:ff
cumulus@switch-2$
 
 
cumulus@leaf-1$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:b9:bd:83 brd ff:ff:ff:ff:ff:ff
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master spine1-2 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master spine1-2 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
5: swp3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:5e:9d:5e brd ff:ff:ff:ff:ff:ff
6: swp4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:81:b3:30 brd ff:ff:ff:ff:ff:ff
7: swp5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000
    link/ether 08:00:27:f2:af:84 brd ff:ff:ff:ff:ff:ff
20: spine1-2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
21: spine1-2.10@spine1-2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br10 state UP mode DEFAULT
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
22: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
cumulus@leaf-1$
Photo of Hemant Sharma

Hemant Sharma

  • 134 Points 100 badge 2x thumb
  • anxious

Posted 3 years ago

  • 1
Photo of Scott Suehle

Scott Suehle, Alum

  • 3,772 Points 3k badge 2x thumb
Original ask from GNS3 blog post.

ISSUE

The ping works only when the CLAG is up, as soon as I either reboot the switch-1 or shutdown all the interfaces, it stops working.

I was trying to get it working for a couple of days but with your reply , I thought maybe it was because I was not running ifplug, but it is still not working.

 

Kindly help and let me know if you need any other output

Thanks

Photo of Scott Emery

Scott Emery, Official Rep

  • 980 Points 500 badge 2x thumb
Hemant,

From the information you've provided I can't give you the solution. Everything looks like it is configured correctly. But I do have some suggestions.
  1.  Turn off the ifplugd stuff. That isn't really needed at this point, and may be causing unexpected problems. Once you get the basics working, you can turn this back on. In any case, I don't think you want to react to the peerlink interface. This is usually used when you have uplinks that you want to monitor and then make the downlinks follow the status of the uplinks. Since you don't have uplinks in this config, I don't think you want to use ifplugd.
  2. The crux of your issue is the link state of bond0 on switch-2. It is down, but should be up. If you can get that interface to be up, I think your system will work properly. The first thing to check is lldpctl, that will tell you what's at the other end of each swp link. Make sure everything is connected properly. It's possible that one of your links is not connected to what you think it is.
  3. If everything is connected properly, then check /proc/net/bonding/bond0. This will tell you what the bonding driver thinks, especially the 802.3ad status.
  4. And finally, you can run clagctl -v. That show more detailed information about what clag thinks of the state of the system(s).
Photo of Scott Emery

Scott Emery, Official Rep

  • 980 Points 500 badge 2x thumb
Hemant,

Thinking about your issue some more reminded me of a problem I had with VirtualBox. I'm not a GNS3 expert. As a matter of fact, I don't know anything about it. But perhaps by describing the issue I had with VirtualBox, you can translate that to GNS3.

The problem I had was with the MAC addresses of interfaces in bonds, swp1 and swp2 in spine1-2 on leaf-1 in your example. When Linux forms a bond, it assigns the same MAC address to all interfaces in the bond. The first interface added to a bond is left alone and is used as the MAC address of the bond. This is swp1 in your example. When the second interface is added to the bond its MAC address is changed to the MAC address of the bond, which is the MAC address of the first interface added to the bond. So in your example, swp2's MAC address is changed to the same value as swp1's MAC address. You can see this in the "ip link show" output for leaf-1. Notice that all of the MAC addresses are the same for swp1, swp2, and spine1-2:
3: swp1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master spine1-2 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master spine1-2 state UP mode DEFAULT qlen 1000
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
...
20: spine1-2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
    link/ether 08:00:27:e3:a4:b1 brd ff:ff:ff:ff:ff:ff
The problem I ran into on VirtualBox is that when the kernel changes the MAC address of swp2 it does not propagate to VirtualBox. VirtualBox still thinks the NIC is using the original MAC address, and so it filters packets not matching the original MAC address. For this reason, the LACP protocol doesn't work on swp2 and swp2 is not an active member of the spine1-2 bond. You can verify if this is the case by looking in the /proc/net/bonding/spine1-2 file on leaf-1. Does it show two ports in the active aggregator, or just one?
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
So, how do you fix this? What I did was kind of a hack, but it works. I set the MAC addresses of the swp1 and swp2 NICs in VirtualBox to the same value. That way, when Linux changes the MAC address of swp2 to be equal to swp1 it's really doing nothing, because the MAC address of swp2 is already equal to swp1. And VirtualBox will receive packets destined to this MAC on the swp2 NIC because it is the same value as what's in the Linux kernel. You could get into trouble if you change the configuration of leaf-1 (for example not putting swp1 and swp2 in the same bond) and not changing the VirtualBox MAC address assignments.

So, if you see the number of ports in the active aggregator for bond spine1-2 is 1 instead of 2, try setting the MAC addresses of the NICs for swp1 and swp2 to the same value in GNS3.

Scott
Photo of Hemant Sharma

Hemant Sharma

  • 134 Points 100 badge 2x thumb
Hi Scott,

Thanks for investigating.

I am also running Cumulus VX in virtualbox but using GNS3 for connecting the nodes to each other.

I tried with setting permanent MAC on leaf-1 but no luck. I am not sure as to what the issue is, because CLAG peerlink is also a bond and it comes up and running, every time.

The issue is only with the bond on leaf-1 node.

NOTE: Some time back when I was configuring VRR with CLAG for the first time, the bond came up on leaf-1. I ran a ping test from leaf to virtual address and then shutdown the primary node, the ping was still running.
Some other work came up and I shut it down and deleted the topology, as I thought I'll get it up again but since then it is not working.

The output of "sudo clagctl" was also showing the bond0 interfaces on switch-1 and switch-2 that time, which is not the case now.

cumulus@leaf-1$ sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    swp1, via: LLDP, RID: 5, Time: 0 day, 00:00:14
  Chassis:
    ChassisID:    mac 08:00:27:89:f5:2d
    SysName:      switch-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, off
    Capability:   Router, on
  Port:
    PortID:       ifname swp3
    PortDescr:    swp3
-------------------------------------------------------------------------------
cumulus@leaf-1$ sudo cat /proc/net/bonding/spine1-2
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 1
Aggregator selection policy (ad_select): stable
System Identification: 65535 08:00:27:e3:a4:b1
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 1
        Actor Key: 17
        Partner Key: 17
        Partner Mac Address: 44:38:39:ff:40:94
LACP Bypass Info:
        Allowed: 0
        Timeout: 0
        All-active: 0

Slave Interface: swp2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e3:a4:b1
Aggregator ID: 1
LACP bypass priority: 0
Slave queue ID: 0

Slave Interface: swp1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e3:a4:b1
Aggregator ID: 2
LACP bypass priority: 0
Slave queue ID: 0
cumulus@leaf-1$



cumulus@switch-1$ sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    swp1, via: LLDP, RID: 8, Time: 0 day, 00:02:08
  Chassis:
    ChassisID:    mac 08:00:27:38:d1:e9
    SysName:      switch-2
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp1
    PortDescr:    swp1
-------------------------------------------------------------------------------
Interface:    swp2, via: LLDP, RID: 8, Time: 0 day, 00:02:08
  Chassis:
    ChassisID:    mac 08:00:27:38:d1:e9
    SysName:      switch-2
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp2
    PortDescr:    swp2
-------------------------------------------------------------------------------
Interface:    swp3, via: LLDP, RID: 7, Time: 0 day, 00:02:14
  Chassis:
    ChassisID:    mac 08:00:27:b9:bd:83
    SysName:      leaf-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp1
    PortDescr:    swp1
-------------------------------------------------------------------------------
cumulus@switch-1$ sudo ptmctl
-----------------------------------------
port  cbl     BFD     BFD   BFD    BFD
      status  status  peer  local  type
-----------------------------------------
swp1  pass    N/A     N/A   N/A    N/A
swp2  pass    N/A     N/A   N/A    N/A
swp3  pass    N/A     N/A   N/A    N/A
cumulus@switch-1$




cumulus@switch-2$ sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    swp1, via: LLDP, RID: 7, Time: 0 day, 00:02:18
  Chassis:
    ChassisID:    mac 08:00:27:89:f5:2d
    SysName:      switch-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp1
    PortDescr:    swp1
-------------------------------------------------------------------------------
Interface:    swp2, via: LLDP, RID: 7, Time: 0 day, 00:02:18
  Chassis:
    ChassisID:    mac 08:00:27:89:f5:2d
    SysName:      switch-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp2
    PortDescr:    swp2
-------------------------------------------------------------------------------
Interface:    swp3, via: LLDP, RID: 8, Time: 0 day, 00:02:35
  Chassis:
    ChassisID:    mac 08:00:27:b9:bd:83
    SysName:      leaf-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp2
    PortDescr:    swp2
-------------------------------------------------------------------------------
cumulus@switch-2$ sudo ptmctl
-----------------------------------------
port  cbl     BFD     BFD   BFD    BFD
      status  status  peer  local  type
-----------------------------------------
swp1  pass    N/A     N/A   N/A    N/A
swp2  pass    N/A     N/A   N/A    N/A
swp3  pass    N/A     N/A   N/A    N/A
cumulus@switch-2$



cumulus@leaf-1$ sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    swp1, via: LLDP, RID: 5, Time: 0 day, 00:02:39
  Chassis:
    ChassisID:    mac 08:00:27:89:f5:2d
    SysName:      switch-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp3
    PortDescr:    swp3
-------------------------------------------------------------------------------
cumulus@leaf-1$ sudo ptmctl
-----------------------------------------
port  cbl     BFD     BFD   BFD    BFD
      status  status  peer  local  type
-----------------------------------------
swp1  pass    N/A     N/A   N/A    N/A
swp2  N/A     N/A     N/A   N/A    N/A
cumulus@leaf-1$



cumulus@switch-1$ sudo clagctl -v
The peer is alive
     Our Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
    Peer Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.2
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

Our LACP Information
Our Interface      Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        -

Peer LACP Information
Peer Interface     Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              00:00:00:00:00:00   None          1         down      -

Backup info:
IP: -; State: inactive; Role: secondary
Peer priority and id: -; Peer role: -

Our Bond           CLAG Id   Conflicts              Proto-Down
----------------   -------   --------------------   ---------------
bond0              1         -                      -

Our Interface      Dynamic MAC         VLAN Id
----------------   -----------------   -------
bond0              08:00:27:e3:a4:b1   10

Peer Interface     Dynamic MAC         VLAN Id
----------------   -----------------   -------

Our Multicast Group      Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Peer Multicast Group     Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Our Router Port    Device             Age
----------------   ----------------   ---

Peer Router Port   Device             Age
----------------   ----------------   ---

Our VLAN Information
Our Interface      VLAN Id
----------------   -------

Peer VLAN Information
Peer Interface     VLAN Id
----------------   -------

cumulus@switch-1$




cumulus@switch-2$ sudo clagctl -v
The peer is alive
    Peer Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
     Our Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.1
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

Our LACP Information
Our Interface      Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              00:00:00:00:00:00   None          1         down      -

Peer LACP Information
Peer Interface     Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        -

Backup info:
IP: -; State: inactive; Role: secondary
Peer priority and id: -; Peer role: -

Our Bond           CLAG Id   Conflicts              Proto-Down
----------------   -------   --------------------   ---------------
bond0              1         -                      -

Our Interface      Dynamic MAC         VLAN Id
----------------   -----------------   -------

Peer Interface     Dynamic MAC         VLAN Id
----------------   -----------------   -------
peerlink           08:00:27:e3:a4:b1   10

Our Multicast Group      Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Peer Multicast Group     Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Our Router Port    Device             Age
----------------   ----------------   ---

Peer Router Port   Device             Age
----------------   ----------------   ---

Our VLAN Information
Our Interface      VLAN Id
----------------   -------

Peer VLAN Information
Peer Interface     VLAN Id
----------------   -------

cumulus@switch-2$

(Edited)
Photo of Hemant Sharma

Hemant Sharma

  • 134 Points 100 badge 2x thumb
Just got lucky again after recreating the topology in GNS3 and rebooting the nodes a couple of times

1) leaf-1 had discovered both the LLDP neighbors
2) CLAG was down on both switch-1 and switch-2
3) Rebooted switch-1 and switch-2
4) Voila, CLAG is up with bond0

It appears that the issue revolves around the mac address & bond setup, this time, the aggregator ID was same for both the slave interfaces on leaf-1, which is why it was working.

Output from the working solution:

cumulus@leaf-1$ sudo lldpctl
-------------------------------------------------------------------------------
LLDP neighbors:
-------------------------------------------------------------------------------
Interface:    swp1, via: LLDP, RID: 2, Time: 0 day, 00:04:09
  Chassis:
    ChassisID:    mac 08:00:27:89:f5:2d
    SysName:      switch-1
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       1.1.1.1
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp3
    PortDescr:    swp3
-------------------------------------------------------------------------------
Interface:    swp2, via: LLDP, RID: 1, Time: 0 day, 00:06:53
  Chassis:
    ChassisID:    mac 08:00:27:38:d1:e9
    SysName:      switch-2
    SysDescr:     Cumulus Linux version 2.5.3 running on innotek GmbH VirtualBox
    MgmtIP:       10.0.2.15
    Capability:   Bridge, on
    Capability:   Router, on
  Port:
    PortID:       ifname swp3
    PortDescr:    swp3
-------------------------------------------------------------------------------
cumulus@leaf-1$ sudo cat /proc/net/bonding/spine1-2
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Min links: 1
Aggregator selection policy (ad_select): stable
System Identification: 65535 08:00:27:e3:a4:b1
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 2
        Actor Key: 17
        Partner Key: 17
        Partner Mac Address: 44:38:39:ff:40:94
LACP Bypass Info:
        Allowed: 0
        Timeout: 0
        All-active: 0

Slave Interface: swp2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:e3:a4:b1
Aggregator ID: 2
LACP bypass priority: 0
Slave queue ID: 0

Slave Interface: swp1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:0f:b8:6a
Aggregator ID: 2
LACP bypass priority: 0
Slave queue ID: 0
cumulus@leaf-1$


cumulus@switch-1$ sudo clagctl
The peer is alive
     Our Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
    Peer Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.2
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

       Dual Attached Ports
Our Interface      Peer Interface     CLAG Id
----------------   ----------------   -------
           bond0   bond0              1

cumulus@switch-1$ sudo clagctl -v
The peer is alive
     Our Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
    Peer Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.2
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

       Dual Attached Ports
Our Interface      Peer Interface     CLAG Id
----------------   ----------------   -------
           bond0   bond0              1

Our LACP Information
Our Interface      Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        D

Peer LACP Information
Peer Interface     Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        D

Backup info:
IP: -; State: inactive; Role: secondary
Peer priority and id: -; Peer role: -

Our Bond           CLAG Id   Conflicts              Proto-Down
----------------   -------   --------------------   ---------------
bond0              1         -                      -

Our Interface      Dynamic MAC         VLAN Id
----------------   -----------------   -------

Peer Interface     Dynamic MAC         VLAN Id
----------------   -----------------   -------
bond0              08:00:27:e3:a4:b1   10

Our Multicast Group      Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Peer Multicast Group     Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Our Router Port    Device             Age
----------------   ----------------   ---

Peer Router Port   Device             Age
----------------   ----------------   ---

Our VLAN Information
Our Interface      VLAN Id
----------------   -------
bond0              10

Peer VLAN Information
Peer Interface     VLAN Id
----------------   -------
bond0              10

cumulus@switch-1$



cumulus@switch-2$ sudo clagctl
The peer is alive
    Peer Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
     Our Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.1
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

       Dual Attached Ports
Our Interface      Peer Interface     CLAG Id
----------------   ----------------   -------
           bond0   bond0              1

cumulus@switch-2$ sudo clagctl -v
The peer is alive
    Peer Priority, ID, and Role: 32768 08:00:27:42:28:ce primary
     Our Priority, ID, and Role: 32768 08:00:27:ad:87:f4 secondary
          Peer Interface and IP: peerlink.4094 169.254.1.1
                      Backup IP: - (inactive)
                     System MAC: 44:38:39:ff:40:94

       Dual Attached Ports
Our Interface      Peer Interface     CLAG Id
----------------   ----------------   -------
           bond0   bond0              1

Our LACP Information
Our Interface      Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        D

Peer LACP Information
Peer Interface     Partner MAC         CIST PortId   CLAG Id   Oper St   Flags
----------------   -----------------   -----------   -------   -------   -----
bond0              08:00:27:e3:a4:b1   None          1         up        D

Backup info:
IP: -; State: inactive; Role: secondary
Peer priority and id: -; Peer role: -

Our Bond           CLAG Id   Conflicts              Proto-Down
----------------   -------   --------------------   ---------------
bond0              1         -                      -

Our Interface      Dynamic MAC         VLAN Id
----------------   -----------------   -------
bond0              08:00:27:e3:a4:b1   10

Peer Interface     Dynamic MAC         VLAN Id
----------------   -----------------   -------

Our Multicast Group      Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Peer Multicast Group     Port               VLAN Id   Device             Age
----------------------   ----------------   -------   ----------------   ---

Our Router Port    Device             Age
----------------   ----------------   ---

Peer Router Port   Device             Age
----------------   ----------------   ---

Our VLAN Information
Our Interface      VLAN Id
----------------   -------
bond0              10

Peer VLAN Information
Peer Interface     VLAN Id
----------------   -------
bond0              10

cumulus@switch-2$




Redundancy Test:


Ping test from leaf-1
cumulus@leaf-1$ ping 1.1.1.254
PING 1.1.1.254 (1.1.1.254) 56(84) bytes of data.
64 bytes from 1.1.1.254: icmp_req=1 ttl=64 time=0.961 ms
64 bytes from 1.1.1.254: icmp_req=2 ttl=64 time=1.92 ms
64 bytes from 1.1.1.254: icmp_req=3 ttl=64 time=1.26 ms
64 bytes from 1.1.1.254: icmp_req=4 ttl=64 time=2.19 ms
64 bytes from 1.1.1.254: icmp_req=5 ttl=64 time=2.19 ms
64 bytes from 1.1.1.254: icmp_req=6 ttl=64 time=0.966 ms
64 bytes from 1.1.1.254: icmp_req=7 ttl=64 time=1.40 ms
64 bytes from 1.1.1.254: icmp_req=8 ttl=64 time=16.6 ms
64 bytes from 1.1.1.254: icmp_req=9 ttl=64 time=1.24 ms
64 bytes from 1.1.1.254: icmp_req=10 ttl=64 time=1.11 ms
64 bytes from 1.1.1.254: icmp_req=11 ttl=64 time=9.33 ms
64 bytes from 1.1.1.254: icmp_req=12 ttl=64 time=1.13 ms
64 bytes from 1.1.1.254: icmp_req=13 ttl=64 time=1.52 ms
64 bytes from 1.1.1.254: icmp_req=14 ttl=64 time=1.15 ms
64 bytes from 1.1.1.254: icmp_req=15 ttl=64 time=1.18 ms
64 bytes from 1.1.1.254: icmp_req=16 ttl=64 time=2.23 ms
64 bytes from 1.1.1.254: icmp_req=17 ttl=64 time=0.916 ms
64 bytes from 1.1.1.254: icmp_req=21 ttl=64 time=1.36 ms
64 bytes from 1.1.1.254: icmp_req=22 ttl=64 time=2.71 ms
64 bytes from 1.1.1.254: icmp_req=23 ttl=64 time=0.765 ms
64 bytes from 1.1.1.254: icmp_req=24 ttl=64 time=0.910 ms
64 bytes from 1.1.1.254: icmp_req=25 ttl=64 time=1.56 ms
64 bytes from 1.1.1.254: icmp_req=26 ttl=64 time=2.32 ms
64 bytes from 1.1.1.254: icmp_req=27 ttl=64 time=1.55 ms
64 bytes from 1.1.1.254: icmp_req=28 ttl=64 time=1.60 ms
64 bytes from 1.1.1.254: icmp_req=29 ttl=64 time=0.967 ms
64 bytes from 1.1.1.254: icmp_req=30 ttl=64 time=2.05 ms
64 bytes from 1.1.1.254: icmp_req=31 ttl=64 time=1.57 ms
^C
--- 1.1.1.254 ping statistics ---
31 packets transmitted, 28 received, 9% packet loss, time 30077ms
rtt min/avg/max/mdev = 0.765/2.311/16.618/3.153 ms
cumulus@leaf-1$


The traffic hits on switch-2

cumulus@switch-2$ sudo tcpdump -i swp3
tcpdump: WARNING: swp3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp3, link-type EN10MB (Ethernet), capture size 65535 bytes
03:58:12.392548 LACPv1, length 110
03:58:12.592317 LLDP, name leaf-1, length 120
03:58:13.011626 LACPv1, length 110
03:58:13.292955 LACPv1, length 110
03:58:13.920234 LACPv1, length 110
03:58:14.492368 LACPv1, length 110
03:58:14.820373 LACPv1, length 110
03:58:15.106372 IP 1.1.1.100 > 1.1.1.254: ICMP echo request, id 4446, seq 1, length 64
03:58:15.106485 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 1, length 64
03:58:15.392436 LACPv1, length 110
03:58:16.019601 LACPv1, length 110
03:58:16.108195 IP 1.1.1.100 > 1.1.1.254: ICMP echo request, id 4446, seq 2, length 64
03:58:16.108380 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 2, length 64
03:58:16.292002 LACPv1, length 110
03:58:16.920055 LACPv1, length 110
03:58:17.113201 IP 1.1.1.100 > 1.1.1.254: ICMP echo request, id 4446, seq 3, length 64
03:58:17.113308 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 3, length 64

<..snipped..>
03:58:29.136573 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 15, length 64
03:58:29.495013 LACPv1, length 110
^C
74 packets captured
74 packets received by filter
0 packets dropped by kernel
cumulus@switch-2$ sudo ifdown -a

Traffic went to switch-1

cumulus@switch-1$ sudo tcpdump -i swp3
tcpdump: WARNING: swp3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on swp3, link-type EN10MB (Ethernet), capture size 65535 bytes
03:58:18.684021 LACPv1, length 110
03:58:19.056206 LACPv1, length 110
<..snipped..>

03:58:35.149549 IP 1.1.1.100 > 1.1.1.254: ICMP echo request, id 4446, seq 21, length 64
03:58:35.150089 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 21, length 64
03:58:35.316004 LACPv1, length 110
03:58:35.785240 LACPv1, length 110
03:58:36.152189 IP 1.1.1.100 > 1.1.1.254: ICMP echo request, id 4446, seq 22, length 64
03:58:36.152549 IP 1.1.1.254 > 1.1.1.100: ICMP echo reply, id 4446, seq 22, length 64
03:58:36.216567 LACPv1, length 110
03:58:36.681621 LACPv1, length 110
<..snipped..>
cumulus@switch-1$


Just a few packet loss and traffic switches over to the redundant link.

Question: VRR is active-active, so how is it decided as to which switch is going to be active for which leaf?

Question: In the above scenario, switch-1 is CLAG Primary but ping traffic went to switch-2, why?

Question: VRR also appears to be preemptive, the traffic falls back to switch-2 as soon as I bring its interfaces up, can we also influence the active role with any priority command?

Thanks !!


NOTE: I'll give few more tries by deleting and recreating the topology to find out how exactly this came up.
Photo of Scott Emery

Scott Emery, Official Rep

  • 980 Points 500 badge 2x thumb
Hemant,

I'm glad to see that you got everything working. But I sure wish that we could have root-caused your issue.

To answer you questions:

Question: VRR is active-active, so how is it decided as to which switch is going to be active for which leaf?
It is the host (leaf-1 in your case) which decides how to distribute packets on a bond. Usually some sort of hash is done on the MAC/IP/UDP/TCP header fields, which guarantees that all the packets for a single flow will travel on the same path. But this isn't required. The hosts are free to distribute the traffic they are sending amongst the members of the bond in whatever way they want. When the packets arrive at the switch they will simply be routed. It doesn't matter which switch gets the packet, they are routed from there in the same way.

When a switch receives a packet from upstream that it routes to the host, the packet will be sent directly to the host, never through the peer link and then to the host (unless the host is singly connected, and then the only way to reach the host may be through the peer link). The upstream device(s) will have some sort of ECMP to the switches, which will have its own mechanism for distributing the traffic between the two switches.

Question: In the above scenario, switch-1 is CLAG Primary but ping traffic went to switch-2, why?
The primary/secondary role of the two CLAG switches has nothing to do with the distribution of traffic to or from the switches. They are both active and both can send and receive traffic on their interfaces. The primary/secondary roles are used when the peer link between the switches goes down. In that case the two CLAG switches are operating in what we call "split-brain", where they are both running CLAG, but can't talk to each other. In "split-brain" the primary will continue to operate normally, and the secondary will shut down its CLAG interfaces.

Question: VRR also appears to be preemptive, the traffic falls back to switch-2 as soon as I bring its interfaces up, can we also influence the active role with any priority command?
As far as the distribution between the two interfaces in a bond, see the answer to your first question. Remember that the host (leaf-1) is simply sending packets through the bond. It is the bonding driver which determines how to distribute the traffic through the interfaces in the bond. The hosts may not be running Cumulus Linux. They could be running Ubuntu, Redhat, Windows, Solaris, anything really. So Cumulus Linux doesn't have control of how those hosts distribute traffic to the members of the bond.

As I said above, the role doesn't change the distribution of traffic. It's primarily there to handle the split-brain case. But you can set/change the roles by using the "clagd-priority <N>" keyword in /etc/network/interfaces or with the "clagctl priority <N>" command on the command line, where <N> is the priority, between 0 and 65535. The switch with the lowest priority value will be primary.

Once again, I'm glad you got everything up and running. Let me know if you have more questions.

P.S. We just released VX 2.5.4. Give it a try!
Photo of Hemant Sharma

Hemant Sharma

  • 134 Points 100 badge 2x thumb
Thanks Scott!!

Yes, it is a bond and would run hashing algorithm to determine the physical link to forward the traffic.