cumulus@leaf$ ip rule show | grep mark
100: from all fwmark 0xfe lookup main
it is installed as part of cl-mgmtvrf do_start()
sudo ip rule del fwmark 254 lookup main prio 100 2>/dev/null 1>/dev/null
am i suppose to see a entry in mangle table mark some traffic to 0xfe?
I checked mangle table, it is empty, what is this entry used for?
You are correct that rule is installed as part of cl-mgmtvrf package in the /etc/init.d/cl-mgmtvrf script. It is an experimental feature that allows the separation of the main routing table and the routing table for the management interface eth0. Similar to VRF but I wouldn't call it such from a traditional networking VRF definition.
Once the feature is enabled it creates another Linux routing table number 252 named mgmt. (To see the full list check: /etc/iproute2/rt_tables) The main table is number 254 hence why you see the fwmark to (0xfe) 254 for default lookups to use the main routing table.
The rule after the one you asked about (101: from all iif eth0 lookup mgmt) sets traffic lookups for eth0 to be in the mgmt table. So if you were to issue a ping by default it will use the main routing table if you specify the interface such as ping -I eth0 it will use the mgmt routing table.
To answer your second question you have to dig into the feature a little more in the execution script /usr/sbin/cl-mgmtvrf. There is a list of sys control settings to make the defaults happen for IPv4 and IPv6. So because of changing the system defaults you will not see any mangle rules associated with this.
Documentation for cl-mgmtvrf
Hope this helps.
however I looked at enable() in /usr/sbin/cl-mgmtvrf, it didn't mention anything about 0xfe marking, which of following mark the pkt?
cumulus@leaf$ cat /etc/sysctl.d/cl-mgmtvrf.conf
what I thought about this rule is, when pkt is marked by firewall as 0xfe, they should be looked in main routing table first, but who is doing marking? if not mangle table.
In newer Linux kernel versions and coming in Cumulus Linux 3.0 we have contributed VRF support into the kernel where the inner workings of this feature will change. The user experience will remain the same but all of these details will change.
Is there a specific use case that you are trying to solve where cl-mgmtvrf is needed or do you need to modify the ip rules? I might be able to answer the use case question better because like I mentioned all of these internal workings are going to change.