arp ageing test

  • 1
  • Question
  • Updated 10 months ago

According to the page below link directing to, there is no absolute timer for the arp ageing, rather it is a random timer targeting in the range interval between [base_reachable_time_ms/2] and [3*base_reachable_time_ms/2].


And base_reachable_time_ms can be altered.  

 

https://support.cumulusnetworks.com/hc/en-us/articles/202012933-Changing-ARP-timers-in-Cumulus-Linux

 

According to the page, I changed the default base_reachable_time_ms on Cumulus 3.3.2 to 120000 for simplicity and saving time.

 

cumulus@switch:~$ sudo bash -c "echo 1200000 > /proc/sys/net/ipv4/neigh/default/base_reachable_time_ms"
cumulus@switch:~$ sudo bash -c "for int in /proc/sys/net/ipv4/neigh/swp*; do echo 1200000 > $int/base_reachable_time_ms; done"\

 

I was expecting to see the status turning to stale less than 30 mins per the article.  But after 50 mins, I still see the same ip neigh show table. 

 

Before:

root@dut-1:/home/cumulus# ip neighbor show

23.23.23.3 dev swp2 lladdr 00:0c:29:ae:24:35 REACHABLE

12.12.12.3 dev swp1 lladdr 00:0c:29:ae:24:2b REACHABLE   =============this is a port ip address connected to the interface swp1, and I remove ip address 12.12.12.3 from the remote host and waiting for the arp item aging out.

192.168.1.1 dev eth0 lladdr c0:ff:d4:93:0f:2f REACHABLE

12.12.12.1 dev swp1 lladdr 00:0c:29:bc:62:40 REACHABLE

192.168.1.91 dev eth0 lladdr e4:a7:a0:a2:ac:bf REACHABLE

 

 50 minutes later:

root@dut-1:/home/cumulus# ip -s neighbor show

23.23.23.3 dev swp2 lladdr 00:0c:29:ae:24:35 ref 1 used 17493/1/33 probes 4 REACHABLE

12.12.12.3 dev swp1 lladdr 00:0c:29:ae:24:2b ref 1 used 2470/2470/3424 probes 0 REACHABLE  ====> this item is still existing and REACHABLE.

192.168.1.1 dev eth0 lladdr c0:ff:d4:93:0f:2f ref 1 used 14764/1065/235 probes 4 REACHABLE

12.12.12.1 dev swp1 lladdr 00:0c:29:bc:62:40 ref 1 used 20875/1/133 probes 4 REACHABLE

192.168.1.91 dev eth0 lladdr e4:a7:a0:a2:ac:bf ref 1 used 14966/0/22 probes 4 REACHABLE

root@dut-1:/home/cumulus#

 

Please help to clarify this.

Paul
Photo of Lei Zhang

Lei Zhang

  • 172 Points 100 badge 2x thumb

Posted 10 months ago

  • 1
Photo of Scott Emery

Scott Emery, Official Rep

  • 980 Points 500 badge 2x thumb
Paul,

Your understanding is perfectly correct, except for one thing: garbage collection. The kernel will not advance the state of an ARP entry unless there are at least gc_thresh1 entries in the table. I guess the thinking is that if there are so few entries, it's not worth wasting time trying to optimize the ARP table. So, your entry will not move from REACHABLE to STALE in this case because you have only 5 ARP entries and the default value of gc_thresh1 is 128.

This makes it difficult to do testing like you are trying to do. But you can either change the value of gc_thresh1 or add more entries to the ARP table. To change gc_thresh1 you can:

echo "4" > net.ipv4.neigh.default.gc_thresh1

to set it to 4. The meanings of the gc_threshX sysctls are:

gc_thresh1
The minimum number of entries to keep in the ARP cache. The garbage collector will not run if there are fewer than this number of entries in the cache.

gc_thresh2
The soft maximum number of entries to keep in the ARP cache. The garbage collector will allow the number of entries to exceed this for 5 seconds before collection will be performed.

gc_thresh3
The hard maximum number of entries to keep in the ARP cache. The garbage collector will always run if there are more than this number of entries in the cache.

Scott
Photo of Eric Pulvino

Eric Pulvino, Official Rep

  • 4,082 Points 4k badge 2x thumb
Great info! We should add this to documentation!
Photo of Pete B

Pete B, Official Rep

  • 2,786 Points 2k badge 2x thumb
Good idea! I updated the KB article on changing ARP timers.
Photo of Lei Zhang

Lei Zhang

  • 172 Points 100 badge 2x thumb
Hi Sccot,

Thanks for your reply.

I follow your step and come up with the negtive result.  it is still not going to stale.

root@sw248:~# cat /proc/sys/net/ipv4/neigh/default/base_reachable_time_ms
110000
root@sw248:~# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
4
root@sw248:~# cat /proc/sys/net/ipv4/neigh/eth0/gc_stale_time
60
root@sw248:~# cat /proc/sys/net/ipv4/neigh/eth0/base_reachable_time_ms
110000
root@sw248:~# cat  /proc/sys/net/ipv4/neigh/default/base_reachable_time_ms
110000


root@sw248:~# date
Tue Jul 18 03:07:04 UTC 2017
root@sw248:~# ip neighbor show
10.12.69.2 dev eth0 lladdr 98:e7:f4:f2:1f:ba REACHABLE
1.0.9.2 dev swp8 lladdr 7c:fe:90:f2:b2:7b REACHABLE
1.0.20.2 dev swp19 lladdr 7c:fe:90:f2:b2:6e REACHABLE
10.12.69.247 dev eth0 lladdr 24:8a:07:09:77:14 REACHABLE ===> expected to be stale after (55~165s)
10.12.69.1 dev eth0 lladdr b8:88:e3:f8:e8:b7 REACHABLE
1.0.4.2 dev swp3 lladdr 7c:fe:90:f2:b2:7e REACHABLE
1.0.8.2 dev swp7 lladdr 7c:fe:90:f2:b2:7a REACHABLE
1.0.6.2 dev swp5 lladdr 7c:fe:90:f2:b2:78 REACHABLE
10.12.69.77 dev eth0 lladdr 98:e7:f4:32:74:e2 REACHABLE
1.0.7.2 dev swp6 lladdr 7c:fe:90:f2:b2:79 REACHABLE
1.0.2.2 dev swp1 lladdr 7c:fe:90:f2:b2:7c REACHABLE
1.0.19.2 dev swp18 lladdr 7c:fe:90:f2:b2:6d REACHABLE
1.0.3.2 dev swp2 lladdr 7c:fe:90:f2:b2:7d REACHABLE
1.0.12.2 dev swp11 lladdr 7c:fe:90:f2:b2:76 REACHABLE
1.0.10.2 dev swp9 lladdr 7c:fe:90:f2:b2:74 REACHABLE
1.0.5.2 dev swp4 lladdr 7c:fe:90:f2:b2:7f REACHABLE
1.0.14.2 dev swp13 lladdr 7c:fe:90:f2:b2:70 REACHABLE
1.0.15.2 dev swp14 lladdr 7c:fe:90:f2:b2:71 REACHABLE
1.0.17.2 dev swp16 lladdr 7c:fe:90:f2:b2:73 REACHABLE
1.0.11.2 dev swp10 lladdr 7c:fe:90:f2:b2:75 REACHABLE
1.0.16.2 dev swp15 lladdr 7c:fe:90:f2:b2:72 REACHABLE
1.0.13.2 dev swp12 lladdr 7c:fe:90:f2:b2:77 REACHABLE
1.0.21.2 dev swp20 lladdr 7c:fe:90:f2:b2:6f REACHABLE
1.0.18.2 dev swp17 lladdr 7c:fe:90:f2:b2:6c REACHABLE
fe80::268a:7ff:fe32:785e dev eth0 lladdr 24:8a:07:32:78:5e STALE
fe80::16:32f3:b743:bf2e dev eth0 lladdr b8:88:e3:f8:e8:b7 router STALE
root@sw248:~#

Few minutes later:

root@sw248:~# date
Tue Jul 18 03:12:20 UTC 2017
root@sw248:~# ip neighbor show
10.12.69.2 dev eth0 lladdr 98:e7:f4:f2:1f:ba REACHABLE
1.0.9.2 dev swp8 lladdr 7c:fe:90:f2:b2:7b REACHABLE
1.0.20.2 dev swp19 lladdr 7c:fe:90:f2:b2:6e REACHABLE
10.12.69.247 dev eth0 lladdr 24:8a:07:09:77:14 REACHABLE  ======>
10.12.69.1 dev eth0 lladdr b8:88:e3:f8:e8:b7 REACHABLE
1.0.4.2 dev swp3 lladdr 7c:fe:90:f2:b2:7e REACHABLE
1.0.8.2 dev swp7 lladdr 7c:fe:90:f2:b2:7a REACHABLE
1.0.6.2 dev swp5 lladdr 7c:fe:90:f2:b2:78 REACHABLE
10.12.69.77 dev eth0 lladdr 98:e7:f4:32:74:e2 REACHABLE
1.0.7.2 dev swp6 lladdr 7c:fe:90:f2:b2:79 REACHABLE
1.0.2.2 dev swp1 lladdr 7c:fe:90:f2:b2:7c REACHABLE
1.0.19.2 dev swp18 lladdr 7c:fe:90:f2:b2:6d REACHABLE
1.0.3.2 dev swp2 lladdr 7c:fe:90:f2:b2:7d REACHABLE
1.0.12.2 dev swp11 lladdr 7c:fe:90:f2:b2:76 REACHABLE
1.0.10.2 dev swp9 lladdr 7c:fe:90:f2:b2:74 REACHABLE
1.0.5.2 dev swp4 lladdr 7c:fe:90:f2:b2:7f REACHABLE
1.0.14.2 dev swp13 lladdr 7c:fe:90:f2:b2:70 REACHABLE
1.0.15.2 dev swp14 lladdr 7c:fe:90:f2:b2:71 REACHABLE
1.0.17.2 dev swp16 lladdr 7c:fe:90:f2:b2:73 REACHABLE
1.0.11.2 dev swp10 lladdr 7c:fe:90:f2:b2:75 REACHABLE
1.0.16.2 dev swp15 lladdr 7c:fe:90:f2:b2:72 REACHABLE
1.0.13.2 dev swp12 lladdr 7c:fe:90:f2:b2:77 REACHABLE
1.0.21.2 dev swp20 lladdr 7c:fe:90:f2:b2:6f REACHABLE
1.0.18.2 dev swp17 lladdr 7c:fe:90:f2:b2:6c REACHABLE
fe80::268a:7ff:fe32:785e dev eth0 lladdr 24:8a:07:32:78:5e STALE
fe80::16:32f3:b743:bf2e dev eth0 lladdr b8:88:e3:f8:e8:b7 router STALE
root@sw248:~#

But it failed to stale out.

MEanwhile, to I start data capture on the "10.12.69.247", seems there is no arp request from this 10.12.69.248.


root@sw247:~# tcpdump -i eth0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:52:05.207966 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:52:05.208001 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:52:05.488108 ARP, Request who-has 10.12.69.247 tell 10.12.69.1, length 46
02:52:05.488137 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:53:49.860421 ARP, Request who-has 10.12.69.247 tell 10.12.69.1, length 46
02:53:49.860467 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:53:54.709323 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:53:54.709354 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:54:41.709197 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:54:41.709231 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:55:43.208080 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:55:43.208122 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:55:58.777870 ARP, Request who-has 10.12.69.1 tell 10.12.69.77, length 46
02:55:58.956064 ARP, Request who-has 10.12.69.247 tell 10.12.69.1, length 46
02:55:58.956095 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:57:31.706097 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:57:31.706140 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
02:58:18.705554 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
02:58:18.705603 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:00:23.547574 ARP, Request who-has 10.12.69.77 tell 10.12.69.1, length 46
03:00:28.205598 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:00:28.205628 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:02:50.640190 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:03:01.849335 ARP, Request who-has 10.12.69.1 tell 10.12.69.77, length 46
03:03:05.706779 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:03:05.706818 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:03:59.638376 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:04:04.204867 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:04:04.204905 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:05:07.638811 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:05:12.209617 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:05:12.209657 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:06:13.639006 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:06:18.206485 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:06:18.206528 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:07:19.639044 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:07:24.209467 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:07:24.209505 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:08:54.717658 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.77, length 46
03:08:54.717697 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:08:59.705173 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:08:59.705203 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:09:06.167576 ARP, Request who-has 10.12.69.249 tell 10.12.69.1, length 46
03:09:24.350699 ARP, Request who-has 10.12.69.1 tell 10.12.69.77, length 46
03:09:40.216758 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.77, length 46
03:09:40.216796 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:09:49.811999 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:09:49.998572 ARP, Request who-has 10.12.69.247 tell 10.12.69.1, length 46
03:09:49.998606 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:09:50.312939 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:09:51.313306 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:09:55.202770 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:09:55.202809 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:10:08.721818 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.77, length 46
03:10:08.721858 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:10:52.309828 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:10:57.204135 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:10:57.204171 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:10:57.318606 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:10:58.301183 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:10:59.299536 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:11:05.727156 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.77, length 46
03:11:05.727195 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:11:26.293169 ARP, Request who-has 10.12.69.1 tell 10.12.69.77, length 46
03:12:00.312592 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:12:05.204518 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:12:05.204550 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:13:01.297580 ARP, Request who-has 192.168.1.100 tell 192.168.1.221, length 46
03:13:06.210835 ARP, Request who-has 10.12.69.247 (24:8a:07:09:77:14 (oui Unknown)) tell 10.12.69.2, length 46
03:13:06.210869 ARP, Reply 10.12.69.247 is-at 24:8a:07:09:77:14 (oui Unknown), length 28
03:13:15.300121 ARP, Request who-has 10.12.69.1 tell 10.12.69.77, length 46




I guess some other parameters may need to alter.

Appreciate your help.

Regards
Paul

 
Photo of Scott Emery

Scott Emery, Official Rep

  • 980 Points 500 badge 2x thumb
Paul,

Could you clarify your expectations? In the "ip neighbor show" dumps you are highlighting the 10.12.69.247 address, expecting it to go to stale, but it stays in the reachable state. The tcpdump output shows lots of ARP activity with the 10.12.69.247 address (requests and replies). Seems like that activity should keep the address in the reachable state. Why would you expect 10.12.69.247 to go stale? Also, you mention address 10.12.69.248, but I don't see it in your neighbor table or tcpdump. What was the point of mentioning address 10.12.69.248? Finally, when dumping the neighbor table it is helpful to use the "-s" option to see the timestamps.

Scott
Photo of Lei Zhang

Lei Zhang

  • 172 Points 100 badge 2x thumb
Hi Scott, Thank you for your reply.

Why would you expect 10.12.69.247 to go stale?

Arp ageing, per my experience on network boxes, should expire after the configured timeout timer. And the arp item can only be refreshed by arp response or gratuitous arp. no mater the correponding ip-mac pair alive or not later on.

For Cumulus case, it is more a linux host, you may see the arp request and reply, in fact there is no arp ping-pone between 10.12.60.247 and 10.12.60.248. This makes me beleive that the arp (10.12.60.247 +mac) did not refresh on arp table of 10.12.60.248, and it is to be ageing out per the time during the X/2 ~3X/2 period of time. But I did not see this ageing happened when the arp related interface (ip) is up in the network. Even there is no arp between the two switches.

When the arp is not pointing to any real interface with ip-mac, then the ageing happened per expected.

Hence I d like tp know if any traffic (in addition to arp) in between the two box can also inlfluce the arp timer?


Finally, when dumping the neighbor table it is helpful to use the "-s"

Yes, thank you for this small option.

I have no idea about the probe of this "-s" bring about, can you share the secret behind. I suspect something in probe may link the arp time.


I may take something wrong, thank you for your support.
 

BRs
Paul