Linux ARP reply requires FIB lookup
TL;DR: In order for Linux kernel to send an ARP reply, it must be able to find a local route with the local IP address and interface.
Imagine we have an interface:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 11:45:14:19:19:81 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/24 metric 1024 brd 10.0.0.255 scope global
valid_lft forever preferred_lft forever
By default, it comes with the following routes:
10.0.0.1/24 dev eth0 proto kernel scope link src 10.0.0.1 metric 1024
local 10.0.0.1 dev eth0 table local proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev eth0 table local proto kernel scope link src 10.0.0.1
The second one, local 10.0.0.1 dev eth0 is required to send ARP replies asking
for 10.0.0.1. The type has to be local, not unicast (default for
ip r add).
According to Miao Wang in the TUNA group chat, this is because Linux needs FIB lookup to make sure if the asked address is a local address (i.e., bound to a local interface) or not.
When an ARP request is received, Linux looks up the FIB normally - following all
policy routing rules. Although the above route is added to local table by
default, it is OK to place it in any table - as long as the route policies can
find it.
Having the local route in a non-default table is a common setup in VRF environments. When the above interface is moved into a VRF, all its routes, including local routes, are moved to the VRF table.
In VRF setups, a PBR rule 1000: from all lookup [l3mdev-table] is inserted
by default. It is effectively oif vrfX table xxx and iif vrfX table xxx.
The later rule, iif vrfX table xxx, tells Linux to lookup routes from the VRF
table, for packets coming from the VRF interface. This makes sure the local
route can be reached, even if it is not in the default local table.
When writing custom Linux PBR rules, it is necessary to ensure the local route is always reachable, otherwise Linux won’t reply to ARP.
Linux source code ref
I am no Linux kernel development expert, nor did I read any source code on the Linux networking stack either. The below source is just for referencial purposes only. It may be completely unrelated.
Linux v7.0, net/ipv4/arp.c:
699 /*
700 * Process an arp request.
701 */
702
703 static int arp_process(struct net *net, struct sock *sk, struct sk_buff *skb)
704 {
...
841 if (arp->ar_op == htons(ARPOP_REQUEST) &&
842 ip_route_input_noref(skb, tip, sip, 0, dev) == 0) {
843
844 rt = skb_rtable(skb);
845 addr_type = rt->rt_type;
846
847 if (addr_type == RTN_LOCAL) {
848 int dont_send;
849
850 dont_send = arp_ignore(in_dev, sip, tip);
851 if (!dont_send && IN_DEV_ARPFILTER(in_dev))
852 dont_send = arp_filter(sip, tip, dev);
853 if (!dont_send) {
854 n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
855 if (n) {
856 arp_send_dst(ARPOP_REPLY, ETH_P_ARP,
857 sip, dev, tip, sha,
858 dev->dev_addr, sha,
859 reply_dst);
860 neigh_release(n);
861 }
862 }
863 goto out_consume_skb;
864 } else if (IN_DEV_FORWARD(in_dev)) {