External ICMP client is able to ping just the adjacent interface IP of the VAP, the others don't respond

Products

• XOS
• APM

Issue/Introduction

In this scenario, a client is trying to ping the VAP interface IP 192.168.yy.1 (not adjacent to client side) in the following setup and is failing:

10.10.10.10(client) -> router(192.168.xx.193) -> tt.73(192.168.xx.196) -> VAP member -> tt.27(192.168.yy.1)

The client pings 192.168.xx.196 VAP member IP successfully.
Routing on the VAP towards the client IP is setup properly.

NPM packet capture would show the following ARP traffic towards the router:

00:03:d2:f1:22:03 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 72: vlan 2, p 0, ethertype 802.1Q, vlan 73, p 0, ethertype ARP, arp who-has 192.168.xx.193 tell 192.168.yy.1 

This is causing ARP entry for 192.168.xx.193 to remain incomplete.

By pinging host 192.168.xx.193 (router) from 192.168.xx.196 (tt.73 IF) we see the expected ARP request/reply which made the ping work: 

00:00:5e:00:08:d2 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 72: vlan 2, p 0, ethertype 802.1Q, vlan 73, p 0, ethertype ARP, arp who-has 192.168.xx.193 tell 192.168.xx.196 
00:00:0c:07:ac:1c > 00:00:5e:00:08:d2, ethertype 802.1Q (0x8100), length 68: vlan 2050, p 0, ethertype 802.1Q, vlan 73, p 5, ethertype ARP, arp reply 192.168.xx.193 is-at 00:00:0c:07:ac:1c 

Cause

The ARP request contains the interface IP being pinged (tt.27) and not the IP of interface adjoining the router (tt.73).

Resolution

We can modify the Linux kernel ARP settings to make the ping work as follows:

On the fly change on the VAP member (not persistent):
# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce 

Permanent change in /etc/sysctl.conf: 
net.ipv4.conf.all.arp_announce=2 

The change is based on the following sysctl conf Linux kernel parameter:
 

arp_announce - INTEGER
	Define different restriction levels for announcing the local
	source IP address from IP packets in ARP requests sent on
	interface:
	0 - (default) Use any local address, configured on any interface

	1 - Try to avoid local addresses that are not in the target's
	subnet for this interface. This mode is useful when target
	hosts reachable via this interface require the source IP
	address in ARP requests to be part of their logical network
	configured on the receiving interface. When we generate the
	request we will check all our subnets that include the
	target IP and will preserve the source address if it is from
	such subnet. If there is no such subnet we select source
	address according to the rules for level 2.

	2 - Always use the best local address for this target.
	In this mode we ignore the source address in the IP packet
	and try to select local address that we prefer for talks with
	the target host. Such local address is selected by looking
	for primary IP addresses on all our subnets on the outgoing
	interface that include the target IP address. If no suitable
	local address is found we select the first local address
	we have on the outgoing interface or on all other interfaces,
	with the hope we will receive reply for our request and
	even sometimes no matter the source IP address we announce.

	The max value from conf/{all,interface}/arp_announce is used.

	Increasing the restriction level gives more chance for
	receiving answer from the resolved target while decreasing
	the level announces more valid sender's information.

(Above from the Linux kernel documentation.)

Workaround

N/A