Guest OS intermittently fails to ping when a physical NIC is added to the virtual switch in Cisco ACI environment.

Products

• VMware vSphere ESXi

Issue/Introduction

• Adding/Removing uplink to/from vSwitch.
• ESXi uplinks are connected to Cisco ACI.
• Ping to Guest OS through physical network fails for a few minutes.
• RARP was sent after adding/removing but physical switch is not sending packets to ESXi.
  
  To capture both in/out packets at new uplink:

  pktcap-uw --uplink vmnicX --dir 2 -o /vmfs/volumes/<VOLUME>/vmnicX.pcapng --ng

  NOTE:
  Replace vmnicX and <VOLUME>.
  Depending on workloads, uplink may be handling huge traffic. Specify count with -c option to limit a number of packets captured.

  Run tshark command to show RARP and ICMP packets.
  In this example, RARP is sent from ESXi to physical switch multiple times to notify uplink change on adding/removing uplink to/from vSwitch.
  However, ESXi is not receiving ICMP request from Cisco ACI for 2 minutes.
  
  $ tshark -r vmnicX.pcapng 'eth.addr==<MAC Address> && (icmp || eth.type == 0x8035)'
 35.705439 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
 40.705274 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
 49.705776 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
 58.706403 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
 76.707094 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
103.708129 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
148.709893 <MAC_ADDRESS> → Broadcast    RARP 60 Who is <MAC_ADDRESS>? Tell <MAC_ADDRESS>
158.492451  <SENDER_IP_ADDRESS> → <RECEIVER_IP_ADDRESS>  ICMP 98 Echo (ping) request  id=0x0002, seq=3254/46604, ttl=64
158.492529  <RECEIVER_IP_ADDRESS> → <SENDER_IP_ADDRESS>  ICMP 98 Echo (ping) reply    id=0x0002, seq=3254/46604, ttl=64

  NOTE:
  <MAC Address>: Guest OS MAC address.
  EtherType 0x8035 is Reverse Address Resolution Protocol (RARP)

Environment

ESXi 8.0

Cause

When ESXi adds/removes physical NIC to vSiwtch, ESXi sends RARP to notify physical switches. The physical switches update their MAC table and start forwarding ethernet frames to new port.

However, in this case ESXi is not receiving packets from Cisco ACI.

Resolution

Contact Cisco ACI vendor to investigate why Cisco ACI is not sending ICMP packets to ESXi after receiving RARP.