HCX Network Extension HA Cluster Status Fluctuates Between Degraded and Healthy

Products

• VMware HCX

Issue/Introduction

• VMware HCX Network Extension (NE) High Availability (HA) cluster status continuously fluctuates between "degraded" and "healthy" without user intervention:
• Review of the /var/log/messages on the HCX NE appliance reveals repeated Bidirectional Forwarding Detection (BFD) session unreachable errors transitioning between Unreachable and Up:

• ha-agent 1951 - - ovs|39210|hb_heartbeat_lib|INFO|BFD session 192.168.1.1:192.168.1.2 state changed to Unreachable(1/0)
  ha-agent 1951 - - ovs|39214|hb_heartbeat_lib|INFO|BFD session 192.168.1.1:192.168.1.2 state changed to Up(0/3)
  ha-agent 1951 - - ovs|39218|hb_heartbeat_lib|INFO|BFD session 192.168.1.1:192.168.1.2 state changed to Up(0/0)
  ha-agent 1951 - - ovs|39230|hb_heartbeat_lib|INFO|BFD session 192.168.1.1:192.168.1.2 state changed to Unreachable(1/0)

Cause

The issue is can be caused by a duplicate IP address on the network interfering with the BFD heartbeat traffic between the HCX Network Extension HA appliance interfaces.

Resolution

1. Identify the specific IP addresses assigned to the HCX Network Extension HA appliances for the heartbeat network.
2. Perform a packet capture filtering on the affected HA IP address at the ESXi host level to capture all traffic (including ARP and non-BFD packets from the rogue device):
   1. Log in to the ESXi host where the active HCX Network Extension HA appliance resides using SSH with root privileges.
   2. Identify the physical uplink (vmnicX) associated with the ESXi vSwitch managing the HCX appliances by running: esxtop (press 'n' for network view) or esxcli network vm list followed by esxcli network vm port list -w <world_id>.
   3. Run the pktcap-uw utility to capture all traffic associated with the affected IP address, replacing vmnicX with your identified uplink and <SUSPECTED_DUPLICATE_IP_ADDRESS>  with the IP in question: pktcap-uw --uplink vmnicX --capture UplinkSndKernel,UplinkRcvKernel --ip <SUSPECTED_DUPLICATE_IP_ADDRESS> -o - | tcpdump-uw -enr -
   4. Monitor the packet trace output for unexpected MAC addresses or rogue sources responding for or transmitting from the HA IP address.
   5. To terminate the active trace, press CTRL-C. Ensure all background captures are stopped by running: kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u).
   6. Identify the rogue device or virtual machine on the local network broadcasting the duplicate IP address.
3. Reconfigure or isolate the conflicting device to eliminate the IP conflict.
4. Verify via the HCX UI that the BFD sessions stabilize and the Network Extension HA cluster status remains consistently "healthy".

Additional Information

Understanding Network Extension High Availability
Resolving Health Issues on Network Extensions (NE)