FortiADC HA communication fails when virtual machines are on different ESXi hosts
Article ID: 443509
Updated On:
Products
Issue/Introduction
When deploying FortiADC virtual appliances in a High Availability (HA) configuration within a VMware vSphere environment, you may observe the following:
- HA communication works correctly when both FortiADC VMs are residing on the same ESXi host, allowing them to establish a Primary/Secondary relationship.
- When the VMs are migrated to different ESXi hosts, HA communication fails, causing both appliances to assume the "Primary" state (split-brain).
- Standard VM-to-VM networking on the same VLAN works correctly.
- Security policies (Promiscuous Mode, MAC Address Changes, and Forged Transmits) are set to Accept on the Port Group.
Environment
VMware ESXi 8.x
Cause
The issue is typically caused by the physical network infrastructure (upstream switches) failing to forward HA heartbeat traffic between the ESXi hosts.
FortiADC uses Ethernet protocol 0x8890 for cluster discovery and heartbeats. In many cases, these packets are sent to a multicast MAC address. While the virtual switch handles this traffic internally when VMs are on the same host, the physical switches may drop this protocol or fail to forward the multicast traffic across physical links.
Resolution
To resolve this issue, you must verify that the HA heartbeat traffic is successfully leaving the source host and arriving at the destination host. This can be achieved by running packet captures on the ESXI hosts. The captures should be directed to a datastore which can then be downloaded and analyzed with a tool such as Wireshark.
1. Prepare a directory in a shared datastore where you will store the captures, e.g. /vmfs/volumes/<datastore>/<directory>
2. Verify traffic leaving the source host
- Log in to the ESXi shell of the host where the source VM resides. Identify the VM's switchport that is used for sending the heartbeat packets.
net-stats -l | grep <vm-name>This will list all the switchports (listed under the PortNum column) that are mapped to vm/vmk - Identify the uplink for the specific VM's switchport that is sending the heartbeat packets
run the command "vsish" and press enter. "cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink". - Run the following packet capture command to see if heartbeats are sent from the physical uplink:
pktcap-uw --uplink vmnicX --capture UplinkSndKernel -o /vmfs/volumes/<datastore>/directory>/source_uplink.pcapng
(Replace vmnicX with the actual uplink identified from above used by the HA port group).
3. Verify traffic arriving at the destination host
- Log in to the ESXi shell of the host where the destination VM resides. Identify the VM's switchport that is used for sending the heartbeat packets.
net-stats -l | grep <vm-name>
(This will list the all the switchports (listed under the PortNum column) that are mapped to vm/vmk) - Identify the uplink for the specific VM's switchport that is sending the heartbeat packets:
run the command "vsish" and press enter. "cat /net/portsets/vSwitch_name/ports/switchport_number/teamUplink". - Run the following packet capture command to see if heartbeats are reaching the physical uplink:
pktcap-uw --uplink vmnicY --capture UplinkRcvKernel -o /vmfs/volumes/<datastore>/directory>/dest_uplink.pcapng
(Replace vmnicX with the actual uplink identified from above used by the HA port group).
4.Review the captures in Wireshark. If protocol 0x8890 is seen leaving the source host but not arriving at the destination, the issue is external to VMware.
5. Physical Network Requirements
Engage your network team to ensure the following:
Protocol 0x8890: Ensure the physical switches are not filtering or dropping Ethernet protocol 0x8890.
VLAN Trunking: Confirm the HA VLAN is correctly trunked on all physical switch ports connected to the ESXi hosts.
Multicast Forwarding: Verify that the physical network is configured to forward multicast traffic for the HA VLAN.
Additional Information
Feedback
