NE HA Failover Threshold Guidelines - VMware HCX
Article ID: 367619
Updated On:
Products
Issue/Introduction
In some cases, HCX Network Extension High Availability (NE HA) Failover can happen more often than expected in a busy network that is prone to occasional packet loss.
HCX NE HA periodically transmits Bidirectional Forwarding Detection (BFD) UDP packets as a heartbeat mechanism between the active and standby NE appliance at the same local site.
When the sustained loss of these packets is detected it will trigger a standby to active NE failover.
The default failover detection time depends on the HCX version and Service Mesh (SM) configuration:
| HCX Version | Service Mesh (SM) Type | BFD Threshold / Interval | Detection Time | Script Supported |
|---|---|---|---|---|
| 4.9.1+ | All | 6 heartbeats * 250ms | 1.5 seconds | Yes |
| 4.10+ | All | 60 heartbeats * 250ms | 15 seconds | Yes |
| 9.0+ | All | 60 heartbeats * 250ms | 15 seconds | Yes |
| 9.1+ | Non-Enhanced | 60 heartbeats * 250ms | 15 seconds | Yes |
| 9.1+ | Enhanced | 3 heartbeats * 1000ms | 3 seconds | No |
Note: These default values can be changed, except for VCF Operations HCX 9.1 using the Enhanced Service Mesh, where these defaults are hardcoded and cannot be changed using a script.
Heartbeat Interface Mechanisms
HCX NE HA uses two independent BFD heartbeat streams to prevent false failovers. A failover only occurs if both streams detect sustained packet loss simultaneously.
- Non-Enhanced Appliances: Heartbeats are sent via the Management and Uplink interfaces:
- Enhanced Appliances: Heartbeats are sent via the Management and dedicated HCX Intrasite Control interfaces.
NE HA is designed to send two independent BFD heartbeat streams between the active and standby NE appliance at the same local site.
One BFD stream is sent between the NE management interfaces on the active and standby NE appliance at the same local site and the other BFD stream is sent between the NE uplink or the HCX Intrasite Control interfaces for non-enhanced and enhanced appliances, respectively, on the active and standby NE appliance at the same local site. If a period of sustained packet loss is only experienced on one of the interfaces, either the management or uplink / Intrasite Control but not both, then no NE HA failover will take place. This is to avoid false positives where the NE appliance is not truly down. Sustained packet loss must be detected by both BFD streams simultaneously to trigger a standby to active NE failover.
The dual BFD heartbeat fail safe may not work as planned however when a shared single interface is used on an NE appliance for both management and uplink / Intrasite Control. In this scenario, a period of sustained packet loss may impact both BFD streams since they share the same interface. Because of this, it is recommended to always use separate management and uplink / Intrasite Control interfaces on an NE appliance that is configured for HA.
Environment
VMware HCX 4.11
VCF Operations HCX
Resolution
- For the appliances, it is strongly recommended to use separate physical interfaces for management and uplinks or Intrasite Control. Sharing a single interface removes the fail-safe benefit, as a local network disruption will impact both heartbeat streams simultaneously and trigger an unnecessary failover.
- Threshold Adjustment: If you need to modify default values for better network tolerance:
- For supported versions (see table above): Please open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.
- For HCX 9.1+ Enhanced SMs: The BFD interval is not configurable, and the adjustment script is not supported.
Feedback
