HCX - NE appliance In-Service upgrade workflow may create L2 Loop in the network

Products

• VMware HCX

Issue/Introduction

The HCX Network Extension (NE) appliance upgrade, when performed using the In-Service option, may result in a Layer 2 loop occurring within a specific segment during the upgrade workflow.

Location of App Engine log:

• HCX Manager : /common/log/admin/app.log

2022-11-10 22:58:49.473 UTC [InterconnectService_SvcThread-48, IX:########-####-####-####-########09a9, J:b27948c0, , TxId: ########-####-####-####-########bd1a] WARN c.v.v.h.s.i.DeployAppliance- Unable to set the bridgeStateFlag to down for the appliance ########-####-####-####-########8277. Retrying the operation.

 

Environment

VMware HCX

Cause

This issue is related to timing in the HCX Network Extension (NE) appliance when using the In-Service workflow for upgrades. 
Under certain infrastructure conditions, the NE appliance may experience delays in completing the boot-up process. 
As a result, the HCX Manager fails to create the "bridgeStateFlag" for the newly deployed NE appliance, leading to the new appliance operating with an active bridge data path while the old NE appliance remains in service. 
This causes a temporary Layer 2 loop in the extended data path, which may last for a few seconds.

Resolution

This is fixed in HCX 4.5.2 release.

Workaround:
The recommendation is to use Standard Upgrade workflow as an alternative, if NE appliance version needs to be upgraded.
Also, this won't impact HCX NE running in High Availability (HA) pair.

IMPORTANT: NE HA workflow doesn't depend upon In-Service mechanism. We perform failover during upgrade and it won't have a regular downtime. The only downtime expected in HA workflow is during failover.

Additional Information

Impact/Risks:

• All HCX versions are affected.
• Network Extension appliance may continue operating using existing version without upgrade.
• Network extension service will remain active.
• MON enabled VMs will continue to operate as expected.
• This behavior will NOT impact upgrade workflow with HCX NE running in HA pair.
• There will be NO impact to HCX migration services.