MON-enabled VMs lose /32 routes after updating NSX credentials in HCX

Products

• VMware HCX

Issue/Introduction

• Mobility Optimized Networking (MON) enabled VMs lose network connectivity.
• The connectivity loss occurs a few minutes after correcting NSX credentials or restoring the management plane connection in the HCX Administrator portal (9443).
• In the HCX UI, the task is stuck at 60%.

 

Environment

VMware HCX

Cause

This issue is not fundamentally related to the NSX credentials, and under normal conditions, updating credentials in the HCX Administrator portal should not cause any network impact.

The impact occurs only when a configuration job becomes stuck in the HCX database and it's typically triggered by a vCenter event that occurred while the management plane connection to NSX was broken (HTTP 403).

Once the NSX credentials are corrected and the connection is restored, HCX begins processing its backlog of pending tasks. If one of these stuck jobs was a deletion task for a MON-enabled VM, its execution results in the unexpected removal of required /32 host routes, leading to a loss of connectivity for the affected VMs.

Resolution

This is a known issue impacting VMware HCX.

 

Engineering is actively working on design improvements to introduce better guardrails.

The objective is to ensure that when communication is restored, HCX re-evaluates the relevance of any pending tasks before they are executed to prevent automated cleanups from occurring unexpectedly.

 

Workaround:

1. Verify and correct the NSX credentials in the HCX Administrator portal (port 9443).

2. Toggle MON: Disable and then re-enable MON for the affected VMs. This forces HCX to re-synchronize the state with NSX and restore the necessary /32 routes.