HCX Service-Mesh Upgrade Blocked Due to Stale Ongoing Migration Entries

Products

• VMware HCX

Issue/Introduction

VMware HCX (HCX) upgrades to version 4.11.4 are blocked due to the environment reporting a "failed state".

• The HCX Manager UI continuously reports "ongoing migrations" despite no active jobs currently running, which prevents the Service Mesh upgrade from proceeding.
• Virtual machines in the vCenter inventory are stuck with active replication flags from a previous migration that stalled and failed to clear properly.

Environment

VMware HCX

Cause

• The issue is caused by stale entries in the HCX Manager and active replication flags set on the virtual machines in vCenter.
• These records were not properly purged when a previous migration failed or was cancelled.
• This lack of cleanup occurred due to a loss of connectivity between HCX Manager and vCenter and/or insufficient permissions (e.g., the service account utilized by HCX being disabled) during the cleanup/cancel phase of the migration.

Resolution

1. Attempt the steps listed in KB below to force cancel all stale migrations. Error when trying to upgrade HCX service-mesh "Operation not allowed as appliance has ongoing RAV Migrations."

   1. If this fails and running a supported HCX version, open a case with Broadcom Support to assist with DB Cleanup. Creating and managing Broadcom cases

2. If the HCX version deployed is EOS (end of support), redeploy a fresh HCX Manager appliances on a supported version at both the source and destination sites.

   1. On the Target site, if network extensions are used, remove any virtual machines associated with previous L2 extensions (move to temp segment) and clean up the NSX inventory. This will allow the re-extension of the network once the new HCX appliance is deployed.

3. Re-extend the networks using the newly deployed HCX 4.11.4 instances.

4. Utilize the steps in the following KB to check and disable replication if needed on VMs which were apart of previous failed migration. Disabling VMware vSphere Replication on a virtual machine