HCX-IX appliance upgrade failure "Operation not allowed as appliance has ongoing Bulk Migrations"

Products

• VMware HCX

Issue/Introduction

• An upgrade for an HCX Service Mesh appliance (IX) failed with the error: "Operation not allowed as appliance has ongoing Bulk Migrations"
  
  
• The following error is observed in /common/logs/admin/app.log :
Appliance operation failed for applianceld ########-####-####-####-############ with error Interconnect Service Workflow InterconnectRedeploy failed. Error: Unable to proceed. Appliance Name: ##-IX-##, Blocking Service:ReplicationTransferService, Reason: Operation not allowed as appliance has ongoing Bulk Migrations.
• From /common/logs/admin/app.log, the migration causing the issue can be found by searching for: "Active bulk migration replications found", similar output is:
  ApplianceUpdateCallbackMobilityBulk- Active bulk migration replications found : ["########-####-####-####-############"]

Environment

VMware HCX

Cause

This issue occurs when outdated or stale migration records, or incomplete cleanup of terminated or cancelled migrations, remain in the system.

Resolution

Ensure there are no active, ongoing migrations during the HCX Manager IX appliance.

Workaround:

Option 1:

If you are able to find the migration ID causing the upgrade to be blocked, you can search for the migration ID on the HCX Manager UI, and proceed with the force cancel.
For more information, please access: Force Cleanup for a Failed or Canceled Migration

If you are unable to find the migration ID from the HCX Manager UI, this can be due to the fact that the migration was archived, and you can proceed with the HCX API, for more information: Canceling an HCX Migration using APIs

Note:
From HCX 4.11.1 and onwards, when a migration is archived from the HCX Manager UI, the HCX Manager will automatically perform the cleanup and remove the stale entries.

Option 2:

Proceed with the Force Upgrade:

1. Log in to the HCX Manager at the source site.
2. Navigate to Interconnect --> Service Mesh -->View Appliances --> Update Appliance
         Interconnect appliances show a green flag in the Available Versions column if there is an update available.
3. Select Force Update