HCX Migration failed with the error: "Unexpected status code: 503"

Products

• VMware HCX

Issue/Introduction

• HCX migrations failed with the error: "Unexpected status code: 503"
• In the /common/logs/admin/app.log, similar error messages are observed:
  

  <timestamps> UTC [Service_SvcThread-741, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ##] ERROR c.v.v.h.a.vcenter.VcConnection- Error Logging onto VCenter:https://<VC-FQDN-or-IP> User:<username>, message: Unexpected status code: 503
  
  <timestamps> UTC [ReplicationTransferService_SvcThread-996, Ent: HybridityAdmin, , TxId: ##] WARN  c.v.h.s.r.j.ReplicationTransferMonitor- Error connecting to vCenter as vCenter service is down: Unexpected status code: 503

• The /common/logs/admin/app.log file also displays connectivity issues or unresponsive vCenter services prior to the 503 error code:
  

  com.vmware.vchs.hybridity.adapters.vsphere.vim.VsphereServiceUnavailableException: Error communicating to VC endpoint <FQDN-or-IP>:443. Reason: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fecf8009ca0] _serverNamespace = /sdk action = Allow _port = 8085)

Environment

VMware HCX 4.x

Cause

This issue is caused by vCenter becoming unresponsive during an ongoing migration or being upgraded while HCX migrations are in progress. Since HCX relies on vCenter services, any disruption to vCenter can interrupt the migration process.

Resolution

This issue is resolved in VMware HCX 4.11.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Note:
If the vCenter endpoint remains unreachable for more than 2 to 2.5 hours, ongoing migrations may be impacted regardless of its current active stage. In such cases, the Retry Migration action needs to be performed to resume or recover the affected migrations once the vCenter endpoint is reachable.

Additional Information

We strongly advise against rebooting the HCX Manager, vCenter Server, or the source and target cluster hosts while migrations are in progress. Several backend tasks run concurrently during migration, and restarting any of these components or their associated services may lead to disruptions or failures in the migration process.

When using Bulk or RAV (Replication Assisted vMotion) migration, enable 'Seed Checkpoint'. In the event that a migration is unsuccessful or canceled, Seed Checkpoint retains the target disks created at the target site. Without Seed Checkpoint, the HCX roll back process cleans up the target disks created during the migration and all transferred data is lost. For more information, please access HCX Health Check and Best Practices