HCX Migration failed with the error: "Unexpected status code: 503"

Products

• VMware HCX

Issue/Introduction

• HCX migrations failed with the error: "Unexpected status code: 503"
• In the /common/logs/admin/app.log, similar error messages are observed:
  

  <timestamps> UTC [Service_SvcThread-741, Enterprise: HybridityAdmin, Username: HybridityAdmin, TxId: ##] ERROR c.v.v.h.a.vcenter.VcConnection- Error Logging onto VCenter:https://<VC-FQDN-or-IP> User:<username>, message: Unexpected status code: 503
  
  <timestamps> UTC [ReplicationTransferService_SvcThread-996, Ent: HybridityAdmin, , TxId: ##] WARN  c.v.h.s.r.j.ReplicationTransferMonitor- Error connecting to vCenter as vCenter service is down: Unexpected status code: 503

• The /common/logs/admin/app.log file also displays connectivity issues or unresponsive vCenter services prior to the 503 error code:
  

  com.vmware.vchs.hybridity.adapters.vsphere.vim.VsphereServiceUnavailableException: Error communicating to VC endpoint <FQDN-or-IP>:443. Reason: Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x00007fecf8009ca0] _serverNamespace = /sdk action = Allow _port = 8085)

Environment

VMware HCX

Cause

This issue is caused by vCenter becoming unresponsive during an ongoing migration or being upgraded while HCX migrations are in progress. Since HCX relies on vCenter services, any disruption to vCenter can interrupt the migration process.

Resolution

We strongly advise against rebooting the HCX Manager, vCenter Server, or the source and target cluster hosts while migrations are in progress. Several backend tasks run concurrently during migration, and restarting any of these components or their associated services may lead to disruptions or failures in the migration process.

Workaround:
Once vCenter is fully operational, you may retry the HCX migration if the job has failed. Refer Retry a Failed or Canceled Migration

Additional Information

When using Bulk or RAV (Replication Assisted vMotion) migration, enable 'Seed Checkpoint'. In the event that a migration is unsuccessful or canceled, Seed Checkpoint retains the target disks created at the target site. Without Seed Checkpoint, the HCX roll back process cleans up the target disks created during the migration and all transferred data is lost. 

For more information, please access: HCX - Health Check and Best Practices