Migration fails with "No target host could be choosen after capacity fit filtering is done"

Products

• VMware HCX

Issue/Introduction

• HCX Migration fails at switchover with the error: "No target host could be choosen after capacity fit filtering is done". NOTE:  The typo in error message word 'choosen' is matching the code.
• The following error is observed in Target HCX in /common/logs/admin/app.log:
  

  2024-08-29 03:10:08.230 UTC [VmotionService_SvcThread-20061, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-############] WARN  c.v.v.h.m.c.CompatibleDatastoreSelector- Not choosing datastore:ManagedObjectReference: type = Datastore, value = datastore-30, serverGuid = null having dsFreeSpace:-682651287552 as it does not satisfy Vm capacity requirement vmTotalDiskSpace:2069569206691

  2024-08-29 03:10:08.233 UTC [VmotionService_SvcThread-20061, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-############] WARN  c.v.h.s.v.j.CollectTargetConfigWorkflow- [migId=########-###-####-####-############] Failed to check candidate datastore '' for VM required disk space.
  com.vmware.vim.binding.vmodl.fault.ManagedObjectNotFound: The object 'vim.ManagedEntity:' has already been deleted or has not been completely created
  2024-08-29 03:10:08.233 UTC [VmotionService_SvcThread-20061, Ent: HybridityAdmin, , TxId: TxId: ########-####-####-####-############] ERROR c.v.h.s.v.j.CollectTargetConfigWorkflow- Method doGetDestinationDetails Failed with exception:
  java.lang.RuntimeException: No destination datastore can be chosen
  
  

• There are multiple mobility groups waiting for switchover.

   

Cause

The VM's capacity requirement exceeds the available space on the target datastore.  The datastore does not have sufficient capacity to accommodate the VM to be cutover.

The space consumption of the datastore increased while existing migrations were already waiting for switchover, possibly due to addition of migrations to the system.
In such cases, HCX does have a validation on migration UI to indicate the storage is being overcommitted, if this notification is ignored by user or the space was consumed by entities other than the migrating workloads' disks.
This issue occurs if the VMs are all of 'Thin Provisioning' type where disks are not reserving any space on datastore before it starts transfer.

Resolution

• Ensure that the target datastore has enough free space available to accommodate the Source VM.
• In additional, up to 20% extra space may be used temporarily during the migration.
• While optimizations are being built for future releases of HCX to manage disk placement and providing actionable validations before switchover begins for thin provisioned disks, for the workaround, cancellation of a some migrations would be needed to free up the space: 
  
  
  • Cancel migrations that provides storage space calculated in the error message from the logs (best would be to cancel the migrations that has seed checkpoint enabled).
  • Switchover the migrations (one group at a time) to satisfy migration space requirements.
  • Retry cancelled migrations.
  • Add more migrations and ensure destination datastore is not overcommitted.