When trying to create a Workload Domain on SDDC Manager, it fails with Error " Validate Management Workload Domain has enough Resources for NSX Deployment"

Products

• VMware SDDC Manager

Issue/Introduction

• Creating Workload Domain failed on SDDC Manager fails with Error: " Validate Management Workload Domain has enough Resources for NSX Deployment"
• /var/log/vmware/vcf/domainmanager/Domainmanager.log
  
  

  DEBUG [vcf_dm,67f7dd3d0b#####7d86dc1847d165e8a,3a75] [c.v.e.s.c.c.v.vsphere.VcManagerBase,dm-exec-16]  Connecting to https://VC_FQDN:443/sdk
  ERROR [vcf_dm,67f7dd3d0b#####7d86dc1847d165e8a,3a75] [c.v.e.s.o.model.error.ErrorFactory,dm-exec-16]  [KQ##VK] VSPHERE_CONNECTION_FAILED_KEY Cannot connect to vCenter Server or host VC_FQDN
  com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Cannot connect to vCenter Server or host VC_FQDN
  .
  .
  at java.base/java.lang.Thread.run(Thread.java:840)
  Caused by: java.lang.NullPointerException: Cannot invoke "com.vmware.vim.binding.vim.vm.ConfigInfo.getHardware()" because the return value of "com.vmware.vim.binding.vim.VirtualMachine.getConfig()" is null
   at com.vmware.evo.sddc.common.services.DomainResourcesFactory.computeResourcesAllocatedToVms(DomainResourcesFactory.java:149)
   at com.vmware.evo.sddc.common.services.DomainResourcesFactory.create(DomainResourcesFactory.java:86)        ... 18 common frames omitted

• /var/log/vmware/vpxd/vpxd.log
  
  

  info vpxd[06603] [Originator@6876 sub=MoCluster opID=PodCrxMgr-domain-c#-63] Dumping vCLS Pod Crx host infos; domain-c#, [{[vim.HostSystem:host-##67,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##16,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##81,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##77,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##72,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##21,FQDN], st: 2, vm: [vim.VirtualMachine:vm-##169,vCLS-c266b268-67e0-####-####-000000000030], t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##23,FQDN], st: 2, vm: [vim.VirtualMachine:vm-##173,vCLS-c266b268-67e0-####-####-000000000031], t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##25,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##62883,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }, {[vim.HostSystem:host-##62878,FQDN], st: 0, vm: (null), t: (null), ghost vm: (null), aa: false, f: 0, }]

• Disable/Enable retreat mode on the impacted cluster still shows as invalid/orphaned vCLS VMs in the vCenter inventory
• Manually unregistering vCLS VMs from VC inventory did not clean up. After refreshing the vCenter inventory still shows up invalid vCLS VMs .  

Environment

VCF 5.x

Cause

The workload domain creation fails due to orphan VMs in the vCenter inventory. SDDC manager performs check on all the VMs' resources and during this we get "com.vmware.vim.binding.vim.VirtualMachine.getConfig()" is null. 

 In this case we found VCLS VMs as invalid/orphan. 

Resolution

To fix this issue you need to remove the Invalid/orphan VMs from vCenter inventory. 

Note : If the vCenter Server is part of a Linked Mode replication group, backups/offline snapshots need to be created for every member of the Linked Mode group

• Login the vCenter server using HTML5 client

• Disable vCLS using vCenter server UI. Available from vSphere 7.0 U3o/8.0 U2 and Later ( Reference KB: Disable vCLS on a Cluster via Retreat Mode )

• In Hosts and Clusters inventory, select a cluster.

• Click on the Configure tab.

• Under vSphere Cluster Services, select General.

• In the top right, click on EDIT vCLS MODE.

• In the Edit vCLS Mode pop up window, click on the second radio option Retreat Mode.

• Manually removing a stale VM from the vCenter Server vpostgres database ( Reference KB: Manually removing a stale VM from the vCenter Server vpostgres database )

• Enable the vCLS , refer Step 6

• Re-try the workflow from the SDDC manager.