VCF 9.x Bring-Up Fails: "Insufficient space to deploy vCenter Server" (vSAN State Desync)

Products

• VMware vSAN
• VMware Cloud Foundation

Issue/Introduction

During the VMware Cloud Foundation (VCF) 9.x bring-up process, the deployment may halt at the "Deploy vCenter" stage. The VCF Installer UI will display the following error:

"Insufficient space to deploy vCenter Server"

Further investigation of the environment typically reveals contradicting symptoms:

• The vSAN Datastore capacity reports as 0 bytes (or significantly less than the actual physical disks installed).

• The domainmanager.log contains the following stack trace: com.vmware.evo.sddc.common.client.vmware.vsphere.VsphereException: Failed to query ESA certified disks.

• The physical disks appear perfectly fine in the host view, showing as "Eligible" and "Healthy".

Environment

VMware Cloud Foundation 9.x, VMware vSAN 9.x (ESA)(OSA)

Cause

This issue is caused when ESX maintenance mode state is out of sync with the vSAN Decom state. While a host is in vSAN Decom state the storage from the host is not included in the size of the vSAN datastore resulting in a decrease in size. If all hosts within the cluster are in a vSAN Decom state then the vSAN datastore will report a size of 0 bytes. For more details see KB vSAN and vSphere maintenance modes may diverge

 

If the vSAN Datastore is reporting size of 0 bytes then vCenter can not be deployed.

Resolution

To resolve this issue and bring ESX maintenance mode and vSAN Decom state back into sync with each other perform the below steps:

Step 1: Confirm the State Mismatch Establish an SSH session to the affected ESX host(s) and execute the following commands:

1. Verify the ESX host maintenance mode state::

esxcli system maintenanceMode get

(Expected output: Disabled)

2. Verify the vSAN Decom state:

esxcli vsan cluster get | grep "Maintenance Mode State"

(Expected output: Maintenance Mode State: ON)

Conclusion: If the ESX host reports 'Disabled' while vSAN reports 'ON', the ESX maintenance mode and vSAN Decom states are out of sync.

Step 2: Bring the 2 states in sync with each other

1. Enter Maintenance Mode:

esxcli system maintenanceMode set -e true -m noAction 

2. Exit Maintenance Mode:

esxcli system maintenanceMode set -e false 

This can also be done via the Host UI as well.

Step 3: Verify vSAN Decom is now set off by running the below command:

esxcli vsan cluster get | grep "Maintenance Mode State"

The output should now correctly report OFF.

Once the state is OFF, return to the VCF Installer UI and click Retry. The installer will now recognize the actual storage capacity and proceed with the VCSA deployment.

Additional Information

Related Error Strings for Search:

• • "Insufficient space to deploy vCenter Server"
  • "No space available on the vSAN datastore"
  • "Failed to query ESA certified disks"
  • "Maintenance Mode State: ON"
  • VCF 9.0, VCF 9.0.1, VCF 9.1