VCF 5.x Pre-check Failure: "VM free space check failed internally"

Products

• VMware SDDC Manager

Issue/Introduction

vCenter Major Upgrade through Upgrade Runner mechanism fails during bundle precheck/upgrade with error: vCenter precheck failed while getting the DNS server from VM.

vCenter Patch Upgrade through RDU with STATIC network configuration can fail during upgrade with error: vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage

From the SDDC Manager UI vCenter Pre-check you may see the follow Error:

"Validates vCenter VM's free space on root"

"VM free space check failed internally" AND "Check the operationsmanager logs and if this issue persists contact VMware support..."

You may not see any specific space issues with vCenter or SDDC Manager VMs. Check the space on each component with "df -h" and/or "du -sh *" on any specific partition/directory where you may notice a space issue. However, in this instance, no space issues were observed.

You may observe from the /var/log/vmware/vcf/operationsmanager.log the following snippet:

2024-08-21T13:48:32.470-0500 DEBUG [vcf_om,66c6368098c70380bdf062fd675a541a,ec4d] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-23] Searching for VM with address <vcenter fqdn>
2024-08-21T13:48:32.582-0500 DEBUG [vcf_om,66c6368098c70380bdf062fd675a541a,ec4d] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-23] Found VM <vcenter hostname>8.0.2.00100-22617221 with address <vcenter fqdn>
........

2024-08-21T13:48:32.594-0500 ERROR [vcf_om,66c6368098c70380bdf062fd675a541a,ec4d] [c.v.v.b.p.t.v.SimpleNativeValidator,om-exec-23] Native validator with name 'mgmt-vm-free-space-validator' failed internally
. Reference token: SOQT3B
java.lang.NullPointerException: Cannot read the array length because "array" is null

 

Environment

3.x to 4.x upgrade followed by 4.x to 5.2.1 Upgrade

4.x to 5.x upgrade followed by 5.x to 5.2.1 upgrade using RDU (Reduced Downtime Migration) mechanism and STATIC network configuration for target vCenter.

 

Cause

vCenter Major Upgrade through Upgrade Runner mechanism, does not delete the old vCenter VM post upgrade.

As per VCF design guidelines, the administrator has to delete the old vCenter VM manually post the upgrade. https://docs.vmware.com/en/VMware-Cloud-Foundation/5.2/vcf-lifecycle/GUID-3B41CF79-C721-4AFC-A263-0672143DF41E.html#GUID-A2E3B776-F64A-42B5-8266-51FF74B96D6C__GUID-13BC04AD-A851-46DD-9DBB-114F609B1551 (Refer Step 9 in Procedure).

If stale vCenter VM is still present during the next upgrade, precheck/upgrade workflow can fail when fetching the DNS Server name or vCenter name. The DNS Server details/source vCenter name is fetched based on IP or FQDN of vCenter. If old vCenter is still present, sometimes old VM reference can be returned resulting in subsequent failure.

 

Resolution

Manually delete the old stale vCenter VM before initiating the vCenter Upgrade workflow. Clean up stale objects and ensure only one VM has the DNS name of the required vCenter.

i.e. Delete any and all stale vCenter's from previous upgrades.