VCF 5.x Pre-check Failure: "VM free space check failed internally"

Products

• VMware SDDC Manager

Issue/Introduction

• vCenter Major Upgrade through Upgrade Runner mechanism fails during bundle precheck/upgrade with error: vCenter precheck failed while getting the DNS server from VM.
• vCenter Patch Upgrade through RDU with STATIC network configuration can fail during upgrade with error: vCenter upgrade failed at the VCENTER_UPGRADE_PRECHECK stage
• From the SDDC Manager UI vCenter Pre-check you may see the following Error: "Validates vCenter VM's free space on root"
  
  

 

• Check the space on each component with "df -h" and/or "du -sh *" on any specific partition/directory where you may notice a space issue. 
• You may observe from the /var/log/vmware/vcf/operationsmanager/operationsmanager.log the following snippet:

  YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,#################,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-23] Searching for VM with address <vcenter fqdn>
  YYYY-MM-DDTHH:MM:SS.sssZ DEBUG [vcf_om,#################,####] [c.v.e.s.c.c.v.vsphere.VcManagerBase,om-exec-23] Found VM <vcenter hostname>#.#.#.#.####-####### with address <vcenter fqdn>
  ........
  
  YYYY-MM-DDTHH:MM:SS.sssZ ERROR [vcf_om,#################,####] [c.v.v.b.p.t.v.SimpleNativeValidator,om-exec-23] Native validator with name 'mgmt-vm-free-space-validator' failed internally
  . Reference token: ######
  java.lang.NullPointerException: Cannot read the array length because "array" is null

Environment

3.x to 4.x upgrade followed by 4.x to 5.2.1 Upgrade

4.x to 5.x upgrade followed by 5.x to 5.2.1 upgrade using RDU (Reduced Downtime Migration) mechanism and STATIC network configuration for target vCenter.

 

Cause

vCenter Major Upgrade through Upgrade Runner mechanism, does not delete the old vCenter VM post upgrade. As per VCF design guidelines, the administrator has to delete the old vCenter VM manually post the upgrade. 

Upgrade vCenter Server for VMware Cloud Foundation 5.2.x

(Refer to the Note in Step 8 of the Procedure).

If stale vCenter VM is still present during the next upgrade, precheck/upgrade workflow can fail when fetching the DNS Server name or vCenter name. The DNS Server details/source vCenter name is fetched based on IP or FQDN of vCenter. If old vCenter is still present, sometimes old VM reference can be returned resulting in subsequent failure.

 

Resolution

To resolve this issue, you must manually clean up all stale vCenter objects and ensure that only the active production vCenter is associated with the required DNS name before restarting the upgrade process. Step-by-Step Instructions:

1. Log in to the ESXi host (or the management vCenter) where the vCenter VMs reside. Search the inventory for any powered-off or orphaned vCenter VMs left over from previous upgrade attempts.
2. Check the configuration of the suspected stale VMs to confirm they are using the FQDN or IP address of your production vCenter Server.
3. Delete Stale VMs by right-clicking the stale vCenter VM. Select Delete from Disk to permanently remove the stale VM and its associated files from the datastore. Repeat this for any and all stale vCenter VMs in the environment.
4. Ensure that only one IP address resolves to the DNS name of the required vCenter Server. Delete any duplicate or stale records.
5. Retry the Upgrade once the environment is clean and DNS resolution returns a single, correct host, initiate the vCenter Upgrade workflow again.