vCenter Server upgrade precheck via SDDC manager fails

Products

• VMware SDDC Manager

Issue/Introduction

• As part of VCF stack upgrade from 5.x to VCF 9.x, performing upgrade of vCenter from 8.x to 9.x fails at precheck state with one of below errors.
  
  
  • "Could not find VM with address <VC_FQDN> in vCenter <VC_FQDN> to verify free space in '/'."




  • "vCenter <VC_FQDN> multiple VMs check failed internally"

Environment

VCF 5.x

Cause

• The issue occurs due to case mismatch between the vCenter Server PNID stored on vCenter and the vCenter hostname stored in the SDDC Manager postgres database.
• The issue originally occurs if there is a mismatch between the vCenter hostname and vCenter PNID. In this case, when the vCenter is connected to an SDDC Manager, the 'vm_hostname' column in 'vcenter' table gets updated with the 'hostname' value of the vCenter causing a mismatch between the vCenter PNID and the SDDC Manager database entry.
  
  
• To validate PNID on vCenter Server:
  1. SSH to VCenter Server as root. Enter shell.
  2. Run the below command to find the PNID:
     /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

sample output:
VCSA02.example.com


• To validate the vCenter hostname in SDDC postgres database:
  1. SSH to SDDC Manager as vcf user. Run su to login as root.
  2. Find the vCenter hostname in SDDC database:
     psql -h localhost -U postgres -d platform -c "select * from vcenter;"

                 sample output

                    id                  |     vm_hostname    | status
        --------------------------------------+---------------------+--------
        2730####-####-####-########d780 | vcsa02.example.com | ACTIVE
        (1 row)

Resolution

For PNID and hostname mismatch on vCenter caused due to case difference, updating the PNID on vCenter Server using VAMI will fail with error 'Adding new CN entries failed'.

Follow the steps below to resolve the issue:

1. Take a snapshot of the vCenter Server Virtual Machine. If the vCenter is in enhanced linked mode, follow VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.
2. Login to the vCenter Appliance Management Interface (https://<VCSA_FQDN>:5480) as root.
3. Navigate to Networking -> Network Settings -> Edit.
4. Update the hostname with a placeholder. Example: localhost
5. Reboot the vCenter Server. Do not skip this step as this is necessary to update the Likewise registry and restart the underlying services.
6. Update the hostname that matches the value in the SDDC postgres database. Example: vcsa02.example.com