vCenter Upgrade to 9.x fails with "A problem occurred while - Starting VMware vStats Service" due to Host SSL thumbprint mismatch

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server upgrade to 9.x fails at Stage 2 with an error similar to below :
  
  A problem has occurred. The source vCenter might have been Powered Off during this process. A problem occurred while - Starting VMware vStats Service...
  
  
• On the vCenter 9.x appliance, the vpxd service is found to create coredumps at /var/core and crashes. When manually starting vpxd, it eventually crashes with the following entries in /var/log/vmware/vpxd/vpxd.log :
  
  YYYY-MM-DDThh:mm:ss ERROR vpxd #### [vc@#### sub="HostAccess" opId="####-####"] The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate
  
  
• Additionally the /var/log/firstboot/vstats-firstboot.py_####_stderr.log displays firstboot failure similar to below:
  
  YYYY-MM-DDThh:mm:ss ERROR starting vstats rc: 4, stdout: , stderr: Start service request failed. Error: A system error occurred. Check logs for details
Traceback (most recent call last):
  File "/usr/lib/vmware-vstats/firstboot/vstats-firstboot.py", line 183, in Main
    vstatsvc_fb.start_vstatsvc()
...
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vstats"
            ],
            "localized": "An error occurred while starting service 'vstats'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
...
VStats Service firstboot failed due to <cis.componentStatus.ErrorInfo object>

Environment

vCenter 9.x

Cause

This issue occurs because of an SSL certificate thumbprint mismatch between the ESXi host and the vCenter Server database (VCDB) records prior to the upgrade.

Resolution

1. Power down and remove the failed 9.x vcenter virtual machine and power on the source vCenter appliance

2. Identify the hosts showing a discrepancy between the expected and actual host SSL thumbprints.

   1. Retrieve the SSL thumbprints for the hosts in the VCDB by running the following command on the source vCenter Server appliance:
      
      /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "SELECT dns_name, host_ssl_thumbprint, expected_ssl_thumbprint FROM vpx_host;"

   2. Retrieve the actual Host SSL thumbprint from the vCenter by using OpenSSL command:
      
      openssl s_client -connect <esxi_fqdn>:443  </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha1
      
      
3. Update the certificates on the vCenter's database by either of the following methods depending on your environment and feasibility: 

   1. If the affected hosts are using VMCA-issued certificates and disconnecting is not feasible, regenerate the certificates for these hosts from the vCenter Server UI.

   2. If the affected hosts are using Custom certificates, disconnect and reconnect the hosts in the vCenter Server UI. Note: It is recommended to place the host into maintenance mode prior to disconnecting and reconnecting it, especially if vSAN is enabled.

4. Verify that the SSL thumbprints have been updated and there are no mismatches

5. Proceed with the vCenter Server upgrade to 9.x

Additional Information

• In case the vpxd crashes caused due to Host SSL thumprint mismatches are observed post successful upgrade to vCenter 9.x, refer to  Error: "The 'sslThumbprint' and 'sslCertificate' parameters are both set, but the thumbprint does not match the certificate"
  
  
• To compare all hosts stored in the VCDB against their live certificate thumbprints, run the following script on the vCenter Server appliance:
  
  /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -t -A -F"," -c "SELECT dns_name, ip_address, expected_ssl_thumbprint, host_ssl_thumbprint FROM vpx_host WHERE expected_ssl_thumbprint IS NOT NULL;" | while IFS=',' read -r hostname ip expected host_ssl; do
    # Fetch live thumbprint from the ESXi host
    actual=$(echo -n | openssl s_client -connect "${ip}:443" 2>/dev/null | openssl x509 -noout -fingerprint | cut -d'=' -f2)
    
    # Handle unreachable hosts
    if [ -z "$actual" ]; then
        echo "[ERROR] Host: $hostname ($ip) - Unreachable or certificate not returned."
        continue
    fi
    
    # Compare both VCDB thumbprints with the live host thumbprint
    if [ "$expected" == "$actual" ] && [ "$host_ssl" == "$actual" ]; then
        echo "[PASS] Host: $hostname ($ip) - All thumbprints match."
    else
        echo "[FAIL] Host: $hostname ($ip) - MISMATCH DETECTED"
        echo "       VCDB Expected  : $expected"
        echo "       VCDB Host_SSL  : $host_ssl"
        echo "       Host Actual    : $actual"
    fi
done