"Certificate validation failed during pre-upgrade check" while upgrading vCenter Server - "Regenerate certificates for SSO and try again"

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vCenter Server

Issue/Introduction

Symptoms:

Pre-upgrade check fails with error "Certificate validation failed during pre-upgrade check"

In the vCenter logs location var/log/vmware/upgrade/requirements-upgrade-runner.log file, the following are comparable entries:

'description': {'id': 'upgrade.sso.precheck.error.description', 'translatable': 'Certificate has expired', 'localized': 'Certificate has expired'}, 'problemId': None, 'resolution': {'id': 'upgrade.sso.precheck.error.resolution', 'translatable': 'Regenerate certificates for sso and try again', 'localized': 'Regenerate certificates for sso and try again'}}]}},

Pre-check fails when upgrading from vCenter 7 to 8 or 9 or 8 to 9

Legacy Lookup service certificate is "Expired"

Cause

An expired 7444 certificate, likely the lookup service certificate, stored in the STS_INTERNAL_SSL_CERT store can be replaced with the machine certificate from the MACHINE_SSL_CERT store using the vCert utility. This process ensures proper functioning of the vCenter Server and its communication with other services.

Pre-upgrade checks were introduced in vCenter Server 7.0 Update 1 and later to identify and resolve known SSO database issues on the vCenter Server Appliance prior to upgrade. The expired 7444 certificate stored in the STS_INTERNAL_SSL_CERT with the machine cert from the MACHINE_SSL_CERT store caused the issue.

Resolution

There are two available options to address this issue:

OPTION 1:

To resolve this issue, replace the STS_INTERNAL_SSL_CERT with the machine cert from the MACHINE_SSL_CERT store.

Process to replace STS_INTERNAL_SSL_CERT with machine cert from MACHINE_SSL_CERT store:

Check if the lookup certificate is expired by running this command:
openssl s_client -connect <PSC/VCSA-FQDN/IP>:7444 | less
1. From the about output copy the contents which starts with "-----BEGIN CERTIFICATE-----" till the "-----END CERTIFICATE-----".
2. Save this file as 7444-lookup.txt and go to that location and rename this file extension as 7444-lookup.crt
3. Now open this 7444-lookup.crt file and check if the certificate is valid or expired.
4. If its expired then follow the Step 2.
Replace the expired 7444 certificate stored in the STS_INTERNAL_SSL_CERT with the machine cert from the MACHINE_SSL_CERT store. Implement the commands below as you see them one by one:

Appliance:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/MachineSSL.crt --key /var/tmp/MachineSSL.key

Windows:

Open Command prompt as administrator and navigate to cd C:\Program Files\VMware\vCenter Server\vmafdd

vecs-cli.exe entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\MachineSSL.crt
vecs-cli.exe entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\MachineSSL.key
vecs-cli.exe entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\sts_internal_backup.crt
vecs-cli.exe entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\sts_internal_backup.key
vecs-cli.exe entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
vecs-cli.exe entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert C:\Windows\Temp\MachineSSL.crt --key C:\Windows\Temp\MachineSSL.key

Restart the vCenter Services.
- service-control --stop --all && service-control --start --all
Re-run the openssl mentioned in Step #1 to validate if the lookup service certificate is valid.
And then retry the VC upgrade.
If Lookup service certificate is valid and you are facing the same issue, check certificates in VECS as well as SSO endpoints for possible expiration and replace them.
- Check if the lookup service is valid by running the below command again
  openssl s_client -connect <PSC/VCSA-FQDN/IP>:7444 | less

OPTION 2:

Remove the stale port 7444 by running the lsdoctor tool stalefix option python lsdoctor.py -s

Take snapshot of vCenter
Download the lsdoctor tool in Using the 'lsdoctor' Tool
Copy and extract lsdoctor to the filesystem of the affected node
Run “python lsdoctor.py -s”
- Verify that you have taken the appropriate snapshots
- Provide the password for your SSO administrator account
Once the script completes, restart all services
service-control --stop --all && service-control --start --all

Note: Re-register any external solutions that were previously pointed to the affected node (SRM, vSphere Replication, NSX-V, etc. – See product documentation for instructions)

If this article didn’t solve your issue, kindly raise a case with Broadcom support.

Additional Information

Latest version of lsdoctor takes care of updating STS_INTERNAL_SSL_CERT. In case an old version of lsdoctor.py is used, use the below manual steps to remove STS_INTERNAL_SSL_CERT from VECS

Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH:

Download the script named Delete_ STS_INTERNAL_SSL_CERT.sh attached to Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH
Provide permission to the script file with the command chmod +rx Delete_ STS_INTERNAL_SSL_CERT.sh
Run script ./Delete_ STS_INTERNAL_SSL_CERT.sh
Services will restart automatically