"Certificate Validation Failure During Pre-Check for vCenter Server Upgrade - Regenerate certificates for SSO and try again"

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Symptoms:

• Pre-upgrade check fails with error "Certificate validation failed during pre-upgrade check"

• In the requirements-upgrade-runner.log file, you see entries similar to:

'description': {'id': 'upgrade.sso.precheck.error.description', 'translatable': 'Certificate has expired', 'localized': 'Certificate has expired'}, 'problemId': None, 'resolution': {'id': 'upgrade.sso.precheck.error.resolution', 'translatable': 'Regenerate certificates for sso and try again', 'localized': 'Regenerate certificates for sso and try again'}}]}},

 

• Pre-check fails when upgrading from vCenter 7 to 8

     

• Legacy Lookup service certificate is "Expired"

     

Resolution

OPTION 1:

To resolve this issue, replace the STS_INTERNAL_SSL_CERT with the machine cert from the MACHINE_SSL_CERT store.

Process to replace STS_INTERNAL_SSL_CERT with machine cert from  MACHINE_SSL_CERT store:

1. Check if the lookup certificate is expired by running this command:
   
   openssl s_client -connect <PSC/VCSA-FQDN/IP>:7444 | less
   1. From the about output copy the contents which starts with "-----BEGIN CERTIFICATE-----"  till the "-----END CERTIFICATE-----".
   2. Save this file as 7444-lookup.txt and go to that location and rename this file extension as 7444-lookup.crt
   3. Now open this 7444-lookup.crt file and check if the certificate is valid or expired.
   4. If its expired then follow the Step 2.
2. Replace the expired 7444 certificate stored in the STS_INTERNAL_SSL_CERT with the machine cert from the MACHINE_SSL_CERT store. Implement the commands below as you see them one by one:

Appliance:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT > /var/tmp/MachineSSL.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT > /var/tmp/sts_internal_backup.key
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert /var/tmp/MachineSSL.crt --key /var/tmp/MachineSSL.key

Windows:

Open Command prompt as administrator and navigate to cd C:\Program Files\VMware\vCenter Server\vmafdd

vecs-cli.exe entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\MachineSSL.crt
vecs-cli.exe entry getkey --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\MachineSSL.key
vecs-cli.exe entry getcert --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\sts_internal_backup.crt
vecs-cli.exe entry getkey --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --output C:\Windows\Temp\sts_internal_backup.key
vecs-cli.exe entry delete --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT -y
vecs-cli.exe entry create --store STS_INTERNAL_SSL_CERT --alias __MACHINE_CERT --cert C:\Windows\Temp\MachineSSL.crt --key C:\Windows\Temp\MachineSSL.key

3. Restarted the vCenter Services.
   • service-control --stop --all
   • service-control --start --all
4. Re-run the openssl mentioned in Step #1 to validate if the lookup service certificate is valid.
5. And then retry the VC upgrade.
6. If Lookup service certificate is valid and you are facing the same issue, check certificates in VECS as well as SSO endpoints for possible expiration and replace them.
   • Check if the lookup service is valid by running the below command again
   • openssl s_client -connect <PSC/VCSA-FQDN/IP>:7444 | less
     
     
     
     

OPTION 2:

1. Remove the stale port 7444 by running the lsdoctor tool stalefix option python lsdoctor.py -s

 - Take snapshot of vCenter
 - Download the lsdoctor tool in KB 
 - copy and extract lsdoctor to the filesystem of the affected node
 - Run “python lsdoctor.py -s”
 - Verify that you have taken the appropriate snapshots
 - Provide the password for your SSO administrator account

Follow up actions needed:

- Once the script completes, restart all services

    service-control --stop --all && service-control --start --all

- Re-register any external solutions that were previously pointed to the affected node (SRM, vSphere Replication, NSX-V, etc. – See product documentation for instructions)

 

2. Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH 


 - Download the script named Delete_ STS_INTERNAL_SSL_CERT.sh attached to KB
 - Provide permission to the script file with the command chmod +rx Delete_ STS_INTERNAL_SSL_CERT.sh
 - Run script  ./Delete_ STS_INTERNAL_SSL_CERT.sh
 - Services will restart automatically

If this article didn’t solve your issue, kindly raise a case with Broadcom support or :

Ask for help here : Click for Broadcom Communities