vCenter Upgrade fails while attempting to start VMware Security Token Service

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

vCenter Upgrade fails during stage 2 with error:

A problem occurred while - Starting VMware Security Token Service...
Upgrade logs reporting the following error:

Error
Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1170, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 281, in boot self.reregisterSTSUsingLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 435, in reregisterSTSUsingLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 431, in reregisterSTSUsingLookupService ls_obj.reregister_service(service_id, mutable_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 350, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 366, in reregister_service self.service_content.serviceRegistration.Set(svc_id, svc_set_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [], invalidProperty = 'Invalid certificate' }
Resolution
This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

Cause

The issue is related to an SSL Trust Anchors mismatch.

Resolution

To resolve the issue follow the below options to update the SSL Trust Anchors

Please ensure you have a valid VAMI-based backup or offline snapshots of ALL vCenter/PSC nodes in the SSO domain before continuing.

See VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

OPTION #1: VCF Certificate Management Utility (vCert)

For detailed, step-by-step instructions on downloading, running of the vCert tool, refer to the main Knowledge Base article linked below.

vCert - Scripted vCenter expired certificate replacement
On the main menu select option 4 (Manage SSL trust anchors)

VCF/VVF Certificate Management Utility (version 6.0.1)
-----------------------------------------------------------------
1. Check current certificate status
2. View certificate info
3. Manage certificates
4. Manage SSL trust anchors
5. Check configurations
6. Reset all certificates with VMCA-signed certificates
7. ESXi certificate operations
8. Restart services
9. Generate certificate report
E. Exit
On the Manage SSL Trust Anchors menu select option 1 to confirm the issue (Check SSL Trust Anchors)

Manage SSL Trust Anchors
-----------------------------------------------------------------
1. Check SSL Trust Anchors
2. Update SSL Trust Anchors
On the sub-menu select to have the Service IDs and endpoint URIs on the output to see which certificate is being used for each service registration/endpoint.
If we are hitting the issue we expect the output to show some endpoints using an invalid certificate (the output should be marked with a red colour).
If issue is confirmed return to the previous menu and select option 2 (Update SSL Trust Anchors)

Manage SSL Trust Anchors
-----------------------------------------------------------------
1. Check SSL Trust Anchors
2. Update SSL Trust Anchors

Note: Once the update is finished you can re-check the SSL Trust Anchors to validate that invalid certificate is reported.
Restart the vCenter services. You can do that either from the vCert Utility or by running the command: service-control --stop && service-control --start
Reattempt the upgrade

OPTION #2:- lsdoctor Utility

This issue can also be resolved by fixing the SSL trust mismatch issues in the lookup service.

For detailed, step-by-step instructions on downloading, running of the lsdoctor tool, refer to the main Knowledge Base article linked below.

Using the 'lsdoctor' Tool
Run the lsdoctor utility using the -t or --trustfix option. This command will diagnose and automatically correct SSL trust issues within the Lookup Service configuration.
Reattempt the upgrade