Patching vCenter Server report "Exception occurred in postInstallHook" due to "vmware-certificatemanagement" service failing to start

Products

• VMware vCenter Server

Issue/Introduction

• Attempting to update vCenter Server to version 8.0 Update 3h Build 25092719 using the vCenter Server Appliance Management Interface (VAMI), the operation fails with the following error:
  "Exception occurred in postInstallHook for B2B-patching. Please check the logs for more details. Take corrective action and then resume".

• /var/log/vmware/sso/vmware-identity-sts.log: (on vCenter Server)

YYYY-MM-DDTHH:MM:SS INFO sts[74:tomcat-http--39] [CorId=<>] [com.vmware.identity.sts.impl.STSImpl] Entering issue() token...
YYYY-MM-DDTHH:MM:SS INFO sts[74:tomcat-http--39] [CorId=<>] [com.vmware.identity.idm.server.IdentityManager] User <vCenter Server FQDN@SSO_Domain> attempting to login via unsupported domain provider <SSO_Domain> type com.vmware.identity.idm.server.provider.vmwdirectory.SystemDomainAliasedProvider on federated tenant <SSO_Domain>.  This is not supported.
YYYY-MM-DDTHH:MM:SS ERROR sts[74:tomcat-http--39] [CorId=<>] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [<vCenter Server FQDN@SSO_Domain>] for tenant [<SSO_Domain>]

• /var/log/vmware/applmgmt/PatchRunner.log: (on vCenter Server)

  stderr:

  YYYY-MM-DDTHH:MM:SS INFO service_manager Starting service 'vmware-certificatemanagement' ...
YYYY-MM-DDTHH:MM:SS INFO service_manager Executing command '[['/bin/service-control', '--start', 'vmware-certificatemanagement']]'
YYYY-MM-DDTHH:MM:SS INFO service_manager Command '[['/bin/service-control', '--start', 'vmware-certificatemanagement']]' has exit-code='1' and stdout: Operation not cancellable. Please wait for it to finish...
Performing start operation on service certificatemanagement...
stderr: Error executing start on service certificatemanagement. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "certificatemanagement"
            ],
            "localized": "An error occurred while starting service 'certificatemanagement'"
  
  

• /var/log/vmware/certificatemanagement/certificatemanagement_prestart.log (on vCenter Server)

YYYY-MM-DDTHH:MM:SS  Done running command
Traceback (most recent call last):
  File "/usr/lib/vmware-certificatemanagement/scripts/certificatemanagement_prestart.py", line 252, in <module>
pyVim.sso.SoapException: SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Invalid credentials
faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

 

Environment

vCenter 8.x

vCenter 9.x

Cause

This issue is caused by the persistence of a legacy identity source SYSTEM-DOMAIN, following a historical migration from vCenter Server 5.1 to version 5.5 or later. 

Resolution

Note: Revert the snapshot/restore from backup for vCenter Server before proceeding with the steps

To resolve the issue, execute the update_system_domain.py script to clear the legacy entries. Refer to Fix legacy SYSTEM-DOMAIN artifacts in vCenter Servers upgraded from vCenter Server 5.1+