vCenter update fails at 93% with error "Exception occurred in postInstallHook for B2B-patching"

Products

• VMware vCenter Server

Issue/Introduction

• The update failed with the message: "Exception occurred in postInstallHook for B2B-patching. Please check the logs for more details. Take corrective action and then resume".

• The following log snippets are observed on the vCenter Server:

  • /var/log/vmware/applmgmt/PatchRunner.log.

YYYY-MM-DDTHH:MM INFO service_manager Command '[['/bin/service-control', '--start', 'vmware-certificatemanagement']]' has exit-code='1' and stdout: Operation not cancellable. Please wait for it to finish...
Performing start operation on service certificatemanagement...
stderr: Error executing start on service certificatemanagement. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "certificatemanagement"
            ],
            "localized": "An error occurred while starting service 'certificatemanagement'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
YYYY-MM-DDTHH:MM WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM ERROR __main__ Patch vCSA failed

• • /var/log/vmware/vmon/vmon.log

YYYY-MM-DDTHH:MM In(05) host-###### Received start request for certificatemanagement
YYYY-MM-DDTHH:MM In(05) host-###### <certificatemanagement-prestart> Constructed command: /usr/bin/python/usr/lib/vmware-certificatemanagement/scripts/certificatemanagement_prestart.py
YYYY-MM-DDTHH:MM Wa(03) host-###### <certificatemanagement> Service pre-start command's stderr: YYYY-MM-DDTHH:MM  Security error: (vmodl.fault.SecurityError)
YYYY-MM-DDTHH:MM Wa(03) host-###### <certificatemanagement> Service pre-start command's stderr: raise RuntimeError("generator didn't stop after throw()")
YYYY-MM-DDTHH:MM Wa(03)+ host-###### RuntimeError: generator didn't stop after throw()
YYYY-MM-DDTHH:MM Er(02) host-###### <certificatemanagement> Service pre-start command failed with exit code 1.
YYYY-MM-DDTHH:MM Wa(03) host-###### [ReadSvcSubStartupData] No startup information from certificatemanagement.

• • /var/log/vmware/sso/vmware-identity-sts.log.

YYYY-MM-DDTHH:MM ERROR sts[70:tomcat-http--24] [CorId=e246e4ac-####-####-####-0b96ad19b058] [com.sun.xml.ws.servlet.http] caught throwable javax.xml.ws.WebServiceException: javax.xml.bind.MarshalException
 - with linked exception:
[com.sun.istack.SAXException2: com.sun.istack.SAXException2: java.io.IOException: Broken pipe
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
javax.xml.transform.TransformerException: com.sun.istack.SAXException2: java.io.IOException: Broken pipe

Environment

vCenter 8.x

Cause

A broken Security Token Service (STS) pipeline caused the Certificate Management service to fail during startup, which subsequently led to the update process failing.

Resolution

   1. Validate the STS certificate validity by using vCert script and ensure the STS certificate is valid. Refer: vCert - Scripted vCenter expired certificate replacement

a. Run the vCert script and choose option 2 for "View certificate info" and option 8 for "STS signing certificates". 
b. If the STS certificate is expired, take backup of the vCenter Server Appliance. Run vCert script and choose option 3 for "manage certificates" and option 8 for "STS signing certificates". Proceed to renew the certificate with VMCA certificate.
c. If the STS certificate is valid, the issue is transient. Proceed with next step.

   2. Start application management (applmgmt) service.

a. SSH in to the vCenter Server Appliance as a root user.
b. Start applmgmt service by running command:
      service-control --start applmgmt

   3. In the VAMI update tab, choose "Resume vCenter update" radio button and click on Apply. The vCenter Server update will retry and proceed to succeed.