"Exception occured in postInstallHook", Secure Token Service (vmware-stsd) crash while performing vCenter Server Patching to 7.0 U3f
search cancel

"Exception occured in postInstallHook", Secure Token Service (vmware-stsd) crash while performing vCenter Server Patching to 7.0 U3f

book

Article ID: 322182

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • vCenter Server patching to 7.0 U3f (Build 20051473) fails with error message "Exception occured in postInstallHook" in VAMI page.
  • Patching VCSA through CLI fails with error "An error occured while starting sts"
  • Upgrading vCenter Server from 6.x to 7.0 U3f will fail with "A problem occurred while - Starting VMware Security Token Service"
  • VCSA is joined to Active Directory domain and is currently using, or has used in the past, Integrated Windows Authentication for identity source
  • Log file /var/log/vmware/applmgmt/Patchrunner.log will show similar to below entries :
YYYY-MM-DD HH:MM ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/seat/software-updatew2oofv0c/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service sts. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "sts"
            ],
            "localized": "An error occurred while starting service 'sts'"
        }
  • STS runtime log file log/vmware/sso/sts-runtime.log.stderr will show below error message :
Starting service process with pid: 38715.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M     -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true     -Dorg.apache.xml.security.ignoreLineBreaks=true
free(): corrupted unsorted chunks

Log file will show any of the memory corruption errors mentioned below :
free(): corrupted unsorted chunks

double free or corruption (!prev)

corrupted size vs. prev_size

free(): invalid next size (normal)
  • /var/core directory will show core dump files like below :
core.Thread-2.38715
core.Thread-2.54064



Environment

VMware vCenter Server 7.0.3

Cause

This issue is caused due to memory corruption in Secure Token Service (vmware-stsd) when VCSA is joined to Active Directory and is currently using, or has used in the past, Integrated Windows Authentication for identity source.

Resolution

This issue is resolved in VMware vCenter Server 7.0 U3g build 20150588, available at Broadcom Downloads.

Note:

  • vCenter Server needs to be reverted to a healthy state to apply this patch, applying a patch in already failed state is not feasible.
  • CVE-2021-22048 stands as un-resolved in 7.0 U3g as well, please refer to VMSA-2021-0025 for more information.



Workaround:
To workaround this issue, the identity source configuration needs to be changed from IWA to "AD over LDAP" or "AD over LDAPs" and remove the VCSA from AD domain.

You may follow the below steps to change the identity source configuration :

  • Revert/restore vCenter Server to a healthy state, which was taken before initiating the patch
  • Remove the IWA Identity Source configuration
  • Remove vCenter Server from the Active Directory domain.
/opt/likewise/bin/domainjoin-cli leave
  • Reboot the vCenter Server
  • Add the identity source as "AD over LDAP" or "AD over LDAPS"
  • Retry the Patching

Note:

  • This workaround cannot be applied in the Patching failed state, vCenter needs to be reverted to a healthy state, then apply the workaround.
  • You may move back to IWA (Integrated Windows Authentication) Identity Source after patching to VMware vCenter Server 7.0 U3g build 20150588
  • Detailed steps for each action is available in the documents mentioned in Related Information



Additional Information

Powered by Wolken Software