"Exception occurred in postInstallHook" error when upgrading vCenter Server from 7.0 U2 to 7.0 U3c
search cancel

"Exception occurred in postInstallHook" error when upgrading vCenter Server from 7.0 U2 to 7.0 U3c

book

Article ID: 305766

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • In the /var/log/vmware/vmon/vmon.log you see Connection refused errors while performing a health check against the imagebuilder service

2022-03-17T16:09:44.293Z In(05) host-12345 <imagebuilder> Running the API Health command as user imagebuilder
2022-03-17T16:09:44.293Z In(05) host-12345 <imagebuilder-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n imagebuilder -u /vmw/imagebuilder/httpd/healthstatus -t 30
2022-03-17T16:09:44.457Z Wa(03) host-12345 <imagebuilder> Service api-health command's stderr: Exception while retrieving health xml from url http://localhost:8099/vmw/imagebuilder/httpd/healthstatus. Exception: <urlopen error [Errno 111] Connection refused>

  • After 240 seconds vmon terminates the imagebuilder service, since it deems it unresponsive:
2022-03-17T16:13:43.185Z In(05) host-49155 <imagebuilder> Running the API Health command as user imagebuilder
2022-03-17T16:13:43.185Z In(05) host-49155 <imagebuilder-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n imagebuilder -u /vmw/imagebuilder/httpd/healthstatus -t 30
2022-03-17T16:13:43.293Z In(05) host-49155 <imagebuilder> Service start operation timed out.
2022-03-17T16:13:43.293Z In(05) host-49155 <imagebuilder-healthcmd> Cancelling execution of pid 44601
2022-03-17T16:13:43.293Z Wa(03) host-49155 <imagebuilder> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
  • The /var/log/vmware/applmgmt/PatchRunner.log shows the following failure:
2022-03-17T16:14:14.609Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 794, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/core/software-packages/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service imagebuilder. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "imagebuilder"
            ],
            "localized": "An error occurred while starting service 'imagebuilder'"

2022-03-17T17:11:48.598+01:00 warning vpxd[55330] [Originator@6876 sub=Default opID=63f1aadc] [VdbStatement] SQL execution took too long:  SELECT EVENT_ID, CHAIN_ID, EVENT_TYPE, EXTENDED_CLASS, CREATE_TIME, USERNAME, CATEGORY, VM_ID,        VM_NAME, HOST_ID, HOST_NAME, COMPUTERESOURCE_ID, COMPUTERESOURCE_TYPE,        COMPUTERESOURCE_NAME, DATACENTER_ID, DATACENTER_NAME, DATASTORE_ID, DATASTORE_NAME, NETWORK_ID,        NETWORK_NAME, NETWORK_TYPE, DVS_ID, DVS_NAME, STORAGEPOD_ID, STORAGEPOD_NAME, CHANGE_TAG_ID FROM VPXV_EVENT_ALL  WHERE CREATE_TIME>=? AND EVENT_ID<? ORDER BY EVENT_ID DESC LIMIT 10;

Environment

VMware vCenter Server 7.0.x

Cause

This issue can occur if there are a large number of entries in some tables in the vCenter Server database e.g. vpxv_event_all. If there are slow queries serialized on these tables after the vpxd service starts it can block the updateExtension request for the imagebuilder service. This causes the service startup failure and results in the upgrade failure.


Resolution

This is a known issue and currently there is no resolution.

Workaround:

To workaround this issue, please follow the below steps:

Based on the feedback from the database size and the tables involved, a clean up of the database will be required. If the table impacted is the VPXV_EVENT_ALL as per the example above you can follow the steps in the following KB to remove those entries: /storage/seat disk 100% full on vCenter Server Appliance 6.x/7.x

If there are other tables impacted or you are unsure of the steps to clean up these entries then please contact VMware Support and note this Article ID (88940) in the problem description. For more information, see Creating and managing Broadcom support cases

Additional Information

Powered by Wolken Software