Patching fails with "Exception occured in postInstallHook" and vMON service fails to start.

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• VAMI fails at 80% and displays error, "exception occurred in postInstallHook"
• Logs located at /var/log/vmware/applmgmt/PatchRunner.log show similar messages to below

YYYY-MM-DDTHH:MM:SS INFO service_manager Service vmware-vmon reported status stopped. Expected status started 
YYYY-MM-DDTHH:MM:SS ERROR service_manager Service cannot be started. Error: 
YYYY-MM-DDTHH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got unhandled exception.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 84, in _patchComponents
    _startDependentServices(c)
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 53, in _startDependentServices
    serviceManager.start(depService)
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/libs/sdk/service_manager.py", line 901, in wrapper
    return getattr(controller, attr)(*args, **kwargs)
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/libs/sdk/service_manager.py", line 794, in start
    super(VMwareServiceController, self).start(serviceName)
  File "/storage/updatemgr/software-updatebv9bhw1d/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
    raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: 
YYYY-MM-DDTHH:MM:SS WARNING root stopping status aggregation... 
YYYY-MM-DDTHH:MM:SS ERROR __main__ Patch vCSA failed


Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment

Environment

VMware vCenter Server 7.0.x

Cause

Issue is caused by duplicate service configuration file under /etc/vmware/vmware-vmon/svcCfgFiles directory. The duplicate files are created during the upgrade process if the original service configuration file was edited. This is most commonly caused by increasing the timeout updatemgr.json ( vCenter Server upgrade from 7.0 U2 to 7.0 U3c fails with error "Exception occured in postInstallHook")

ls -ltr /etc/vmware/vmware-vmon/svcCfgfiles
-rw-r--r-- 1 root            root 1021 May 25 13:45 updatemgr.json
-r--r--r-- 1 root            root 1064 May 25 14:06 vsan-health.json
-r--r--r-- 1 root            root  977 May 25 14:48 imagebuilder.json
-rw-r--r-- 1 root            root  981 May 26 08:11 updatemgr.json.rpmsave

Expected timeout value:

grep -i timeout /etc/vmware/vmware-vmon/svcCfgfiles/updatemgr.json
   "StartTimeout" : 630,
   "StopTimeout" : 10,

   
Altered timeout value:

grep -i timeout /etc/vmware/vmware-vmon/svcCfgfiles/updatemgr.json
   "StartTimeout" : 1500,
   "StopTimeout" : 10,

Resolution

This issue is resolved in VMware vCenter Server release 7.0 U3f. 


Workaround:

To workaround this issue follow the below steps:

1.Revert/Restore the vCenter to a healthy state prior to upgrade attempt.

2.Ensure you have a current snapshot/backup of the vCenter.

3.Remove the altered json file from /etc/vmware/vmware-vmon/svcCfgFiles/ directory  (The updatemgr configuration file will be re-created during the vCenter upgrade/patching)

mv /etc/vmware/vmware-vmon/svcCfgFiles/updatemgr.json /root/

4.Run the upgrade/Patch again on the vCenter.