Patching vCenter from 7.0 to 7.0U3x fails with Exception occurred in postInstallHook

Products

• VMware vCenter Server

Issue/Introduction

From /var/log/vmware/applmgmt/Patchrunner.log of the failed vCenter, below logs snippets are available:

YYYY-MM-DDTHH:MM:SS  Done running command, stderr: YYYY-MM-DDTHH:MM:SS  pg_upgrade --check returned error code 1 with error  out Performing Consistency Checks on Old Live Server
------------------------------------------------
YYYY-MM-DDTHH:MM:SS  Done running command
YYYY-MM-DDTHH:MM:SS  Running: su -s /bin/bash vpostgres -c cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/13/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/13/bin -b /opt/vmware/vpostgres/13/../11/bin -D /storage/db/vpostgres.13 -d /storage/db/vpostgres --check
YYYY-MM-DDTHH:MM:SS  Running command: ['su', '-s', '/bin/bash', 'vpostgres', '-c', 'cd /var/log/vmware/vpostgres/upgrade && /opt/vmware/vpostgres/13/bin/pg_upgrade -U postgres -B /opt/vmware/vpostgres/13/bin -b /opt/vmware/vpostgres/13/../11/bin -D /storage/db/vpostgres.13 -d /storage/db/vpostgres --check']
YYYY-MM-DDTHH:MM:SS  Done running command, stderr: YYYY-MM-DDTHH:MM:SS  pg_upgrade --check returned error code 1 with error  out Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions                                   ok
connection to database failed: fe_sendauth: no password supplied
Failure, exiting
.
YYYY-MM-DDTHH:MM:SS vpostgres:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vpostgres:Patch' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/payload/components-script/vpostgres/__init__.py", line 183, in patch
    'PostgreSQL failed to run upgrade command.')))
patch_errors.UserError: PostgreSQL failed to run upgrade command.
YYYY-MM-DDTHH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/updatemgr/software-updateckp7d8cf/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
YYYY-MM-DDTHH:MM:SS WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SS ERROR __main__ Patch vCSA failed

Cause

This issue occurs if there are any leftover files from previous failed/successful update in the vCenter in the below path.

/storage/updatemgr/software-updates/7.0.3.xxxxx/scripts/pre-install.py

Resolution

1. Take Snapshot of the vCenter (Offline Snapshots of the vCenters in ELM).

2. Delete the below mentioned file.
/storage/updatemgr/software-updates/7.0.3.xxxxx/scripts/pre-install.py

3. Restart the patching once again.

Additional Information

This will not be an issue from 7.0U3x onwards. As the update files are removed after the update.