vCSA 7 patch/upgrade failed for vtsdb "Exception('Inplace pg_upgrade failed')"
search cancel

vCSA 7 patch/upgrade failed for vtsdb "Exception('Inplace pg_upgrade failed')"

book

Article ID: 313300

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
vCenter 7 patch/upgrade failed with "Exception('Inplace pg_upgrade failed').

/var/log/vmware/applmgmt/PatchRunner.log
YYYY-MM-DDTHH:MM:SS vtsdb:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vtsdb:Patch' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
    result = self.extension(*args)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
    return func(*args)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/payload/components-script/vtsdb/__init__.py", line 144, in patch
    raise Exception('Inplace pg_upgrade failed')
Exception: Inplace pg_upgrade failed
YYYY-MM-DDTHH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
    _patchComponents(ctx, userData, statusAggregator.reportingQueue)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
    executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
    reportQueue, identifier, expectedResultType)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
    result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
  File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
YYYY-MM-DDTHH:MM:SS WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SS ERROR __main__ Patch vCSA failed

/var/log/vmware/vtsdb/upgrade/pg_upgrade_internal.log

Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
connection to database failed: could not connect to server: Connection refused
        Is the server running locally and accepting
        connections on Unix domain socket "/var/run/vtsdb/.s.PGSQL.5433"?

/var/log/vmware/vtsdb/upgrade/pg_upgrade_server.log

command: "/opt/vmware/vpostgres/current/../12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/storage/vtsdb/vtsdbdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....YYYY-MM-DDTHH:MM:SS 629abcf5.74fe 0 FATAL: pre-existing shared memory block (key 5433001, ID 32769) is still in use
YYYY-MM-DDTHH:MM:SS 629abcf5.74fe 0 HINT: Terminate any old server processes associated with data directory "/storage/vtsdb/vtsdbdata".
 stopped waiting
pg_ctl: could not start server


Environment

VMware vCenter Server 7.0.x

Cause

vmon was issuing SIGTERM StopSignal(as default) to shutdown vtsdb during the upgrade, which was leading to a force kill of vtsdb as a result of shutdown process timeout.

Resolution

This issue is fixed in vCenter version 7.0 U3i.

In the fix provided, we are explicitly issuing SIGINT as StopSignal, which leads to fast shutdown, as compared to smart shutdown through SIGTERM which was issued earlier by default. We have also specified StopBefore param to shutdown vstats before vtsdb.
Smart Shutdown vs Fast Shutdown: https://www.postgresql.org/docs/current/server-shutdown.html