vCSA 7 patch/upgrade failed for vtsdb "Exception('Inplace pg_upgrade failed')"
Article ID: 313300
Updated On:
Products
Issue/Introduction
Symptoms:
vCenter 7 patch/upgrade failed with "Exception('Inplace pg_upgrade failed').
/var/log/vmware/applmgmt/PatchRunner.log
YYYY-MM-DDTHH:MM:SS vtsdb:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vtsdb:Patch' failed.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/payload/components-script/vtsdb/__init__.py", line 144, in patch
raise Exception('Inplace pg_upgrade failed')
Exception: Inplace pg_upgrade failed
YYYY-MM-DDTHH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
reportQueue, identifier, expectedResultType)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
YYYY-MM-DDTHH:MM:SS WARNING root stopping status aggregation...
YYYY-MM-DDTHH:MM:SS ERROR __main__ Patch vCSA failed
/var/log/vmware/vtsdb/upgrade/pg_upgrade_internal.log
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
connection to database failed: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/run/vtsdb/.s.PGSQL.5433"?
/var/log/vmware/vtsdb/upgrade/pg_upgrade_server.log
command: "/opt/vmware/vpostgres/current/../12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/storage/vtsdb/vtsdbdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....YYYY-MM-DDTHH:MM:SS 629abcf5.74fe 0 FATAL: pre-existing shared memory block (key 5433001, ID 32769) is still in use
YYYY-MM-DDTHH:MM:SS 629abcf5.74fe 0 HINT: Terminate any old server processes associated with data directory "/storage/vtsdb/vtsdbdata".
stopped waiting
pg_ctl: could not start server
Environment
Cause
Resolution
In the fix provided, we are explicitly issuing SIGINT as StopSignal, which leads to fast shutdown, as compared to smart shutdown through SIGTERM which was issued earlier by default. We have also specified StopBefore param to shutdown vstats before vtsdb.
Smart Shutdown vs Fast Shutdown: https://www.postgresql.org/docs/current/server-shutdown.html
Feedback
