vCSA 7 patch/upgrade failed for vtsdb "Exception('Inplace pg_upgrade failed')"
Article ID: 313300
Updated On:
Products
VMware vCenter Server
Issue/Introduction
Symptoms:
vCenter 7 patch/upgrade failed with "Exception('Inplace pg_upgrade failed').
/var/log/vmware/applmgmt/PatchRunner.log
2022-06-04 02:01:25,190.190Z vtsdb:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vtsdb:Patch' failed.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/payload/components-script/vtsdb/__init__.py", line 144, in patch
raise Exception('Inplace pg_upgrade failed')
Exception: Inplace pg_upgrade failed
2022-06-04T02:01:25.195Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
reportQueue, identifier, expectedResultType)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
2022-06-04T02:01:26.197Z WARNING root stopping status aggregation...
2022-06-04T02:01:26.197Z ERROR __main__ Patch vCSA failed
/var/log/vmware/vtsdb/upgrade/pg_upgrade_internal.log
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
connection to database failed: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/run/vtsdb/.s.PGSQL.5433"?
/var/log/vmware/vtsdb/upgrade/pg_upgrade_server.log
command: "/opt/vmware/vpostgres/current/../12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/storage/vtsdb/vtsdbdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2022-06-04 02:01:25.096 UTC 629abcf5.74fe 0 FATAL: pre-existing shared memory block (key 5433001, ID 32769) is still in use
2022-06-04 02:01:25.096 UTC 629abcf5.74fe 0 HINT: Terminate any old server processes associated with data directory "/storage/vtsdb/vtsdbdata".
stopped waiting
pg_ctl: could not start server
vCenter 7 patch/upgrade failed with "Exception('Inplace pg_upgrade failed').
/var/log/vmware/applmgmt/PatchRunner.log
2022-06-04 02:01:25,190.190Z vtsdb:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'vtsdb:Patch' failed.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/payload/components-script/vtsdb/__init__.py", line 144, in patch
raise Exception('Inplace pg_upgrade failed')
Exception: Inplace pg_upgrade failed
2022-06-04T02:01:25.195Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
reportQueue, identifier, expectedResultType)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
File "/storage/updatemgr/software-updatew0jceyl0/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
raise ex
patch_errors.ComponentError
2022-06-04T02:01:26.197Z WARNING root stopping status aggregation...
2022-06-04T02:01:26.197Z ERROR __main__ Patch vCSA failed
/var/log/vmware/vtsdb/upgrade/pg_upgrade_internal.log
Performing Consistency Checks on Old Live Server
------------------------------------------------
Checking cluster versions ok
connection to database failed: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/var/run/vtsdb/.s.PGSQL.5433"?
/var/log/vmware/vtsdb/upgrade/pg_upgrade_server.log
command: "/opt/vmware/vpostgres/current/../12/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/storage/vtsdb/vtsdbdata" -o "-p 50432 -c autovacuum=off -c autovacuum_freeze_max_age=2000000000 -c listen_addresses='' -c unix_socket_permissions=0700" start >> "pg_upgrade_server.log" 2>&1
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2022-06-04 02:01:25.096 UTC 629abcf5.74fe 0 FATAL: pre-existing shared memory block (key 5433001, ID 32769) is still in use
2022-06-04 02:01:25.096 UTC 629abcf5.74fe 0 HINT: Terminate any old server processes associated with data directory "/storage/vtsdb/vtsdbdata".
stopped waiting
pg_ctl: could not start server
Environment
VMware vCenter Server 7.0.x
Cause
vmon was issuing SIGTERM StopSignal(as default) to shutdown vtsdb during the upgrade, which was leading to a force kill of vtsdb as a result of shutdown process timeout.
Resolution
This issue is fixed in vCenter version 7.0 U3i.
In the fix provided, we are explicitly issuing SIGINT as StopSignal, which leads to fast shutdown, as compared to smart shutdown through SIGTERM which was issued earlier by default. We have also specified StopBefore param to shutdown vstats before vtsdb.
Smart Shutdown vs Fast Shutdown: https://www.postgresql.org/docs/current/server-shutdown.html
In the fix provided, we are explicitly issuing SIGINT as StopSignal, which leads to fast shutdown, as compared to smart shutdown through SIGTERM which was issued earlier by default. We have also specified StopBefore param to shutdown vstats before vtsdb.
Smart Shutdown vs Fast Shutdown: https://www.postgresql.org/docs/current/server-shutdown.html
Feedback
Yes
No
Powered by
