After upgrading to vCenter Server 7.0 Update 2 or U3, the vtsdb service fails to start

Products

• VMware vCenter Server

Issue/Introduction

This article is to bring awareness to a known issue with vCenter Server 7.0 where a postgres mismatch in versions occurs during an upgrade. This does not stop an upgrade and will need to be corrected by running the attached script on this KB. Make sure you are on the specific build documented in the workaround section.

Symptoms:
After upgrading to vCenter Server 7.0 Update 2 or Update 3, the vtsdb service does not start.

The log file /var/log/vmware/vtsdb/vtsdb-runtime.log.stderr contains:

Starting service process with pid: 46623.
YYYY-MM-DD HH:MM:SS UTC 60d###b0.b61f 0   FATAL:  database files are incompatible with server
YYYY-MM-DD HH:MM:SS UTC 60d###b0.b61f 0   DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 12.5 (VMware Postgres 12.5.0-17385562 release).

Log file /var/log/vmware/applmgmt/PatchRunner.log contains:

YYYY-MM-DD HH:MM:SS vtsdb:Patch INFO vtsdb Check if pg_upgrade is required
YYYY-MM-DD HH:MM:SS vtsdb:Patch INFO vtsdb.utils Checking if in place pg_upgrade is required
YYYY-MM-DD HH:MM:SS vtsdb:Patch INFO vtsdb.utils Major upgrade not needed, upgrade source version 11 same as this version 11

Environment

VMware vCenter Server 7.0.x

Cause

Despite the vCenter Server upgrade succeeding, there is a segment within postgres for vtsdb that may get missed and generate the error logged above.

Resolution

Workaround:
Note: This is for specific upgrade scenarios to vCenter Server Update 2 build 17958471 or Update 3. If you are not on these builds do not proceed further with the workaround instructions.

1. Ensure that you have a valid backup of the vCenter Server prior to performing the instructions. This script will modify postgres data, and it is recommended to have a good recovery strategy should you need to revert any changes.

2. Using WinSCP, copy the attached python script (pgInplaceUpgrade.py) to the vCenter Server appliance

3. Run the following command to set the session variable for the required file dependences:
 

export PYTHONPATH="/usr/lib/vmware/site-packages:/opt/vmware/vpostgres/current/share/python-modules:/usr/lib/vmware-vtsdb/scripts"

4. On the vCenter Server appliance, run the following command to execute the script to upgrade the postgres instance:

 python /<path>/pgInplaceUpgrade.py

Note: A reboot of the vCenter Server appliance is recommended after running the above instructions

Additional Information

Impact/Risks:
No risk is present running the workaround, however it is recommended a backup is taken prior to running the workaround instructions.