VSTATS and VTSDB Service Fail to Start Following Upgrade of vCenter to 8.x or 9.x

Products

• VMware vCenter Server

Issue/Introduction

Following an upgrade of vCenter to 8.0.3 may discover that following services are in stopped state:
NOTE:  This issue may carry over to 9.x if the problem isn't fixed

• vstats(VMware vStats Service)
• vtsdb (VMware vTsdb Service)

vCenter VAMI backups will also fail to run due to the stopped service state.

• At the location /var/log/vmware/vstats in the vstats-stdout.log logfile you see lines similar to:

<timestamp>  PANIC   vcrecoverable/vcrecoverable.go:126      Error while getting providers.  {"vStats": "MVP", "PANIC": "Error while getting providers."}
vstats/util/vcrecoverable.handlePanic
        bora/vpx/vstats/vstats-svc/src/vstats/util/vcrecoverable/vcrecoverable.go:126
runtime.gopanic

• At the location /var/log/vmware/vtsdb in log file postgresql.log logfile you see lines similar to:

<timestamp> LOG: invalid magic number 0000 in log segment xxxxxxxxx, offset yyyyy
<timestamp> LOG: invalid primary checkpoint record
<timestamp> PANIC: could not locate a valid checkpoint record
<timestamp> LOG : startup process (PID <PID>) was terminated by signal 6 : Aborted
<timestamp> LOG : database system is shutting down

Environment

VMware vCenter Server 8.0.3
VMware vCenter Server 9.x

Cause

VMware Engineering is currently investigating the issue.

Resolution

Workaround1:

Run the following commands in a ssh session to the vCenter as root

1. sudo -u vtsdbuser pg_resetwal -f /storage/vtsdb/vtsdbdata
   Write-ahead log reset
2. service-control --restart vtsdb
3. service-control --restart vstats
4. service-control --status

Workaround2:

1. Create snapshot of the vCenter.
2. Run this script against the vstats database

   /opt/vmware/vpostgres/current/bin/psql -Uvstatsuser -dvstats_db -f /usr/lib/vmware-vstats/scripts/sttdb_patch03.sql -p5433 -w -X -h/var/run/vtsdb -v ON_ERROR_STOP=ON

3. The output should look like this:

   CREATE FUNCTION
   vstats_db database is present
   You are now connected to database "vstats_db" as user "vstatsuser".
   SET
   DO
   ALTER TABLE
   DO
   ALTER TABLE
   ALTER TABLE

4. Once that's done, restart the vstats service and confirm it stays up and running

service-control --start vstats

Note: If the above mentioned workaround doesn't help then proceed with the below steps. 

Standard workarounds like pg_resetwal may not be sufficient if the replorigin_checkpoint file specifically is corrupted. Moving the file allows the service to bypass the magic number validation panic.

Workaround3:

1. Create a temporary backup directory for the corrupted file:
   
   mkdir -p /var/vtsdb_backup
   
   
2. Move the corrupted replorigin_checkpoint file to the temporary backup directory:
   
   mv /storage/vtsdb/vtsdbdata/pg_logical/replorigin_checkpoint /var/vtsdb_backup/
   
   
3. Restart the affected services:
   
   service-control --restart vtsdb
service-control --restart vstats
   
   
4. Ensure both services are running using command:
   
   service-control --status --all
   
   
5. Return to the VAMI or upgrade interface and click Resume to complete the patching process.