VSTATS and VTSDB Service Fail to Start Following Upgrade of vCenter to 8.x

Products

• VMware vCenter Server

Issue/Introduction

Following an upgrade of vCenter to 8.0.3 may discover that following services are in stopped state:

• vstats(VMware vStats Service)
• vtsdb (VMware vTsdb Service)

vCenter VAMI backups will also fail to run due to the stopped service state.

• At the location /var/log/vmware/vstats in the vstats-stdout.log logfile you see lines similar to:

<timestamp>  PANIC   vcrecoverable/vcrecoverable.go:126      Error while getting providers.  {"vStats": "MVP", "PANIC": "Error while getting providers."}
vstats/util/vcrecoverable.handlePanic
        bora/vpx/vstats/vstats-svc/src/vstats/util/vcrecoverable/vcrecoverable.go:126
runtime.gopanic

• At the location /var/log/vmware/vtsdb in log file postgresql.log logfile you see lines similar to:

<timestamp> LOG: invalid magic number 0000 in log segment xxxxxxxxx, offset yyyyy
<timestamp> LOG: invalid primary checkpoint record
<timestamp> PANIC: could not locate a valid checkpoint record
<timestamp> LOG : startup process (PID <PID>) was terminated by signal 6 : Aborted
<timestamp> LOG : database system is shutting down

Environment

VMware vCenter Server 8.0.3

Cause

VMware Engineering is currently still investigating the issue.

Resolution

Workaround1:

Run the following commands in a ssh session to the vCenter as root

1. sudo -u vtsdbuser pg_resetwal -f /storage/vtsdb/vtsdbdata
   Write-ahead log reset
2. service-control --restart vtsdb
3. service-control --restart vstats
4. service-control --status

Workaround2:

1. Create snapshot of the vCenter.
2. Run this script against the vstats database

   /opt/vmware/vpostgres/current/bin/psql -Uvstatsuser -dvstats_db -f /usr/lib/vmware-vstats/scripts/sttdb_patch03.sql -p5433 -w -X -h/var/run/vtsdb -v ON_ERROR_STOP=ON

3. The output should look like this:

   CREATE FUNCTION
   vstats_db database is present
   You are now connected to database "vstats_db" as user "vstatsuser".
   SET
   DO
   ALTER TABLE
   DO
   ALTER TABLE
   ALTER TABLE

4. Once that's done, restart the vstats service and confirm it stays up and running

service-control --start vstats