vCenter vstats service in stopped state failing to start, identified from failing vami backups

Products

• VMware vCenter Server

Issue/Introduction

vCenter vstats service in stopped state and fails to start.

In the vstats-stderr.log file located at /var/log/vmware/vstats you se the following error:

0000-00-00T00:00:00Z    ERROR   recoverable/recoverable.go:38   UNEXPECTED PANIC. Will call action()    {"vStats": "MVP", "error": "DbpqError: Could not exec/query: pq.ErrorCode=XX002 pq.ErrMsg=pq: index \"counter_pkey\" contains unexpected zero page at block 1"}
vstats/util/recoverable.GenericPanicHandler
        src/vstats/util/recoverable/recoverable.go:38
vstats/util/recoverable.Func.func1.1
        src/vstats/util/recoverable/recoverable.go:66
runtime.gopanic
        GOROOT/src/runtime/panic.go:1038
vstats/vpm.initializeCounterChandleView
        src/vstats/vpm/implementation.go:311
vstats/vpm.Start.func2
        src/vstats/vpm/implementation.go:227
vstats/util/recoverable.Func.func1
        src/vstats/util/recoverable/recoverable.go:70
vstats/vpm.Start
        src/vstats/vpm/implementation.go:227
main.setupProviderManager
        src/vstats/main.go:630
main.main
        src/vstats/main.go:331
runtime.main
        GOROOT/src/runtime/proc.go:255

Environment

vCenter 7x

Cause

Issue is caused by "contains unexpected zero page at block 1" database index corruption

Resolution

Take snapshot of vCenter, if linked mode offline snapshots of all linked nodes.

1. Access the vstats database:
   /opt/vmware/vpostgres/current/bin/psql  -U postgres -p 5433 -w -X -d vstats_db -h /var/run/vtsdb
2. Perform the following reindexing:
   REINDEX schema vstats_db_schema;
3. exit the database:
   \q
4. start vstats service:
   service-control --start vstats