vstat service is failing to start in vcenter due to it's corrupt DB
Article ID: 387613
Updated On:
Products
Issue/Introduction
vstat service is failing to start in vcenter due to it's corrupt DB
Environment
VMware vCenter server
Cause
vStats DB files are corrupt due to underlying storage issue.
/var/log/vmware/vstats/vstats-stderr.log reports similar to the below error when attempted to start.
YYYY-MM-DDThh:mm:ssZ ERROR dbs/sttdb_cnt.go:460 delete(CM-rows) {"vStats": "MVP", "deleted": 0, "error": "DbpqError: Could not exec/query: pq.ErrorCode=58P01 pq.ErrMsg=pq: could not open file \"base/16384/16652\": No such file or directory", "sqlcmd": "DELETE FROM counter_metadata cm WHERE 0=(SELECT COUNT(*) FROM provider_x_cntrmeta pc WHERE counter_metadata=cm.id)"} YYYY-MM-DDThh:mm:ssZ PANIC dbs/sttdb.go:52 Call to deletePxCrowsAndGhosts() failed {"vStats": "MVP", "error": "DbpqError: Could not exec/query: pq.ErrorCode=58P01 pq.ErrMsg=pq: could not open file \"base/16384/16652\": No such file or directory"} YYYY-MM-DDThh:mm:ss:21Z ERROR dbs/sttdb_cnt.go:460 delete(CM-rows) {"vStats": "MVP", "deleted": 0, "error": "DbpqError: Could not exec/query: pq.ErrorCode=58P01 pq.ErrMsg=pq: could not open file \"base/16384/16652\": No such file or directory", "sqlcmd": "DELETE FROM counter_metadata cm WHERE 0=(SELECT COUNT(*) FROM provider_x_cntrmeta pc WHERE counter_metadata=cm.id)"} YYYY-MM-DDThh:mm:ssZ PANIC dbs/sttdb.go:52 Call to deletePxCrowsAndGhosts() failed {"vStats": "MVP", "error": "DbpqError: Could not exec/query: pq.ErrorCode=58P01 pq.ErrMsg=pq: could not open file \"base/16384/16652\": No such file or directory"}
Resolution
Reset the vstats DB and restart the vstats service.
Make sure to take an offline snapshot of the vCenter before performing the below steps. If the vCenter is part of an ELM configuration, you must power off all the vCenters in ELM take snapshot for all vcenters. Should you need to restore, the snapshots must be reverted on all the vcenters at the same time.
- service-control --stop vstats vtsdb
- sudo mv /storage/vtsdb/vtsdbdata/postmaster.pid /storage/vtsdb/vtsdbdata/postmaster.pid_backup
- ls -l /storage/vtsdb/vtsdbdata/ ==> (confirm that postmaster.pid is renamed as shown below)
root@< VC-NAME >[ ~ ]# Is -I /storage/vtsdb/vtsdbdata/ total 128 drwx------ 6 vtsdbuser users 4096 Oct 21 2022 base -rw------- 1 vtsdbuser users 47 May 30 13:47 current_logfiles drwx------ 2 vtsdbuser users 4096 May 30 13:47 global -rw------- 1 vtsdbuser users 684 Oct 21 2022 health_status_worker.conf drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_commit_ts drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_dynshmem rw-------- 1 vtsdbuser users 255 Oct 21 2022 pg_hba.conf -rw------- 1 vtsdbuser users 245 Oct 21 2022 pg_ident.conf drwx------ 4 vtsdbuser users 4096 May 30 13:52 pg_logical drwx------ 4 vtsdbuser users 4096 Oct 21 2022 pg_multixact drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_notify drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_replslot drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_serial drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_snapshots drwx------ 2 vtsdbuser users 4096 May 30 13:47 pg_stat drwx------ 2 vtsdbuser users 4096 May 30 13:47 pg_stat_tmp drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_subt rans drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_tblspc drwx------ 2 vtsdbuser users 4096 Oct 21 2022 pg_twophase -rw------- 1 vtsdbuser users 3 0ct 21 2022 PG_VERSION l rwxrwxrwx 2 vtsdbuser users 30 0ct 21 2022 pg_wal -> /storage/vtsdblog/vtsdb/pg_wal drwx------ 1 vtsdbuser users 4096 Oct 21 2022 pg_xact -rw------- 1 vtsdbuser users 88 Oct 21 2022 postgresql.auto.conf -rw------- 1 vtsdbuser users26509 Oct 21 2022 postgresql.conf -rw------- 1 vtsdbuser users 299 Oct 21 2022 postgresql.conf.auto -rw------- 1 vtsdbuser users 70 May 30 13:47 postmaster.opts -rw------- 1 vtsdbuser users 101 May 30 13:4/ postmaster.pid_backup - sudo -u vtsdbuser /opt/vmware/vpostgres/current/bin/pg_resetwal -f /storage/vtsdb/vtsdbdata
- service-control --start vstats vtsdb
Feedback
