VMware postgres and VMware vTSDB services showing healthy with warning in vCenter VAMI after vCenter 9.0.2.x Update/Upgrade

Products

• VMware vCenter Server

Issue/Introduction

After Update or Upgrade to 9.0.2.x in the vCenter VAMI UI, services VMware postgres and VMware vTSDB services showing healthy with warning

 

Environment

vCenter 9.0.2.x

Cause

 

In the /var/log/vmware/vpostgres/postgresql.log we see a failed to open file or No such file or directory for vmware-vpostgres.service/memory.stat

xxxx-xx-xx xx:xx:xx.xxx UTC xaxxxxxx.xxxxxx 0    xxxxxxx xxxLOG:  Checking bloat of WAL caused by replication slots
xxxx-xx-xx xx:xx:xx.xxx UTC xaxxxxxx.xxxxxx 0    xxxxxxx xxxWARNING:  Failed to open file "/sys/fs/cgroup/memory/system.slice/vmware-vpostgres.service/memory.stat": No such file or directory
xxxx-xx-xx xx:xx:xx.xxx UTC xaxxxxxx.xxxxxx 0    xxxxxxx xxxLOG:  Writing instance status...
xxxx-xx-xx xx:xx:xx.xxx UTC xaxxxxxx.xxxxxx 0    xxxxxxx xxxLOG:  Wrote instance status successfully.
xxxx-xx-xx xx:xx:xx.xxx UTC xaxxxxxx.xxxxxx 0    xxxxxxx xxxLOG:  Updated instance status successfully.

In the /var/log/vmware/vtsdb/postgresql.log  we see a failed to open file or No such file or directory for vtsdb.service/memory.stat

xxxx-xx-xx xx:xx:xx.xxx UTC xxxxxxxx.xxxxxx 0   WARNING:  Failed to open file "/sys/fs/cgroup/memory/system.slice/vtsdb.service/memory.stat": No such file or directory
xxxx-xx-xx xx:xx:xx.xxx UTC xxxxxxxx.xxxxxx 0   LOG:  Writing instance status...
xxxx-xx-xx xx:xx:xx.xxx UTC xxxxxxxx.xxxxxx 0   LOG:  Wrote instance status successfully.
xxxx-xx-xx xx:xx:xx.xxx UTC xxxxxxxx.xxxxxx 0   LOG:  Updated instance status successfully.
xxxx-xx-xx xx:xx:xx.xxx UTC xxxxxxxx.xxxxxx 0   LOG:  Updating instance status...

In the  /var/log/vmware/vmon/vmon.log we see that the service vmware-vpostgres and vtsdb start successfully, but they stay in an INITIALIZED,UNHEALTHY,0  state, but there are no issues with functionality in the vSphere Client.

xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vmware-vpostgres> Service pre-start command completed successfully.
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vmware-vpostgres> Constructed command: /opt/vmware/vpostgres/current/bin/postgres -D /storage/db/vpostgres 
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vmware-vpostgres> Running the API Health command as user vpostgres
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vmware-vpostgres-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-vpostgres -f /dev/shm/vmware-postgres-health-status.xml 
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vmware-vpostgres> Service STARTED successfully.
xxxx-xx-xxTxx:xx:xx.xxxx Wa(xx) host-xxxxx [ReadSvcSubStartupData] No startup information from vmware-postgres-archiver.
xxxx-xx-xxTxx:xx:xx.xxxx Wa(xx) host-xxxxx [ReadSvcSubStartupData] No startup information from vmware-vpostgres.
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vmware-vpostgres,INITIALIZED,UNHEALTHY,0 
========================================================================================================================================================================================================================
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vtsdb> Service pre-start command completed successfully.
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vtsdb> Constructed command: /usr/lib/vmware-vtsdb/scripts/vtsdb_start.sh 
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vtsdb> Running the API Health command as user vtsdbuser
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vtsdb-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vtsdb -f /dev/shm/vmware-vtsdb-health-status.xml 
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <vtsdb> Service STARTED successfully.
xxxx-xx-xxTxx:xx:xx.xxxx Wa(xx) host-xxxxx [ReadSvcSubStartupData] No startup information from vtsdb.
xxxx-xx-xxTxx:xx:xx.xxxx In(xx) host-xxxxx <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vtsdb,INITIALIZED,UNHEALTHY,0 

Resolution

No changes or adjustments needed as this is cosmetic and the resolution is to upgrade to vCenter 9.1.x.

The vtsdb service is removed from the vCenter 9.1.x onwards.

The vpostgres service will not longer reference that memory.stat file in that directory under cgroup and you are NOT able to create the missing memory.stat file in the directory.