vSAN-health service starts and crashes immediately because of the CNS user password mismatch.

Products

• VMware vCenter Server

Issue/Introduction

• The vSAN-health service is down.
  
  
• When attempting to start the service, it starts but crashes immediately.
  
  
• In /var/log/vmware/vmon.log of the affected vCenter server, you see that vmon service is unable to obtain the startup information for vsan-health. 

         Received restart request for vsan-health
         <vsan-health> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
         <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata vsan-health,UNHEALTHY,HEALTHY,0
         <vsan-health> Sysprocess clean stop timed out. Force kill. Pid 4295
         <vsan-health> Service exited. Exit code 1
         Load runstate service vsan-health.json
         <vsan-health-prestart> Constructed command: /usr/lib/vmware-vpx/firstboot/vsanhealth_firstboot.py --patch
         <vsan-health> Service pre-start command completed successfully.
         <vsan-health> Constructed command: /usr/sbin/vsanvcmgmtd -s -c /usr/lib/vmware-vpx/vsan-health/VsanVcMgmtConfig.xml -u /etc/vmware-vsan-health/VsanMgmtCustomizedConfig.xml
         <vsan-health> Running the API Health command as user vsan-health
         <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
         <vsan-health> Re-check service health since it is still initializing.
         Client info Uid=0,Gid=0,Pid=48749,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
         <vsan-health> Service is dumping core. Coredump count 0. CurrReq: 3
         [ReadSvcSubStartupData] No startup information from vsan-health.
         <vsan-health> Service exited. Exit code 1

• In /var/log/vmware/vpostgres/postgresql.log, you notice that postgres is unable to authenticate user "cns" because of the password mismatch.

          VCDB cns FATAL:  password authentication failed for user "cns"
          VCDB cns DETAIL:  Password does not match for user "cns".

Environment

VMware vCenter server 7.x

Cause

This issue is caused due to the password mismatch between VCDB and .cns_pgpass for CNS user. 

Resolution

These steps are only supposed to be performed under the supervision of Broadcom Support as the same involves altering the VCDB data. Before applying the steps below, take a backup or an offline-snapshot (in powered-off state) of the vCenter Server. If the vCenter Server is part of an ELM environment, take an offline snapshot or a backup of all the vCenter servers within the ELM domain. Refer the following kb on the best practices for vCenter server VM snapshots in ELM mode- https://knowledge.broadcom.com/external/article/313886/ 

The password inside /etc/vmware-vsan-health/.cns_pgpass and VCDB for the cns user needs to match. To correct the same, follow the steps below

• First, obtain the current password using the below command.
  
      cat /etc/vmware-vsan-health/.cns_pgpass


• In the output of above command, the last string past the "cns:" section is the password you're looking for. For example, if the output if the command is "localhost:5432:VCDB:cns:tE$t123", then the password is tE$t123.
  
  
• Now, login inside the vPostgres Database using the below command.
  
      /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
  
  
• Run the below command to change the password for the user cns inside VCDB.
  
      alter user cns with password '<password obtained in step-1>';


• For example, if the password is tE$t123, then the command would look like below.
  
      alter user cns with password 'tE$t123';
  
  
• Exit vPostgres Database and start the vsan-health service using vmon-cli or service-control command line utility. 

Additional Information

The password is stored as a MD5 Hash and there is no way to reverse the hash to obtain the plain text password. However, you can still check if the password got modified or no. Follow the below steps to confirm the same.

• Login inside the vPostgres Database using the below command.
  
   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
  
  
• Run the below command which returns the username along with the hash password.
  
  VCDB=# select usename, passwd from pg_shadow where usename = 'cns';
 usename |               passwd
---------+-------------------------------------
 cns     | md<HASH VALUE>
(1 row)


• After you alter the password using the instructions in step-3 under "Resolution" section, a new MD5 Hash gets generated. To check and confirm the same, run the same command again and you should see a new hash value.
  
  VCDB=# select usename, passwd from pg_shadow where usename = 'cns';
 usename |               passwd
---------+-------------------------------------
 cns     | md<NEW HASH VALUE>