vmware-vsan-health service fails to start | FATAL: password authentication failed for user "cns"
Article ID: 371924
Updated On:
Products
Issue/Introduction
vmware-vsan-health service fails to start :
XXXX-XX-XXTXX:XX:XX.145Z INFO service-control ********** Start ['--start', 'vmware-vsan-health'] **********
XXXX-XX-XXTXX:XX:XX.148Z INFO service-control Perform start operation. vmon_profile=None, svc_names=['vmware-vsan-health'], include_coreossvcs=False, include_leafossvcs=False
XXXX-XX-XXTXX:XX:XX.149Z INFO service-control Performing start operation on service vsan-health...
XXXX-XX-XXTXX:XX:XX.118Z ERROR service-control Error executing start on service vsan-health. Details {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vsan-health"
],
"localized": "An error occurred while starting service 'vsan-health'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
2024-07-08T14:35:15.118Z ERROR service-control Service-control failed. Error: {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vsan-health"
],
"localized": "An error occurred while starting service 'vsan-health'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
vmon.log :
XXXX-XX-XXTXX:XX:XX.222Z In(05) host-1929 Received start request for vsan-health
XXXX-XX-XXTXX:XX:XX.222Z In(05) host-1929 <vsan-health-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-vsan/scripts/vsanvcmgmtd-prestart.pyc
XXXX-XX-XXTXX:XX:XX.683Z In(05) host-1929 <vsan-health> Service pre-start command completed successfully.
XXXX-XX-XXTXX:XX:XX.683Z In(05) host-1929 <vsan-health> Constructed command: /usr/sbin/vsanvcmgmtd -s -c /usr/lib/vmware-vsan/VsanVcMgmtConfig.xml -u /etc/vmware-vsan-health/VsanMgmtCustomizedConfig.xml
XXXX-XX-XXTXX:XX:XX.685Z In(05) host-1929 <vsan-health> Running the API Health command as user vsan-health
XXXX-XX-XXTXX:XX:XX.685Z In(05) host-1929 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
XXXX-XX-XXTXX:XX:XX.935Z In(05) host-1929 <vsan-health> Re-check service health since it is still initializing.
XXXX-XX-XXTXX:XX:XX.936Z In(05) host-1929 <vsan-health> Running the API Health command as user vsan-health
XXXX-XX-XXTXX:XX:XX.936Z In(05) host-1929 <vsan-health-healthcmd> Constructed command: /usr/bin/python /usr/lib/vmware-vpx/vsan-health/vsanhealth-vmon-apihealth.py
XXXX-XX-XXTXX:XX:XX.253Z In(05) host-1929 <vsan-health> Re-check service health since it is still initializing.
XXXX-XX-XXTXX:XX:XX.697Z In(05) host-1929 Client info Uid=0,Gid=0,Pid=359821,Comm=(vmon-coredumper),PPid=2,Comm=(kthreadd),PPid=0
XXXX-XX-XXTXX:XX:XX.697Z In(05) host-1929 <vsan-health> Service is dumping core. Coredump count 0. CurrReq: 1
Under /var/core we will see core.vsanvcmgmtd dump files.
vsanvcmgmtd.log :
XXXX-XX-XXTXX:XX:XX:14.578+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=Default] Initializing plugin CnsService
XXXX-XX-XXTXX:XX:XX:14.578+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=CnsService] Initializing CNS
XXXX-XX-XXTXX:XX:XX:14.578+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=CnsConfigOptions] CnsConfigOptions get configured VolumePlaceAlgo as lottery; MaxConcurrentQueryRequests as 2147483647; MaxQueryAsyncTasksInQueue as 20; VolumeRelocateEnabled as true; NewSyncEnabled as true; NewDbEnabled as true; PropertyCollectorDispatcherEnabled as true; PvscsiCtrlr256DiskSupportEnabled as false; NewSerializationEnabled as true.
XXXX-XX-XXTXX:XX:XX:14.578+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=CnsConfigOptions] SyncStartInvokeEnabled as true0d.
XXXX-XX-XXTXX:XX:XX:14.578+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=VpxdCnx] Initializing VPXD connection for CNS caller
XXXX-XX-XXTXX:XX:XX:14.579+08:00 info vsanvcmgmtd[3995993] [vSAN@6876 sub=vmomi.soapStub[4] opId=vsan-wfu-bd08] SOAP request returned HTTP failure; <<cs p:00007fc66405e070, TCP:localhost:8085>, /sdk>, method: createPropertyCollector; code: 500(Internal Server Error); fault: (vim.fault.NotAuthenticated) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> object = 'vmodl.query.PropertyCollector:6e8afecd-c53f-48e3-802e-98504b435b51:propertyCollector',
--> privilegeId = "",
--> missingPrivileges = <unset>
--> msg = "Received SOAP response fault from [<<cs p:00007fc66405e070, TCP:localhost:8085>, /sdk>]: createPropertyCollector
--> The session is not authenticated."
--> }
XXXX-XX-XXTXX:XX:XX:14.586+08:00 info vsanvcmgmtd[3996268] [vSAN@6876 sub=PyCppVmomi opId=vsan-PC-61cbd51fa4c20] Initialized python thread state 0x00007fc6708ba590.
XXXX-XX-XXTXX:XX:XX:14.586+08:00 info vsanvcmgmtd[3995993] [vSAN@6876 sub=StsUtil opId=vsan-wfu-bd08] Initialized Local Security Token Service. tokenLifetime: 480, SolutionUser: 'vpxd-extension'.
XXXX-XX-XXTXX:XX:XX:14.597+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=CnsCnxFactory] Initializing VPXD connection for CNS caller
XXXX-XX-XXTXX:XX:XX:14.597+08:00 info vsanvcmgmtd[3995991] [vSAN@6876 sub=WorkQueue.CnsTask] Created: WorkQueue.CnsTask, type = fair, priority = 16, itemWeight = 2
XXXX-XX-XXTXX:XX:XX:14.621+08:00 error vsanvcmgmtd[3995991] [vSAN@6876 sub=CnsVdb] InitConnection: Error ODBC error: (08001) - [unixODBC]connection to server on socket "/var/run/vpostgres/.s.PGSQL.5432" failed: FATAL: password authentication failed for user "cns"
-->
postgresql.log :
XXXX-XX 14:21:17.653 UTC 668e98dd.57d8a 0 [unknown] [unknown] [local] 359818 1LOG: connection received: host=[local]
XXXX-XX 14:21:17.662 UTC 668e98dd.57d8a 0 VCDB cns [local] 359818 2FATAL: password authentication failed for user "cns"
XXXX-XX 14:21:17.662 UTC 668e98dd.57d8a 0 VCDB cns [local] 359818 3DETAIL: User "cns" has no password assigned.
Connection matched pg_hba.conf line 70: "local all all
Cause
This issue happens when the CNS user is missing from pg_hba.conf and password for CNS user is missing.
Resolution
To resolve this issue, please follow the steps below :
- Take snapshot of the vCenter. If in linked mode, take powered off snapshots
- SSH to vCenter appliance.
- cd /storage/db/vpostgres
- cp pg_hba.conf pg_hba.conf.orig
- vi pg_hba.conf and the following lines (Above : # hba entries for role all)
# hba entries for role cns
local VCDB cns peer map=vcdb
- Save the file.
- Make sure ownership of the file is : chown vpostgres:vpgmongrp pg_hba.conf
- Run : psql -Upostgres -dVCDB -c "SELECT pg_reload_conf();"
- Connect to DB and set a password for cns : (We can set any random password)
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres
- ALTER USER cns WITH PASSWORD 'VMware@123';
- Exit the DB.
- Restart postgres service : service-contol –restart vpostgres
- Start the vsan-health service.
Feedback
