vCenter services fail to start due to incorrect vCLS configuration in vpxd.cfg

Products

• VMware vCenter Server

Issue/Introduction

• vSphere client is inaccessible after restoring the vCenter Appliance from backup and the services will not start upon attempting to restart the server.
• If vSphere client can be accessed, it shows blank inventory upon login.
• Many vCenter services including vmware-vpxd can be found in a stopped state:

root@<vcenter> [/]# service-control --status
Running:
applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
Stopped:
observability vmcam vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

• vpxd.log (/var/log/vmware/vpxd/vpxd.log) stops recording events or logs are missing.
• Validating the vpxd configuration file (/etc/vmware-vpx/vpxd.cfg) by using the command: /usr/sbin/vpxd fails, resulting in a rapid accumulation of messages in the vpxd.log file.
• The last line in the vpxd.log (/var/log/vmware/vpxd/vpxd.log) ends with repeated ^@ symbols:

--> . Retry attempt: 4449
YYYY-MM-DDTHH:MM:SSZ info vpxd[181242] [Originator@6876 sub=vpxdVdb] [VpxdVdb: : SetDBType] Logging in to DSN: VMware VirtualCenter with username vpxd
YYYY-MM-DDTHH:MM:SSZ info vpxd[181242] [Originator@6876 sub=vpxdVdb] [VpxdVdb: : SetDBType] Failed to connect to database: ODBC error: (08001) - [unixODBC]connection to server on socket "/var/run/vpostgres/.s. PGSQL. 5432" failed: FATAL: Peer authentication failed for user "vpxd"
YYYY-MM-DDTHH:MM:SS info vpxd[07285] [Originator@ 6876 sub-Mocluster opID=CdrsLoadBalancer-########] HDCS mgr updating non-functional DRS workflow cache for workflow: LB_COUNT eventnKey 11377464 count 1
YYYY-MM-DDTHH:MM:SS info vpxd [07285] [Originator@ 6876 sub-cdrs Plmt opID=CdrsLoadBalancer-########] Load-balance iteration 28317 scanned 0 vmotion 0 min 40 max 39867 total 11277533 hosts 0 skipped 0 whatif-run: false YYYY-MM-DDTHH:MM:SS info vpxd[07827] [Originator@6876 sub=vpxLro opID=1yr4ae2c-kxgi-h5:70146535-11] [VpxLRO] FINISH 1ro-39310443

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Environment

• vCenter 7.x
• vCenter 8.x

Cause

vCLS VM deletion or misconfiguration, followed by a vCenter reboot, causes the vpxd.cfg file to retain misconfigured or incorrect data, which subsequently prevents the vpxd service from starting.

Resolution

  A. Validate if vCenter certificates are valid.

1. Check STS cert on vCenter : Checking Expiration of STS Certificate on vCenter Servers
2. Check & Replace other certs on vCenter: CertificateStatusAlarm - There are certificate that expired or about to expire

  B. If the certificates are valid, follow the below steps. 

1. Log in to the vCenter Server Appliance via SSH.
   
   
2. Take a backup for the configuration file vpxd.cfg:
   
   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
   
   
3. Stop the vmware-vpxd service:
   
   service-control --stop vpxd
   
   
4. Remove the vCLS entries from the /etc/vmware-vpx/vpxd.cfg following one of the following workarounds:
   
   
   1. Workaround 1:
      1. Run the below command to remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg:
         
         sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg
         
         
   2. Workaround 2:
      1. Manually edit file vpxd.cfg by using vi editor 
         
         vi /etc/vmware-vpx/vpxd.cfg
         
         
      2. Remove the part that contains vcls section similar to below:
         
         <vcls>
<clusters>
<domain-c##>
<enabled>False</enabled>
</domain-c##>
<domain-c##:f696e05a-####-####-####-############>
<enabled>False</enabled>
</domain-c##:f696e05a-####-####-####-############>
</clusters>
</vcls>
         
         
      3. Save the file by pressing the Esc button, type :wq! to save the file and exit.
         
         
5. Start the vpxd service:
   
   service-control --start vpxd
   
   
6. To stop and start all the vCenter services:
   
   service-control --stop --all && service-control --start --all
   
   

If the issue persists, contact Broadcom Support for further assistance.

Additional Information

For more information about vCLS configuration, refer: vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions.