vCenter is down after incorrect configuration for vCLS and reboot

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server is down after restoring from backup and the services will not start upon attempting to restart the server.
• vCenter services status:

root@vcenter [/]# service-control --status

Running:

applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb

Stopped:

observability vmcam vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

•  
• Sometimes vCenter - /var/log/vmware/vpxd/vpxd.log will stop recording events or logs are missing.
• The execution of /usr/sbin/vpxd is currently hung/stuck, resulting in a rapid accumulation of messages in the /var/log/vmware/vpxd/vpxd.log file.
• The last line in the vCenter - /var/log/vmware/vpxd/vpxd.log looks like below where we see ^@ symbols repeatedly:

-- > . Retry attempt: 4449

YYYY-MM-DDTHH:MM:SSZ info vpxd[181242] [Originator@6876 sub=vpxdVdb] [VpxdVdb: : SetDBType] Logging in to DSN: VMware VirtualCenter with username vpxd
YYYY-MM-DDTHH:MM:SSZ info vpxd[181242] [Originator@6876 sub=vpxdVdb] [VpxdVdb: : SetDBType] Failed to connect to database: ODBC error: (08001) - [unixODBC]connection to server on socket "/var/run/vpostgres/.s. PGSQL. 5432" failed: FATAL: Peer authentication failed for user "vpxd"

YYYY-MM-DDTHH:MM:SS info vpxd[07285] [Originator@ 6876 sub-Mocluster opID=CdrsLoadBalancer-4431b782] HDCS mgr updating non-functional DRS workflow cache for workflow: LB_COUNT eventnKey 11377464 count 1
YYYY-MM-DDTHH:MM:SS info vpxd [07285] [Originator@ 6876 sub-cdrs Plmt opID=CdrsLoadBalancer-4431b782] Load-balance iteration 28317 scanned 0 vmotion 0 min 40 max 39867 total 11277533 hosts 0 skipped 0 whatif-run: false YYYY-MM-DDTHH:MM:SS info vpxd[07827] [Originator@6876 sub=vpxLro opID=1yr4ae2c-kxgi-h5:70146535-11] [VpxLRO] FINISH 1ro-39310443

 

         ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Environment

• vCenter 7.x
• vCenter 8.x

Cause

• VCLS VMs were deleted or previously misconfigured and then vCenter was rebooted
• As a result of previous actions, vpxd.cfg file was left with incorrect data preventing vpxd service from starting.  Refer to vCLS KB  vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions for more information on VCLS configuration.

Resolution

• Check if STS and vCenter certificates are valid.
• • Check STS cert on vCenter : Checking Expiration of STS Certificate on vCenter Servers
  • Check & Replace other certs on vCenter: CertificateStatusAlarm - There are certificate that expired or about to expire

     

      If the certificates are valid, follow all the below steps. 

1. Connect to VCSA using SSH session with root
   
   
2. Take a backup for the configuration file vpxd.cfg:
   
   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
   
   
3. Stop the vmware-vpxd service:
   
   service-control --stop vpxd
   
   
4. Remove the vCLS entries from the /etc/vmware-vpx/vpxd.cfg
   
   
   • Workaround 1:
     
     Run the below command to remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg:
     
     sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg
     
     
   • Workaround 2:
     • Manually edit file vpxd.cfg by using vi editor 
       
       vi /etc/vmware-vpx/vpxd.cfg
       
       
     • Remove the part that contains vcls section similar to below:
       
       <vcls>
<clusters>
<domain-c22>
<enabled>False</enabled>
</domain-c22>
<domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<domain-c86>
<enabled>false</enabled>
</domain-c86>
<f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</f696e05a-6cf8-4d02-af1c-10e81f90ec81>
</clusters>
</vcls>
       
       
     • Save the file by pressing the Esc button, typing :wq! to save the file and exit.
       
       
5. Start the vpxd service:
   
   service-control --start vpxd
   
   
6. To stop and start all the vCenter services:
   
   service-control --stop --all && service-control --start --all