vCenter is down after wrong configuration for vCLS and reboot

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Server is down and the services will not start upon attempting to restart the server.
• vCenter services status:

root@vcenter [/]# service-control --status

Running:

applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb

Stopped:

observability vmcam vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

• STS and certificates are valid.
  • Check STS cert on VCSA/Windows: Checking Expiration of STS Certificate on vCenter Servers (79248)
  • Check & Replace other certs on vCenter: CertificateStatusAlarm - There are certificate that expired or about to expire (68171)
• Sometimes vCenter - /var/log/vmware/vpxd/vpxd.log will stop recording events or logs are missing.
• The last line in the vCenter - /var/log/vmware/vpxd/vpxd.log looks like below where we see ^@ symbols repeatedly:

YYYY-MM-DDTHH:MM:SS info vpxd[07285] [Originator@ 6876 sub-Mocluster opID=CdrsLoadBalancer-4431b782] HDCS mgr updating non-functional DRS workflow cache for workflow: LB_COUNT eventnKey 11377464 count 1
YYYY-MM-DDTHH:MM:SS info vpxd [07285] [Originator@ 6876 sub-cdrs Plmt opID=CdrsLoadBalancer-4431b782] Load-balance iteration 28317 scanned 0 vmotion 0 min 40 max 39867 total 11277533 hosts 0 skipped 0 whatif-run: false YYYY-MM-DDTHH:MM:SS info vpxd[07827] [Originator@6876 sub=vpxLro opID=1yr4ae2c-kxgi-h5:70146535-11] [VpxLRO] FINISH 1ro-39310443

         ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

 

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

• VCLS VMs were deleted or previously misconfigured and then vCenter was rebooted
• As a result for previous action, vpxd.cfg file was left with wrong data preventing vpxd service from starting.  Refer to vCLS KB  vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions for more information on VCLS configuration.

Resolution

Do not skip any of the below steps, follow all the below steps. 

1. Connect to VCSA using SSH session with root
   
   
2. Take a backup for the configuration file vpxd.cfg:
   
   cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
   
   
3. Stop the vmware-vpxd service:
   
   service-control --stop vpxd
   
   
4. Remove the vCLS entries from the /etc/vmware-vpx/vpxd.cfg
   
   
   • Workaround 1:
     
     Run the below command to remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg:
     
     sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg
     
     
   • Workaround 2:
     • Manually edit file vpxd.cfg by using vi editor 
       
       vi /etc/vmware-vpx/vpxd.cfg
       
       
     • Remove the part that contains vcls section similar to below:
       
       <vcls>
<clusters>
<domain-c22>
<enabled>False</enabled>
</domain-c22>
<domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<domain-c86>
<enabled>false</enabled>
</domain-c86>
<f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</f696e05a-6cf8-4d02-af1c-10e81f90ec81>
</clusters>
</vcls>
       
       
     • Save the file by pressing the Esc button, typing :wq! to save the file and exit.
       
       
5. Start the vpxd service:
   
   service-control --start vpxd
   
   
6. To stop and start all the vCenter services:
   
   service-control --stop --all && service-control --start --all