vCenter is down after wrong configuration for vcls and reboot.

Products

• VMware vCenter Server

Issue/Introduction

vCLS VMs were deleted or misconfigured and then vCenter was rebooted.

Symptoms:

• vCenter Server is down and the services will not start upon attempting to restart the server.
• vCenter services status:

root@vcenter [/]# service-control --status

Running:

applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb

Stopped:

observability vmcam vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

 

• STS and certificates are valid.
  • Check STS cert on VCSA/Windows: Checking Expiration of STS Certificate on vCenter Servers (79248)
  • Check & Replace other certs on vCenter: CertificateStatusAlarm - There are certificate that expired or about to expire (68171)

• Sometimes /var/log/vmware/vpxd/vpxd.log will stop recording events or logs are missing.
• The last line in the /var/log/vmware/vpxd/vpxd.log looks like below where we see ^@ symbols repeatedly:

YYYY-MM-DDTHH:MM:SS info vpxd[07285] [Originator@ 6876 sub-Mocluster opID=CdrsLoadBalancer-4431b782] HDCS mgr updating non-functional DRS workflow cache for workflow: LB_COUNT eventnKey 11377464 count 1

YYYY-MM-DDTHH:MM:SS info vpxd [07285] [Originator@ 6876 sub-cdrs Plmt opID=CdrsLoadBalancer-4431b782] Load-balance iteration 28317 scanned 0 vmotion 0 min 40 max 39867 total 11277533 hosts 0 skipped 0 whatif-run: false YYYY-MM-DDTHH:MM:SS info vpxd[07827] [Originator@6876 sub=vpxLro opID=1yr4ae2c-kxgi-h5:70146535-11] [VpxLRO] FINISH 1ro-39310443

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Environment

VMware vCenter Server 7.0.x

VMware vCenter Server 8.0.x

Cause

• VCLS VMs were deleted and or previously misconfigured and then vCenter was rebooted
• As a result for previous action, vpxd.cfg file was left with wrong data preventing vpxd service from starting.  you may visit  VCLS KB  vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions (80472) for more information on VCLS configuration. 

Resolution

Do not skip any steps please do follow all the below-provided steps. 

• Connect to VCSA using SSH session with root
• Take a backup from file /etc/vmware-vpx/vpxd.cfg
• To take backup cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.bak
• Confirm vmware-vpxd service is not running / Stop vmware-vpxd using the command: service-control --stop vpxd
• Remove the vCLS entries from the /etc/vmware-vpx/vpxd.cfg
  
  • Workaround 1:
    • Run the below command to remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg
      • sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg
  • Workaround 2:
    • Manually edit file vpxd.cfg by using vi editor 
      • vi /etc/vmware-vpx/vpxd.cfg
        • Remove the part that contains vcls section, looks like below:

<vcls>
<clusters>
<domain-c22>
<enabled>False</enabled>
</domain-c22>
<domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<domain-c86>
<enabled>false</enabled>
</domain-c86>
<f696e05a-6cf8-4d02-af1c-10e81f90ec81>
<enabled>False</enabled>
</f696e05a-6cf8-4d02-af1c-10e81f90ec81>
</clusters>
</vcls>

• Then press Esc button, type :wq! to save the file and exit
• If only vmware-vpxd is stopped, start the service using the command: service-control --start vpxd
• If many services are stopped, then restart all the services by running the command: service-control --stop --all && service-control --start --all
• vCenter should be accessible now.