vCenter is down after wrong configuration for vcls and reboot.

Products

• VMware vCenter Server

Issue/Introduction

VCLS VMs were deleted or misconfigured and then vCenter was rebooted.

Symptoms:

• vCenter is down and will not start upon attempting.
• vCenter services status:

  • root@vcenter [ /tmp ]# service-control --status
    
    Running:
    
    applmgmt lookupsvc lwsmd observability-vapi pschealth vlcm vmafdd vmcad vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
    
    Stopped:
    
    observability vmcam vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

• STS and certificates are valid.
  • Check STS cert on VCSA/Windows: https://knowledge.broadcom.com/external/article?legacyId=79248
  •  Check & Replace other certs on vCenter: https://knowledge.broadcom.com/external/article?legacyId=68171
• sometimes vpxd.log will not update an error or logs are missing.

Environment

VMware vCenter Server 7.0.x

Cause

• VCLS VMs were deleted and or previously misconfigured and then vCenter was rebooted
• As a result for previous action, vpxd.cfg file was left with wrong data preventing vpxd service from starting.  you may visit  VCLS KB https://knowledge.broadcom.com/external/article?legacyId=80472 for more information on VCLS configuration. 

Resolution

Do not skip any steps please do follow all the below-provided steps. 

• Connect to VCSA using SSH session with root
• Take a backup from file /etc/vmware-vpx/vpxd.cfg
• Confirm vpxd is not running / Stop vpxd using : service-control --stop vpxd
• Remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg
  
  • Workaround 1:
    • Run the below command to remove the VCLS entries from the /etc/vmware-vpx/vpxd.cfg
      • sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg
  • Workaround 2:
    • Manually edit file vpxd.cfg 
      • vi vpxd.cfg
        • Remove the part that contains vcls section, looks like below:
          • <vcls>
            <clusters>
            <domain-c22>
            <enabled>False</enabled>
            </domain-c22>
            <domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
            <enabled>False</enabled>
            </domain-c22:f696e05a-6cf8-4d02-af1c-10e81f90ec81>
            <domain-c86>
            <enabled>false</enabled>
            </domain-c86>
            <f696e05a-6cf8-4d02-af1c-10e81f90ec81>
            <enabled>False</enabled>
            </f696e05a-6cf8-4d02-af1c-10e81f90ec81>
            </clusters>
            </vcls>
• Then save the file and exit
• Start vpxd using : service-control --stop vpxd
• vCenter should be accessible now