The wcp service fails to start on the vCenter server

Products

• VMware vCenter Server

Issue/Introduction

• The wcp service fails to start on the VCSA, and user attempts to start the affected service via VCSA SSH result in the following error message -
  • ERROR service-control: Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start wcp services. Error: Service crashed while starting.
• Additionally, CURL requests to the following URLs return a 404 - Not Found error
  • https://<VC-IP-or-FQDN>/openidconnect/jwks/vsphere.local
  • https://<VC-IP-or-FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration
• On the vcenter server logs, we see entries similar to below:

• The log entries in the following files indicate issues with the VCSA wcp service.
  • /var/log/vmware/cloudvm/service-control.log -

yyyy-mm-ddThh:mm:ss.Z INFO service-control Successfully started service vmware-vmon
yyyy-mm-ddThh:mm:ss.Z ERROR service-control Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start wcp services. Error: Service crashed while starting

• • /var/log/vmware/wcp/wcpsvc.log -

yyyy-mm-ddThh:mm:ss.Z error wcp Failed to parse VC JWKS: invalid character '<' looking for beginning of value
yyyy-mm-ddThh:mm:ss.Z fatal wcp Unable to get VC public key configuration : invalid character '<' looking for beginning of value

• • /var/log/vmware/sso/tomcat/localhost_access.log - 

[dd/mm/yyyy:hh:mm:ss+xxxx] tomcat-http--43 [Request] 127.0.0.1:45786 to local 443 - HTTP/1.1 GET /openidconnect/jwks/vsphere.local [Response] 404 - 431 bytes [Perf] process 2ms / commit 2ms / conn [+]

• • /var/log/vmware/vsphere-ui/logs/catalina.log -

yyyy-mm-ddThh:mm:ss.Z INFO org.apache.catalina.startup.HostConfig Deploying web application directory [/usr/lib/vmware-sso/vmware-sts/webapps/openidconnect] 
yyyy-mm-ddThh:mm:ss.Z INFO org.apache.catalina.startup.HostConfig Deployment of web application directory [/usr/lib/vmware-sso/vmware-sts/webapps/openidconnect] has finished in [11] ms  

Environment

VMware vCenter Server 7.x

Resolution

Please Note - 

• For a standalone vCenter node, take a "no memory" snapshot of the VCSA VM before proceeding.
• For vCenters in an Enhanced Linked Mode (ELM) configuration, ensure that simultaneous powered-down VM snapshots are taken for all vCenter server nodes in the ELM.

Resolution Steps involved the clean up the Tomcat files -

• Stop all services on vcenter server: 

service-control --stop --all

• Review Files in the Webapps Directory by listing all files in the webapps directory to inspect the contents:

ls -l /usr/lib/vmware-sso/vmware-sts/webapps/

• Delete all unnecessary files and directories in the /usr/lib/vmware-sso/vmware-sts/webapps/ directory except for ROOT.war.

Kindly note it is important to leave ROOT.war untouched, as it will be unpacked upon service restart.

• Once the necessary files are removed, restart all services with the following command:

service-control --start --all