WCP service fails to start due to STS openidconnect JWKS endpoint not being available

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• ESXi hosts fail to enter Maintenance Mode when using vSphere Client
• WCP service is stopped and fails to successfully
• /var/log/vmware/wcp/wcpsvc.log contains the following messages:

  2023-02-13T05:20:36.8Z error wcp Failed to parse VC JWKS: invalid character '<' looking for beginning of value
  2023-02-13T05:20:36.8Z fatal wcp Unable to get VC public key configuration : invalid character '<' looking for beginning of value

• In /var/log/vmware/sso/tomcat/localhost_access.log, the an entry similar to the following message can be found:

  [12/Feb/2023:07:26:08 +0000] tomcat-http--19 [Request] 127.0.0.1:39344 to local 443 - HTTP/1.1 GET /openidconnect/jwks/vsphere.local [Response] 404 - 431 bytes [Perf] process 0ms / commit 0ms / conn [+]

Environment

VMware vCenter Server 7.0.x

Cause

The JWKS endpoint of the Security Token Service (STS) in vCenter is missing, causing Workload Control Plane (WCP) to be unable to get the OpenID configuration needed to generate Single Sign-On (SSO) authentication. The issue may be due to missing or corrupted STS files, or the Tomcat web server hosting STS and OpenID Connect may not recognize the path as a valid path to any application.

Resolution

The following steps will delete the existing copy of the Secure Token Service (STS) installation, and force vCenter to re-install STS again.

NOTE: Please ensure to take a fresh backup or an offline snapshot before attempting to follow the steps below. In case the affected vCenter is part of an Enhanced Linked Mode replication (ELM) group, please ensure to take backups or offline snapshots of all members of the ELM setup. When in need of restore, all ELM partners need to be restored, as otherwise the VMDir instances of the ELM partner will go out of sync.
 

1. Open the following URLs in a web browser to confirm that both return an HTTP 404:

   https://<VC-IP-or-FQDN>/openidconnect/jwks/vsphere.local
   https://<VC-IP-or-FQDN>/openidconnect/vsphere.local/.well-known/openid-configuration

2. Open an SSH connection to the VCSA and login with the root account
3. Stop all vCenter Services:

   # service-control --stop --all

4. change into the STS installation folder:

   # cd /usr/lib/vmware-sso/vmware-sts/webapps

5. Move the complete folder ROOT out of the current patch into ~, to ensure there is a copy of it. Do NOT touch the ROOT.war file:

   # mv ROOT ~/ROOT

6. Start all vCenter Services:

   # service-control --start --all

7. Verify that the folder ROOT was recreated in /usr/lib/vmware-sso/vmware-sts/webapps
8. Verify that OpenID Connect JWKS endpoint works by opening the following URL in your browser:

   https://<VC-IP-OR-FQDN>/openidconnect/jwks/vsphere.local

   The output in the browser should look similar to the censored screenshot below:
   
9. Confirm that the WCP service was successfully started:

   # service-control --status wcp