Accessing vCenter Server fails with error "[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server" due to expired solution user certificates

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• Message prompts similar to the messages listed below are encountered when accessing vCenter Server Web Client UI from a web browser:
  
  [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - java.lang.reflect.InvocationTargetException
  
  

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Failed to acquire an API GW service-principal token.

• Multiple services including vmware-vpxd and vmware-sps fail to start.
  
  
  • From /var/log/vmware/vpxd/vpxd.log has the following messages:
  AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
  
  
  • From /var/log/vmware/vmon/vmon.log has the following messages:

  
  YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     hok_token = self.perform_request(soap_message, public_key, private_key,
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-####   File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     raise SoapException(fault, *parsed_fault)
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### pyVim.sso.SoapException: SoapException:
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### faultcode: ns0:FailedAuthentication
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-2405 faultstring: Invalid credentials
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-####5 faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/2005
12">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

  
  
  
• One or more Solution users certificates are found to have expired upon running following command as per Verify and resolve expired vCenter Server certificates using the command line interface :
  
  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

Expired Solution User certificates on the vCenter Server cause services to fail and users will not be able to log in from the vSphere Web Client.

Resolution

Note: Take an appropriate snapshot of the vCenter server VM, as referenced here: Snapshot Best practices for vCenter Server Virtual Machines

To resolve the issue, renew the vCenter Server Solution User certificates using the vCert script with VMCA as the certificate authority. Follow the detailed steps outlined in the VMware KB article: How to replace the vCenter Server Solution User certificates with VMCA issued certificate