Accessing vCenter Server fails with Error: [400] vCenter SSO authentication request failure due to expired certificates in vCenter Server

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• When accessing the vSphere Client, one or more messages similar to those shown below may be encountered:
  
  [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - java.lang.reflect.InvocationTargetException
[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Failed to acquire an API GW service-principal token.


• vSphere UI inaccessible and loads with the following error message: "no healthy upstream".


• Multiple services including vmware-vpxd and vmware-sps fail to start.
  
  
• /var/log/vmware/vpxd/vpxd.log has the following log traces:
  
  AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)
  
  
• /var/log/vmware/vmon/vmon.log has the following log traces:
  
  YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     hok_token = self.perform_request(soap_message, public_key, private_key,
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-####   File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     raise SoapException(fault, *parsed_fault)
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### pyVim.sso.SoapException: SoapException:
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### faultcode: ns0:FailedAuthentication
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### faultstring: Invalid credentials
YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault mlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcodexmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Invalidcredentials</faultstring></S:Fault></S:Body></S:Envelope>
  
  
• /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log has the following log traces:
  
  YYYY-MM-DDTHH:MM:SSZ [ERROR] agw-token-acq6972  com.vmware.vim.sso.client.impl.SoapBindingImpl                    SOAP fault com.sun.xml.ws.fault.ServerSOAPFaultException:
Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
        at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163)
        at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:103)
        at com.sun.xml.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:24)
        at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:197)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902)
        at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509)
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.lambda$doAcquireTokenForServicePrincipal$3(Unknown Source)
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.invokeStsClient(Unknown Source)
Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to acquire a service-principal token
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.wrap(Unknown Source)
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.acquireTokenForServicePrincipal(Unknown Source)
        at com.vmware.vcenter.apigw.sso.impl.ApiGwServicePrincipal.acquireTokenForDomain(Unknown Source)
        at com.vmware.vcenter.apigw.sso.impl.ApiGwServicePrincipal.lambda$doGetTokenAsync$0(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
        at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:212)
        ... 3 common frames omitted
Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: SSO invocation for domain vsphere.local(#########--###-###-#########) to STS http://localhost:1080/external-vecs/http2/<vCenter_FQDN>/443/sts/STSService/vsphere.local failed
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.wrap(Unknown Source)
        at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.invokeRetriable(Unknown Source)
        ... 9 common frames omitted 
               
• Solution User certificates were found to be expired after running the following command as per Verify and resolve expired vCenter Server certificates using the command line interface :
  
  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Environment

• VMware vCenter Server 7.x
• VMware vCenter Server 8.x
• VMware vCenter Server 9.x

Cause

Solution User (machine, vsphere-webclient, vpxd, vpxd-extension)  certificates on the vCenter Server has expired which causes services to fail, hence preventing users from logging into the vSphere UI.

Resolution

1. Take an appropriate snapshot of the vCenter Server Virtual Machine, as referenced here: Snapshot Best practices for vCenter Server Virtual Machines.
2. Login to the vCenter Server via SSH using root access.
3. Renew the vCenter Server Solution User certificates by following the procedure outlined in the Knowledge Base article: vCert - Scripted vCenter expired certificate replacement