Accessing vCenter Server fails with error "[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server" due to expired solution user certificates
search cancel

Accessing vCenter Server fails with error "[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server" due to expired solution user certificates

book

Article ID: 390983

calendar_today

Updated On:

Products

VMware vCenter Server 7.0 VMware vCenter Server 8.0

Issue/Introduction

  • Message prompts similar to the messages listed below are encountered when accessing vCenter Server Web Client UI from a web browser:

    [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - java.lang.reflect.InvocationTargetException
    [400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - com.vmware.vcenter.apigw.api.sso.tokenmgmt.TokenException: Failed to acquire an API GW service-principal token.
  • Multiple services including vmware-vpxd and vmware-sps fail to start.

  • /var/log/vmware/vpxd/vpxd.log has the following messages:

    AcquireToken exception: N9SsoClient27InvalidCredentialsExceptionE(Authentication failed: Invalid credentials)

  • /var/log/vmware/vmon/vmon.log has the following messages:

    YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     hok_token = self.perform_request(soap_message, public_key, private_key,
    YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-####   File "/usr/lib/vmware/site-packages/pyVim/sso.py", line 277, in perform_request
    YYYY-MM-DDTHH:MM:SSZ Wa(03) host-#### <vpxd-svcs> Service pre-start command's stderr:     raise SoapException(fault, *parsed_fault)
    YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### pyVim.sso.SoapException: SoapException:
    YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-#### faultcode: ns0:FailedAuthentication
    YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-2405 faultstring: Invalid credentials
    YYYY-MM-DDTHH:MM:SSZ Wa(03)+ host-####5 faultxml: <?xml version='1.0' encoding='UTF-8'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/2005
    12">ns0:FailedAuthentication</faultcode><faultstring>Invalid credentials</faultstring></S:Fault></S:Body></S:Envelope>

  • /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log has the following messages

    YYYY-MM-DDTHH:MM:SSZ [ERROR] agw-token-acq6972  com.vmware.vim.sso.client.impl.SoapBindingImpl                    SOAP fault com.sun.xml.ws.fault.ServerSOAPFaultException:
    Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
            at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:163)
            at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:103)
            at com.sun.xml.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:244)
            at com.sun.xml.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:274)
            at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:197)
            at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:120)
            at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983)
            at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902)
            at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509)
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.lambda$doAcquireTokenForServicePrincipal$3(Unknown Source)
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.invokeStsClient(Unknown Source)
               Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: Failed to acquire a service-principal token
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.wrap(Unknown Source)
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.acquireTokenForServicePrincipal(Unknown Source)
            at com.vmware.vcenter.apigw.sso.impl.ApiGwServicePrincipal.acquireTokenForDomain(Unknown Source)
            at com.vmware.vcenter.apigw.sso.impl.ApiGwServicePrincipal.lambda$doGetTokenAsync$0(Unknown Source)
            at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
            at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
            at io.opentelemetry.context.Context.lambda$wrap$1(Context.java:212)
            ... 3 common frames omitted
    Caused by: com.vmware.vcenter.apigw.api.sso.SsoServiceException: SSO invocation for domain vsphere.local(#########--###-###-#########) to STS http://localhost:1080/external-vecs/htt
    p2/<vCenter FQDN>/443/sts/STSService/vsphere.local failed
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.wrap(Unknown Source)
            at com.vmware.vcenter.apigw.sso.impl.SsoServiceImpl.invokeRetriable(Unknown Source)
            ... 9 common frames omitted 
                 

  • One or more Solution users certificates are found to have expired upon running following command as per Verify and resolve expired vCenter Server certificates using the command line interface :

    for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Environment

  • VMware vCenter Server 7.x
  • VMware vCenter Server 8.x
  • VMware vCenter Server 9.x

Cause

  • Expired Solution User certificates on the vCenter Server cause services to fail and users will not be able to log in from the vSphere Web Client.

Resolution

  1. Take an appropriate snapshot of the vCenter server VM, as referenced here: Snapshot Best practices for vCenter Server Virtual Machines
  2. Renew the vCenter Server Solution User certificates by following the steps from this KB vCert 
Powered by Wolken Software