Symptoms:
- Post vCenter Server 7.0 U2 upgrade unable to login to VC getting an error "[500] An error occurred while fetching identity providers. Try again"
- You might see similar log snippet in vsphere_client_virgo and trustmanagement-svcs.log log files
vsphere_client_virgo.log
YYYY-MM-DD HH:MM:SS [WARN ] http-nio-5090-exec-9 70000004 100004 ###### c.v.vsphere.client.security.oauth2.logout.LogoutRequestHandler Unable to determine the identity provider type. Logout request will be skipped.
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler Received Multi login request
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vise.vim.vapi.StaticEndpointVapiConnectionManager Connected to vAPI endpoint https://vcenter.test.lab:443/site/api
YYYY-MM-DD HH:MM:SS [ERROR] VapiAsyncCall-101 com.vmware.vise.vim.vapi.DefaultVapiConnectionControl Maximum number of attempts reached while trying to call com.vmware.vcenter.identity.providers.list
YYYY-MM-DD HH:MM:SS [ERROR] http-nio-5090-exec-4 70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler An error occurred while fetching providers com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
id = vapi.method.authentication.required,
defaultMessage = This method requires authentication.,
args = [],
params = <null>,
localized = <null>
}],
data = <null>,
errorType = UNAUTHENTICATED,
challenge = <null>
}
at java.lang.Thread.getStackTrace(Thread.java:1559)
trustmanagement-svcs.log
YYYY-MM-DD HH:MM:SS [tomcat-exec-14 INFO com.vmware.identity.token.impl.SamlTokenImpl opId=] SAML token for SubjectNameId [value=machine-<machineID>@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
YYYY-MM-DD HH:MM:SS [tomcat-exec-14 INFO com.vmware.identity.token.impl.X509TrustChainKeySelector opId=] Failed to find trusted path to signing certificate <STS Certificate Subject, example - C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local>
java.security.cert.CertPathBuilderException: Unable to find certificate chain.
at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at com.vmware.identity.token.impl.X509TrustChainKeySelector.verifyTrustedPathExists(X509TrustChainKeySelector.java:197)
at com.vmware.identity.token.impl.X509TrustChainKeySelector.select(X509TrustChainKeySelector.java:116)
at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(DOMXMLSignature.java:557)
at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(DOMXMLSignature.java:268)
at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:720)
at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:562)
at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:70)
at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112)
at com.vmware.vapi.internal.cis.authn.json.JsonSignerImpl.verifySignature(JsonSignerImpl.java:120)
at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.validateSignature(JsonSignatureVerificationProcessor.java:178)
at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.process(JsonSignatureVerificationProcessor.java:133)
at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:171)
at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:119)
at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:88)
- This error may also be seen when vCenter Machine Certificates have also expired - to confirm, please run below command and check certificates here are valid:
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;