"[500] An error occurred while fetching identity providers" after upgrading VC to 7.0 U2

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• Post vCenter Server 7.0 U2 upgrade unable to login to VC getting an error "[500] An error occurred while fetching identity providers. Try again"

• You might see similar log snippet in vsphere_client_virgo and trustmanagement-svcs.log log files

vsphere_client_virgo.log
YYYY-MM-DD HH:MM:SS [WARN ] http-nio-5090-exec-9         70000004 100004 ###### c.v.vsphere.client.security.oauth2.logout.LogoutRequestHandler    Unable to determine the identity provider type. Logout request will be skipped.
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler     Received Multi login request
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vise.vim.vapi.StaticEndpointVapiConnectionManager      Connected to vAPI endpoint https://vcenter.test.lab:443/site/api
YYYY-MM-DD HH:MM:SS [ERROR] VapiAsyncCall-101             com.vmware.vise.vim.vapi.DefaultVapiConnectionControl             Maximum number of attempts reached while trying to call com.vmware.vcenter.identity.providers.list
YYYY-MM-DD HH:MM:SS [ERROR] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler     An error occurred while fetching providers com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = vapi.method.authentication.required,
    defaultMessage = This method requires authentication.,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = UNAUTHENTICATED,
    challenge = <null>
}
        at java.lang.Thread.getStackTrace(Thread.java:1559)
        
        
trustmanagement-svcs.log
YYYY-MM-DD HH:MM:SS [tomcat-exec-14  INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] SAML token for SubjectNameId [value=machine-<machineID>@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
YYYY-MM-DD HH:MM:SS [tomcat-exec-14  INFO  com.vmware.identity.token.impl.X509TrustChainKeySelector  opId=] Failed to find trusted path to signing certificate <STS Certificate Subject, example - C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local>
java.security.cert.CertPathBuilderException: Unable to find certificate chain.
    at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at com.vmware.identity.token.impl.X509TrustChainKeySelector.verifyTrustedPathExists(X509TrustChainKeySelector.java:197)
    at com.vmware.identity.token.impl.X509TrustChainKeySelector.select(X509TrustChainKeySelector.java:116)
    at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(DOMXMLSignature.java:557)
    at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(DOMXMLSignature.java:268)
    at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:720)
    at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:562)
    at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:70)
    at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112)
    at com.vmware.vapi.internal.cis.authn.json.JsonSignerImpl.verifySignature(JsonSignerImpl.java:120)
    at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.validateSignature(JsonSignatureVerificationProcessor.java:178)
    at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.process(JsonSignatureVerificationProcessor.java:133)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:171)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:119)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:88)

 

 

• This error may also be seen when vCenter Machine Certificates have also expired - to confirm, please run below command and check certificates here are valid: 

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;  

Environment

VMware vCenter Server 7.0.x

Resolution

This is a known issue affecting vCenter Server 7.x. Currently, there is no resolution.

Workaround:
To workaround the issue, please follow the below steps to reset the STS certificate :

Note: These steps are applicable only if we see the error snippets "Failed to find trusted path to signing certificate" & "Unable to find certificate chain" in the trust manager logs - /var/log/vmware/trustmanagement/trustmanagement-svcs.log.

1. Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
2. If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter: chsh -s /bin/bash
3. Connect to the PSC or vCenter Server with an SSH session if you have not already per Step 2.
4. Navigate to the /tmp directory:

cd /tmp

4. Run chmod +x fixsts.sh to make the file executable.
5. Run ./fixsts.sh.
6. Restart services on all vCenters and/or PSCs in your SSO domain by using below commands:
   
   service-control --stop --all
   service-control --start --all

Note: For more details on resetting STS certificate refer this "Signing certificate is not valid" error in vCenter Server Appliance

 

Expired Certificates: 

If you find that some Machine Certificates are expired such as Machine SSL/Solution Users, please renew certificates to resolve the issue:

 

--> Check certificate status with below command, when logged into VC through SSH session as root: 

 

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;  

 

 

--> If some expired, please apply following for steps How to use vSphere Certificate Manager to Replace SSL Certificates