"[500] An error occurred while fetching identity providers" after upgrading VC to 7.0 U2
search cancel

"[500] An error occurred while fetching identity providers" after upgrading VC to 7.0 U2

book

Article ID: 322178

calendar_today

Updated On:

Products

VMware vCenter Server 7.0

Issue/Introduction

  • Post vCenter Server 7.0 U2 upgrade unable to login to VC getting an error "[500] An error occurred while fetching identity providers. Try again"
  • The following entries may be observed in the /var/log/vmware/vsphere_ui/logs/vsphere_client_virgo.log and /var/log/vmware/trustmanagement/trustmanagement-svcs.log files:
vsphere_client_virgo.log
YYYY-MM-DD HH:MM:SS [WARN ] http-nio-5090-exec-9         70000004 100004 ###### c.v.vsphere.client.security.oauth2.logout.LogoutRequestHandler    Unable to determine the identity provider type. Logout request will be skipped.
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler     Received Multi login request
YYYY-MM-DD HH:MM:SS [INFO ] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vise.vim.vapi.StaticEndpointVapiConnectionManager      Connected to vAPI endpoint https://vcenter.example.org:443/site/api
YYYY-MM-DD HH:MM:SS [ERROR] VapiAsyncCall-101             com.vmware.vise.vim.vapi.DefaultVapiConnectionControl             Maximum number of attempts reached while trying to call com.vmware.vcenter.identity.providers.list
YYYY-MM-DD HH:MM:SS [ERROR] http-nio-5090-exec-4         70000005 100004 ###### com.vmware.vsphere.client.security.oauth2.LoginRequestHandler     An error occurred while fetching providers com.vmware.vapi.std.errors.Unauthenticated: Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
    id = vapi.method.authentication.required,
    defaultMessage = This method requires authentication.,
    args = [],
    params = <null>,
    localized = <null>
}],
    data = <null>,
    errorType = UNAUTHENTICATED,
    challenge = <null>
}
        at java.lang.Thread.getStackTrace(Thread.java:1559)
        
        
trustmanagement-svcs.log
YYYY-MM-DD HH:MM:SS [tomcat-exec-14  INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] SAML token for SubjectNameId [value=machine-<machineID>@vsphere.local, format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
YYYY-MM-DD HH:MM:SS [tomcat-exec-14  INFO  com.vmware.identity.token.impl.X509TrustChainKeySelector  opId=] Failed to find trusted path to signing certificate <STS Certificate Subject, example - C=US,CN=ssoserverSign\,dc\=vsphere\,dc\=local>
java.security.cert.CertPathBuilderException: Unable to find certificate chain.
    at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
    at com.vmware.identity.token.impl.X509TrustChainKeySelector.verifyTrustedPathExists(X509TrustChainKeySelector.java:197)
    at com.vmware.identity.token.impl.X509TrustChainKeySelector.select(X509TrustChainKeySelector.java:116)
    at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(DOMXMLSignature.java:557)
    at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(DOMXMLSignature.java:268)
    at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:720)
    at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:562)
    at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:70)
    at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112)
    at com.vmware.vapi.internal.cis.authn.json.JsonSignerImpl.verifySignature(JsonSignerImpl.java:120)
    at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.validateSignature(JsonSignatureVerificationProcessor.java:178)
    at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.process(JsonSignatureVerificationProcessor.java:133)
    at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.requestReceived(JsonServerConnection.java:171)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPostImpl(HttpStreamingServlet.java:119)
    at com.vmware.vapi.protocol.server.rpc.http.impl.HttpStreamingServlet.doPost(HttpStreamingServlet.java:88)
  • This error may also be seen when the vCenter Solution user certificates like machine, webClient, vpxd, vpxd-extension certificates, have expired. To confirm, please run the command below and check that the certificates here are valid: 

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;  

Environment

VMware vCenter Server 7.0.x

Resolution

This is a known issue affecting vCenter Server 7.x. Currently, there is no resolution.

Workaround: Please follow the steps below to reset the STS certificate:

Note: These steps are applicable only if we see the error snippets "Failed to find trusted path to signing certificate" & "Unable to find certificate chain" in the trust manager logs - /var/log/vmware/trustmanagement/trustmanagement-svcs.log.

  1. Download the attached fixsts.sh script from this article and upload to the impacted PSC or vCenter Server with Embedded PSC to the /tmp folder.
  2. If the connection to upload to the vCenter by the SCP client is rejected, run this command from an SSH session to the vCenter: chsh -s /bin/bash
  3. Connect to the PSC or vCenter Server with an SSH session if not done so already.
  4. Navigate to the /tmp directory:
cd /tmp
  1. Run chmod +x fixsts.sh to make the file executable.
  2. Run ./fixsts.sh.
  3. Restart services on all vCenters and/or PSCs in the SSO domain by using below commands:

    service-control --stop --all
    service-control --start --all

Note: For more details on resetting the STS certificate, please refer to "Signing certificate is not valid" error in vCenter Server Appliance"

Expired Certificates: 
  1. If expired certificates are found, such as the Machine SSL/Solution Users, please renew the certificates to resolve the issue. 
  2. Check certificate status with the command below, when logged into VC through SSH session as root:
    • for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done; 
  3. If certs are expired, please refer to this KB to renew the certificates How to use vSphere Certificate Manager to Replace SSL Certificates

Attachments

fixsts get_app
Powered by Wolken Software