"Couldn't establish a connection to the VM web console" error in vSphere UI
search cancel

"Couldn't establish a connection to the VM web console" error in vSphere UI

book

Article ID: 431643

calendar_today

Updated On:

Products

VMware vCenter Server 8.0

Issue/Introduction

  • When attempting to access a Virtual Machine (VM) web console through a vCenter Server configured in Enhanced Linked Mode (ELM), the connection fails. The vSphere Client displays the following error:

        "Couldn't establish a connection to the VM web console."

  • The issue occurs when accessing VMs across different vCenter nodes in the SSO domain.

  • Accessing the VM console directly via the ESXi Host Client works as expected.

  • In vCenter server from /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log,the logs indicate a failure to build a trusted path for signing certificates, leading to authentication data being missing.

    YYYY-MM-DDTHH:MM:SSZ [INFO ] p-nio-127.0.0.1-5090-exec-94 r0006715 ###### ###### com.vmware.identity.token.impl.X509TrustChainKeySelector Failed to find trusted path to signing certificate <CN=ssoserverSign> java.security.cert.CertPathBuilderException: Unable to find certificate chain.
            at org.bouncycastle.jcajce.provider.PKIXCertPathBuilderSpi.engineBuild(Unknown Source)
            at com.vmware.identity.token.impl.X509TrustChainKeySelector.verifyTrustedPathExists(X509TrustChainKeySelector.java:197)
            at com.vmware.vapi.cis.authn.json.JsonSignatureVerificationProcessor.process(JsonSignatureVerificationProcessor.java:134)
    YYYY-MM-DDTHH:MM:SSZ [INFO ] p-nio-127.0.0.1-5090-exec-94 r0006715 ###### ###### com.vmware.vapi.security.AuthenticationFilter Authentication failed java.lang.RuntimeException: Authentication data not found
            at com.vmware.vapi.cis.authn.SamlTokenAuthnHandler.authenticate(SamlTokenAuthnHandler.java:57)
            at com.vmware.vapi.security.AuthenticationFilter.invoke(AuthenticationFilter.java:233)
            at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processApiRequest(JsonServerConnection.java:424)
            at com.vmware.vapi.protocol.server.msg.json.JsonServerConnection.processRequest(JsonServerConnection.java:236)
    aused by: com.vmware.vapi.dsig.json.SignatureException: Cannot verify the signature over the provided data
            at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112)
            at com.vmware.vapi.internal.cis.authn.json.JsonSignerImpl.verifySignature(JsonSignerImpl.java:103)
            ... 58 common frames omitted
    Caused by: com.vmware.vim.sso.client.exception.MalformedTokenException: Signature validation failed
            at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:736)
            ... 64 common frames omitted
    Caused by: javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
            at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(DOMXMLSignature.java:563)
            ... 67 common frames omitted

  • From /var/log/vmware/sso/vmware-identity-sts-default.log,errors confirm that the Security Token Service (STS) cannot issue Holder-of-Key (HoK) tokens because the certificate timestamps are invalid (expiration date is not after the start time).

    YYYY-MM-DDTHH:MM:SSZ INFO sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.saml.impl.TokenLifetimeRemediator] There is a HoK confirmation certificate with end time:  YYYY-MM-DDTHH:MM:SS.000+0000
    YYYY-MM-DDTHH:MM:SSZ ERROR sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.providers.SolutionUserHokTokenProviderImpl] Unable to get SAML HOK token for machine solution user
    java.lang.IllegalArgumentException: EndTime: MM DD:TT:SS GMT YYYY is not after startTime: MM DD:TT:SS GMT GMT YYYY
            at com.vmware.identity.util.TimePeriod.<init>(TimePeriod.java:48) ~[libsamlauthority.jar:?]
            at com.vmware.identity.saml.impl.TokenLifetimeRemediator.remediateTokenValidity(TokenLifetimeRemediator.java:73) ~[libsamlauthority.jar:?]
            at com.vmware.identity.saml.impl.TokenAuthorityImpl.issueToken(TokenAuthorityImpl.java:187) ~[libsamlauthority.jar:?]
    YYYY-MM-DDTHH:MM:SSZ ERROR sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.util.VcTrustCache] Refresh thread failed to retreive Vctrusts.
    java.lang.Exception: Could not get Saml HOK token for solution user machine
            at com.vmware.identity.util.VapiClientConnection.createConnection(VapiClientConnection.java:91) ~[libsamlauthority.jar:?]
            at com.vmware.identity.util.VapiClientConnection.refreshConnection(VapiClientConnection.java:157) ~[libsamlauthority.jar:?]
    YYYY-MM-DDTHH:MM:SSZ INFO sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.saml.impl.TokenLifetimeRemediator] There is a HoK confirmation certificate with end time:  YYYY-MM-DDTHH:MM:SS.000+0000
    YYYY-MM-DDTHH:MM:SSZ ERROR sts-default[30:Thread-9] [CorId= OpId=] [com.vmware.identity.providers.SolutionUserHokTokenProviderImpl] Unable to get SAML HOK token for machine solution user
    java.lang.IllegalArgumentException: EndTime: MM DD:TT:SS GMT YYYY is not after startTime: MM DD:TT:SS GMT YYYY

  • From /var/log/vmware/trustmanagement/trustmanagement-svcs.log, the logs explicitly state that the token expiration date has already passed.

    YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-10 [] INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] SAML token for SubjectNameId [value=machine-4b###4fe-c##5-4##4-8##c-74####[email protected], format=http://schemas.xmlsoap.org/claims/UPN] successfully parsed from XML
    YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-10 [] INFO  com.vmware.identity.token.impl.SamlTokenImpl  opId=] Token expiration date:MM DD:TT:SS GMT YYYY is in the past.
    YYYY-MM-DDTHH:MM:SSZ [tomcat-exec-10 [] INFO  com.vmware.vapi.security.AuthenticationFilter  opId=] Authentication failed
    java.lang.RuntimeException: Authentication data not found
            at com.vmware.vapi.cis.authn.SamlTokenAuthnHandler.authenticate(SamlTokenAuthnHandler.java:57) [vapi-authn-2.100.0.jar:?]
            at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:935) [tomcat-embed-core-9.0.104.jar:9.0.104]
            at java.lang.Thread.run(Thread.java:750) [?:1.8.0_452]
    Caused by: com.vmware.vapi.dsig.json.SignatureException: Cannot verify the signature over the provided data
            at com.vmware.vapi.internal.cis.authn.json.JsonSignatureStruct.parseJsonSignatureStruct(JsonSignatureStruct.java:112) ~[vapi-authn-2.100.0.jar:?]
    Caused by: com.vmware.vim.sso.client.exception.InvalidTimingException: Token expiration date:MM DD:TT:SS GMT YYYY is in the past.
            at com.vmware.identity.token.impl.SamlTokenImpl.validateWithinTokenLifePeriod(SamlTokenImpl.java:915) ~[samltoken-1.0.jar:?]
            at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:573) ~[samltoken-1.0.jar:?]
            ... 27 more

Environment

VMware vCenter Server 7.0.X 

VMware vCenter Server 8.0.X 

Cause

The root cause is an expired Security Token Service (STS) Certificate.

In an ELM environment, vCenter relies on SAML tokens to proxy console connections between nodes. If the STS certificate is expired, the identity provider cannot sign new tokens, and existing tokens are rejected because their expiration timestamps are in the past. This prevents the "Solution User" (machine account) from establishing a trusted session for the web console

Resolution

To resolve the issue, the expired STS certificate needs to be replaced:

  1. Take an offline snapshot of all vCenter Servers that are part of Enhanced Linked Mode.

  2. Execute the vCert tool script on one vCenter Server within the Enhanced Linked Mode environment to replace the expired STS certificate (vCert – Scripted vCenter Expired Certificate Replacement).

  3. After the STS certificate has been replaced successfully, restart the vCenter services on each vCenter Server in Enhanced Linked Mode, once at a time, using the following command:

        service-control --stop --all && service-control --start --all

Powered by Wolken Software