Getting error "FATA[0064] Failed to get available workloads, response from the server was invalid." when trying to login to the Guest cluster using kubectl-vsphere login.
search cancel

Getting error "FATA[0064] Failed to get available workloads, response from the server was invalid." when trying to login to the Guest cluster using kubectl-vsphere login.

book

Article ID: 395552

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime

Issue/Introduction

  • You attempt to login to the guest cluster using "kubectl-vsphere login" and get error "FATA[0064] Failed to get available workloads, response from the server was invalid."
  • The login fails when using a domain user and works fine with the local sso account.
  • Adding the verbose flag (-v=10) to the kubectl command gives below error message.

    DEBU[0064] Got response: <html>
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</h1></center>
    <hr><center>nginx/1.25.2</center>
    </body>
    /html>

    DEBU[0064] Error while getting list of workloads: invalid character '<' looking for beginning of value
    FATA[0064] Failed to get available workloads, response from the server was invalid. 

 

  • The wcp-auth pods confirm a successful login for the domain user.

    INFO:vclib.sso:[] Got bearer token for <user@domain>.
    INFO:vclib.sso:[] Got hok token for /etc/vmware/wcp/tls/wcpusr.cert.
    DEBUG:auth.authentication_sso:isExpired, notBefore: None, notOnOrAfter: None, now: <date> <time>, tolerance: 600
    DEBUG:auth.authentication_sso:isExpired, notBefore: <date>:<time>, notOnOrAfter: <date>:<time>, now: <date> <time>, tolerance: 600
    INFO:auth.filters:[] User authenticated using basic token.
    DEBUG:telemetry.telemetry_object:Adding 1 successful auth request.

 

  • In vmware-identity.sts.log, you see that the communication with the identity provider is broken so sts is unable to authorize the token as a result.

    ERROR sts[77:tomcat-http--31] [CorId=bd4a1bc3-955f-4b6a-a628-d378b1ce44e7] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldaps://<ldap-server.domain>:636] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use secondary URIs, if applicable

 

Environment

vSphere with Tanzu
VMware vCenter Server

Cause

It could be due to following reasons:

- vCenter to Identitiy provider communication: The vCenter server is unable to establish communication with your identity provider. To check and confirm the same run
curl -v -k <Identity-provider-url.domain.com>:636 (for ldaps)
and
curl -v -k <Identity-provider-url.domain.com>:389 (for ldap).
Also check and confirm the certificate if using LDAPS with:
echo | openssl s_client -showcerts -servername  <Identity-provider-url.domain.com> -connectL <Identity-provider-url.domain.com>:443  | openssl x509 -inform pem -noout -text

- The login requires to define the domainname, this issue also can occur when not defining the domainname. For example:
username - fails, [email protected] - works

Resolution

Fix the connectivity issue between the vCenter Server and your Identity provider or use the full domainname when logging in.

Powered by Wolken Software