Getting error "FATA[0064] Failed to get available workloads, response from the server was invalid." when trying to login to the Guest cluster using kubectl-vsphere login.
search cancel

Getting error "FATA[0064] Failed to get available workloads, response from the server was invalid." when trying to login to the Guest cluster using kubectl-vsphere login.

book

Article ID: 395552

calendar_today

Updated On:

Products

Tanzu Kubernetes Runtime

Issue/Introduction

  • You attempt to login to the guest cluster using "kubectl-vsphere login" and get error "FATA[0064] Failed to get available workloads, response from the server was invalid."
  • The login fails when using a domain user and works fine with the local sso account.
  • Adding the verbose flag (-v=10) to the kubectl command gives below error message.

    DEBU[0064] Got response: <html>
    <head><title>504 Gateway Time-out</title></head>
    <body>
    <center><h1>504 Gateway Time-out</h1></center>
    <hr><center>nginx/1.25.2</center>
    </body>
    /html>

    DEBU[0064] Error while getting list of workloads: invalid character '<' looking for beginning of value
    FATA[0064] Failed to get available workloads, response from the server was invalid. 

 

  • The wcp-auth pods confirm a successful login for the domain user.

    INFO:vclib.sso:[] Got bearer token for <user@domain>.
    INFO:vclib.sso:[] Got hok token for /etc/vmware/wcp/tls/wcpusr.cert.
    DEBUG:auth.authentication_sso:isExpired, notBefore: None, notOnOrAfter: None, now: <date> <time>, tolerance: 600
    DEBUG:auth.authentication_sso:isExpired, notBefore: <date>:<time>, notOnOrAfter: <date>:<time>, now: <date> <time>, tolerance: 600
    INFO:auth.filters:[] User authenticated using basic token.
    DEBUG:telemetry.telemetry_object:Adding 1 successful auth request.

 

  • In vmware-identity.sts.log, you see that the communication with the identity provider is broken so sts is unable to authorize the token as a result.

    ERROR sts[77:tomcat-http--31] [CorId=bd4a1bc3-955f-4b6a-a628-d378b1ce44e7] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldaps://<ldap-server.domain>:636] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use secondary URIs, if applicable

 

Environment

vSphere with Tanzu
VMware vCenter Server

Cause

The vCenter server is unable to establish communication with your identity provider. To check and confirm the same run "curl -v -k <Identity-provider-url.domain.com>:636" (for ldaps) and "curl -v -k <Identity-provider-url.domain.com>:389" (for ldap).

Resolution

Fix the connectivity issue between the vCenter Server and your Identity provider.

Powered by Wolken Software