kubectl vsphere login failed with "Failed to get available workloads: invalid character"
search cancel

kubectl vsphere login failed with "Failed to get available workloads: invalid character"

book

Article ID: 390527

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

  • Login Supervisor cluster failed with the below messages:

    # kubectl vsphere login --server <Supervisor-VIP> --vsphere-username <Username> -v=6

    DEBU[0000] Got response:
    <html>
      <head><title>500 - Internal Server Error</title></head>
      <body>
        <h1>Internal Server Error</h1>
        <p>Internal Server Error</p>
      </body>
    </html>
    FATA[0000] Failed to get available workloads: invalid character '<' looking for beginning of value

  • The status of Supervisor cluster is in 'Configuring' with below messages:

    Status Messages:
    Resource Type Deployment, Identifier vmware-system-netop/vmware-system-netop-controller-manager is not found.
    Node Kubernetes health status is Unknown
    Node Kubernetes health status is Unknown
    Node Kubernetes health status is Unknown
    Node Kubernetes health status is Unknown

  • The Supervisor cluster nodes are in Ready status and all Pods are in Running status.

Environment

VMware vSphere with Tanzu

Cause

The EAM service can not connect to vCenter Server service that causes WCP can not get the Supervisor agency information to connect to Supervisor cluster. There is a mismatch between vpxd-extension certificate stored in VECS and the certificate information stored in vCenter Server Database for EAM extension.

Resolution

To verify the issue:

  1. In vCenter Server /var/log/vmware/wcp/wcpsvc.log there may be below messages:

    2025-03-10T03:36:29.713Z error wcp [eamlib/lister.go:84] [opID=EAMAgent] Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later.
    2025-03-10T03:36:29.713Z error wcp [informer/informer.go:129] [opID=EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later.

  2. In vCenter Server /var/log/vmware/eam/eam.log there may be below messages:

    2025-03-09T18:55:35.423Z |  WARN | vim-async-2 | ExtensionSessionRenewer.java | 227 | [Retry:Login:com.vmware.vim.eam:271b1b2773d8356d] Re-login failed, due to:
    com.vmware.eam.security.NotAuthenticated: Failed to authenticate extension com.vmware.vim.eam to vCenter.
    Caused by: com.vmware.vim.binding.vim.fault.InvalidLogin: Cannot complete login due to an incorrect user name or password.

To fix the issue:

  1. Update vpxd-extension certificate to match between VECS and vCenter Server database with the KB EAM "Failed to login to vCenter as extension, Cannot complete login due to an incorrect user name or password" after replacing the vCenter Server certificates
  2. Restart the EAM and WCP services:

    service-control --stop vmware-eam && service-control --start vmware-eam
    service-control --stop wcp && service-control --start wcp
Powered by Wolken Software