"Error while getting list of workloads: invalid character '<' looking for beginning of value" during kubectl vsphere login to a VKS Guest Cluster.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When logging in to a VKS Workload cluster, following error message is received: 
  Error while getting list of workloads: invalid character '<' looking for beginning of value
• Failed login example:
  kubectl vsphere login --server=<Supervisor VIP> --vsphere-username <username> --tanzu-kubernetes-cluster-name <VKS Guest Cluster name> --tanzu-kubernetes-cluster-namespace <Namespace_Name> --insecure-skip-tls-verify
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
FATA[YYYY-MM-DDTHH:MM:SS] Error while getting list of workloads: invalid character '<' looking for beginning of value
  
  
• Log entries observed within the wcp-authproxy pod running on the Supervisor Cluster: 
  kubectl logs wcp-authproxy-<pod_id> -n vmware-system-auth 
YYYY-MM-DD HH:MM:SS [-] Unhandled error in Deferred:
YYYY-MM-DD HH:MM:SS [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 1318, in run
            self.mainLoop()
          File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 1328, in mainLoop
            reactorBaseSelf.runUntilCurrent()
          File "/usr/lib/python3.10/site-packages/twisted/internet/base.py", line 994, in runUntilCurrent
            call.func(*call.args, **call.kw)
          File "/usr/lib/python3.10/site-packages/twisted/internet/task.py", line 251, in __call__
            d = maybeDeferred(self.f, *self.a, **self.kw)
        --- <exception caught here> ---
          File "/usr/lib/python3.10/site-packages/twisted/internet/defer.py", line 206, in maybeDeferred
            result = f(*args, **kwargs)
          File "/authproxy/telemetry/telemetry.py", line 129, in _push
            response = requests.post(
          File "/usr/lib/python3.10/site-packages/requests/api.py", line 117, in post
            return request('post', url, data=data, json=json, **kwargs)
          File "/usr/lib/python3.10/site-packages/requests/api.py", line 61, in request
            return session.request(method=method, url=url, **kwargs)
          File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 542, in request
            resp = self.send(prep, **send_kwargs)
          File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 655, in send
            r = adapter.send(request, **kwargs)
          File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 696, in send
            raise SSLError(e, request=request)
        requests.exceptions.SSLError: HTTPSConnectionPool(host='<vCenter_Server-FQDN>', port=443): Max retries exceeded with url: /analytics/telemetry/ph/api/hyper/send?_c=SVC.1_0_U1&_i=#####-###-####-########### (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
  
  
• On the vCenter Server, following events are seen for Supervisor under Workload Management depicting that vCenter Server's Machine SSL certificate is reported as expired: 
  
• Open an SSH session to the vCenter Server and log in as root and run the below command to confirm that the Machine SSL certificate is expired: 
  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
   

Environment

VMware vSphere Kubernetes Service

Cause

• This issue occurs because the Machine SSL certificate on the vCenter Server has expired.
• Because the certificate is expired, the wcp-authproxy pod on the Supervisor cluster loses trust with vCenter and actively refuses the API connection. As a result, the authentication proxy fails and returns a standard HTTP 500 error page formatted in HTML.

Resolution

To resolve the issue, renew the expired Machine SSL certificate on the vCenter Server by following the steps from any of the below mentioned KB articles. 
Note: Take a proper Snapshot for vCenter Server before making any changes.

• vCert - Scripted vCenter expired certificate replacement
• Using vSphere Certificate Manager to Replace SSL Certificate

Additional Information

Failed to get available workloads: invalid character '<' looking for beginning of value