Failed to get available workloads: invalid character '<' looking for beginning of value
search cancel

Failed to get available workloads: invalid character '<' looking for beginning of value

book

Article ID: 313101

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere with Tanzu

Issue/Introduction

There could be multiple reasons for this problem to occur, to narrow down follow the steps below:

Example of unsuccessful login with -v 6 :
kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6

DEBU[0000] User passed verbosity level: 6
DEBU[0000] Setting verbosity level: 6
DEBU[0000] Setting request timeout:
DEBU[0000] login called as: kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6
DEBU[0000] Creating wcp.Client for <SUPERVISOR VIP>.
INFO[0000] Does not appear to be a vCenter or ESXi address.
DEBU[0000] Got response: Welcome to Photon 3.0 (\m) - Kernel \r (\l)
Welcome to Photon 3.0 (\m) - Kernel \r (\l)


INFO[0000] Using [email protected] as username.
DEBU[0000] Env variable KUBECTL_VSPHERE_PASSWORD is present
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
DEBU[0001] Got response:
<html>
<head><title>500 - Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>Internal Server Error</p>
</body>
</html>

FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value

This error is registered from authproxy pod running on the supervisor cluster:

Verify the logs of the pods:
kubectl get pods -A -owide --show-labels | grep auth
kube-system                                 wcp-authproxy-4209546c7ddd09797916949ef3779dba                    1/1     Running
kube-system                                 wcp-authproxy-42096cc5420db8b630968ce2be973e30                   1/1     Running
kube-system                                 wcp-authproxy-4209c3dc1bd0d9723e07b3df075e2027                    1/1     Running

kubectl logs -n kube-system --selector=component=wcp-authproxy -f
Followed by login from another window


Another method to verify if one of the pods does not work is by manually running a test curl command from each of the Supervisor Control planes:

curl --request POST https://localhost/wcp/login -u '[email protected]:<PASSWORD>' -k

This will result in similar output:

{"session_id": "eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"}

Same session ID should also match the one you are getting from healthy node 

Then to test it use it in the below request 

curl --request GET https://localhost/wcp/workloads  -k  --header "Authorization: Bearer eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"

Responce example:

[{"namespace": "tanzu-support", "master_host": "192.168.xxx.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "svc-velero-vsphere-domain-c8", "master_host": "192.168.xxx.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "velero", "master_host": "192.168.xxx.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}]


Symptoms:

Problem is seen when login to TKGs clusters:

 

kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster>

 

Error

FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value

 

The result is intermittent


Environment

VMware vSphere 7.0 with Tanzu

Cause

Possible cause could be a faulty POD, or clock skew, where the different components are not updated from the NTP.

Resolution

Verify Time and NTP setting as per above, verify each of the pods in case one of them is faulty, try and  restart the pod.

 


Additional Information

Impact/Risks:
It is expected that the login command will eventually complete, however this can be problematic if such login is used in automation or on regular  basis and can be unexpected experience.