Failed to get available workloads: invalid character '<' looking for beginning of value
Article ID: 313101
Updated On:
Products
Issue/Introduction
Problem is seen when login to TKGs clusters:
kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster>
Error
FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value
The result is intermittent
Environment
Cause
Possible cause could be a faulty POD, or clock skew, where the different components are not updated from the NTP.
There could be multiple reasons for this problem to occur, to narrow down follow the steps below:
Example of unsuccessful login with -v 6 :
kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6
DEBU[0000] User passed verbosity level: 6
DEBU[0000] Setting verbosity level: 6
DEBU[0000] Setting request timeout:
DEBU[0000] login called as: kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6
DEBU[0000] Creating wcp.Client for <SUPERVISOR VIP>.
INFO[0000] Does not appear to be a vCenter or ESXi address.
DEBU[0000] Got response: Welcome to Photon 3.0 (\m) - Kernel \r (\l)
Welcome to Photon 3.0 (\m) - Kernel \r (\l)
INFO[0000] Using [email protected] as username.
DEBU[0000] Env variable KUBECTL_VSPHERE_PASSWORD is present
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
DEBU[0001] Got response:
<html>
<head><title>500 - Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>Internal Server Error</p>
</body>
</html>
FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value
This error is registered from authproxy pod running on the supervisor cluster:
Verify the logs of the pods:
kubectl get pods -A -owide --show-labels | grep auth
kube-system wcp-authproxy-4209546c7ddd09797916949ef3779dba 1/1 Running
kube-system wcp-authproxy-42096cc5420db8b630968ce2be973e30 1/1 Running
kube-system wcp-authproxy-4209c3dc1bd0d9723e07b3df075e2027 1/1 Running
kubectl logs -n kube-system --selector=component=wcp-authproxy -f
Followed by login from another window
Another method to verify if one of the pods does not work is by manually running a test curl command from each of the Supervisor Control planes:
curl --request POST https://localhost/wcp/login -u '[email protected]:<PASSWORD>' -k
This will result in similar output:
{"session_id": "eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"}
Same session ID should also match the one you are getting from healthy node
Then to test it use it in the below request
curl --request GET https://localhost/wcp/workloads -k --header "Authorization: Bearer eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"
Responce example:
[{"namespace": "tanzu-support", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "svc-velero-vsphere-domain-c8", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "velero", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}]
Resolution
Verify Time and NTP setting as per above, verify each of the pods in case one of them is faulty, try and restart the pod.
Additional Information
Impact/Risks:
It is expected that the login command will eventually complete, however this can be problematic if such login is used in automation or on regular basis and can be unexpected experience.
Feedback
