Failed to get available workloads: invalid character '<' looking for beginning of value
Article ID: 313101
Updated On:
Products
Issue/Introduction
Problem is seen when login to TKGs clusters:
$ kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster>
Error
FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value.
When enabling the verbose mode for the same command:
$ kubectl vsphere login --server=<SUPERVISOR VIP> --vsphere-username [email protected] --tanzu-kubernetes-cluster-name mycluster --tanzu-kubernetes-cluster-namespace<Tanzu-NS> --insecure-skip-tls-verify -v 6
...
DEBU[YYYY-MM--DD hh:mm:ss] Got response:
<html>
<head><title>500 - Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>Internal Server Error</p>
</body>
</html>
FATA[YYYY-MM--DD hh:mm:ss] Error while getting list of workloads: invalid character '<' looking for beginning of value
wcp-authproxy pod from the Supervisor shows messages similar to below ones:
DEBUG:auth.authentication_sso:isExpired, notBefore: None, notOnOrAfter: None, now: 2025-06-03 00:38:16.304824, tolerance: 600
DEBUG:auth.authentication_sso:isExpired, notBefore: 2025-06-03T00:38:16.212Z, notOnOrAfter: 2025-06-03T00:43:16.147Z, now: 2025-06-03 00:38:16.305235, tolerance: 600
Environment
Cause
Possible cause could be a faulty POD, or clock skew, where the different components are not updated from the NTP.
There could be multiple reasons for this problem to occur, to narrow down follow the steps below:
Example of unsuccessful login with -v 6 :
$ kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6
DEBU[0000] User passed verbosity level: 6
DEBU[0000] Setting verbosity level: 6
DEBU[0000] Setting request timeout:
DEBU[0000] login called as: kubectl-vsphere login --server <SUPERVISOR VIP> --vsphere-username [email protected] --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <Tanzu-NS> --tanzu-kubernetes-cluster-name <Tanzu-cluster> -v6
DEBU[0000] Creating wcp.Client for <SUPERVISOR VIP>.
INFO[0000] Does not appear to be a vCenter or ESXi address.
DEBU[0000] Got response: Welcome to Photon 3.0 (\m) - Kernel (\l)
Welcome to Photon 3.0 (\m) - Kernel (\l)
INFO[0000] Using [email protected] as username.
DEBU[0000] Env variable KUBECTL_VSPHERE_PASSWORD is present
KUBECTL_VSPHERE_PASSWORD environment variable is not set. Please enter the password below
Password:
DEBU[0001] Got response:
<html>
<head><title>500 - Internal Server Error</title></head>
<body>
<h1>Internal Server Error</h1>
<p>Internal Server Error</p>
</body>
</html>
FATA[0001] Failed to get available workloads: invalid character '<' looking for beginning of value
This error is registered from wcp-authproxy pod running on the Supervisor cluster:
You may verify the logs of the pods as below followed by login from another terminal window.
$ kubectl get pods -A | grep auth
kube-system wcp-authproxy-4209546c7ddd09797916949ef3779dba 1/1 Running
kube-system wcp-authproxy-42096cc5420db8b630968ce2be973e30 1/1 Running
kube-system wcp-authproxy-4209c3dc1bd0d9723e07b3df075e2027 1/1 Running
$ kubectl logs -n kube-system --selector=component=wcp-authproxy -f
Another method to verify if one of the pods does not work is by manually running a test curl command from each of the Supervisor Control planes:
curl --request POST https://localhost/wcp/login -u '[email protected]:<PASSWORD>' -k
This will result in similar output:
{"session_id": "eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"}
Same session ID should also match the one from the healthy node
Then to test it, use it in the below request
curl --request GET https://localhost/wcp/workloads -k --header "Authorization: Bearer eyJra....<Session length reduced >...............QRSMKN5nsEoMwcZl8TkDx_"
Response example:
[{"namespace": "tanzu-support", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "svc-velero-vsphere-domain-c8", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}, {"namespace": "velero", "master_host": "192.168.###.130", "control_plane_api_server_port": 6443, "control_plane_DNS_names": []}]Resolution
- Verify Time and NTP setting as per above and make sure system times are in sync across involved entities (vCenter, Supervisor VMs, Guest Cluster CPVMs, ESXi Servers)
- Verify each of the pods(wcp-authproxy) is not in faulty state, try and restart the pod if necessary.
Additional Information
Impact/Risks:
- It is expected that the login command will eventually complete, however this can be problematic if such login is used in automation or on regular basis and can be unexpected experience.
- In case of big time skew, there could be other system pods(like ncp) in CrashLoopBackOff state and you may not be able to login to any of guest clusters with the same error messages.
Feedback
