Using CLI vSphere login failed with "Error while getting list of workloads: invalid or missing credentials"
search cancel

Using CLI vSphere login failed with "Error while getting list of workloads: invalid or missing credentials"

book

Article ID: 425647

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service

Issue/Introduction

1. Authenticating the user account to Supervisor with 'kubectl vsphere login' failed:

kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>   --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify




Username: <username>

Password:

FATA Error while getting list of workloads: invalid or missing credentials

 

2. From the Supervisor cluster logs, you will observe below errors

storage/cluster-info/kube-system/kubectl-plugin-vsphere-xxxxxxxxxxxxxxx/logs.txt


YYYY-MM-DDTHH:MM:SS [error] 6#0: *598036 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: <client-IP>, server: default, request: "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0", upstream: "https://IP-address:12001/wcp/pinniped/.well-known/openid-configuration", h
ost: "IP-address"
<client-IP> - - [YYYY-MM-DDTHH:MM:SS] "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0" 502 157 "-" "pinniped-concierge/v0.0.0 (linux/amd64) kubernetes/$Format"
<external-IP> - - [YYYY-MM-DDTHH:MM:SS] "GET /apis/apps/v1/namespaces/dex/deployments/dex HTTP/2.0" 200 4981 "-" "kubectl/v1.29.7+vmware.wcp.1 (linux/amd64) kubernetes/4946083"
<external IP> - - [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/loginbanner HTTP/2.0" 200 1 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604"
<external IP> - <username> [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/workloads HTTP/2.0" 401 167 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604"
<external IP> - - [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/loginbanner HTTP/2.0" 200 1 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604"
<external IP> - <username> [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/workloads HTTP/2.0" 401 167 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604"
YYYY-MM-DDTHH:MM:SS [error] 6#0: *598049 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: <client-IP>, server: default, request: "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0", upstream: "https://IP-address:12001/wcp/pinniped/.well-known/openid-configuration", host: "<host-ip-address>"
<client-IP> - - [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0" 502 157 "-" "pinniped-concierge/v0.0.0 (linux/amd64) kubernetes/$Format"
YYYY-MM-DDTHH:MM:SS [error] 6#0: *598051 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: <client-IP>, server: default, request: "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0", upstream: "https://IP-address:12001/wcp/pinniped/.well-known/openid-configuration", host: "<host-ip-address>"
<client-IP> - - [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0" 502 157 "-" "pinniped-concierge/v0.0.0 (linux/amd64) kubernetes/$Format"




/var/log/pods/kube-system_wcp-authproxy-xxxxxxxxxxxxxxxxxxxxxxxxx/wcp-authproxy/1.log



YYYY-MM-DDTHH:MM:SS.37248112Z stderr F DEBUG:auth.cache:self-clean of cache begins.
YYYY-MM-DDTHH:MM:SS.448937733Z stderr F DEBUG:telemetry.telemetry_object:AuthProxy Telemetry Object data reported, resetting.
YYYY-MM-DDTHH:MM:SS.449182325Z stderr F DEBUG:telemetry.telemetry_object:AuthProxy Telemetry Object data reset.
YYYY-MM-DDTHH:MM:SS.44931159Z stderr F DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): <VCSA-FQDN>:443
YYYY-MM-DDTHH:MM:SS.464829905Z stderr F DEBUG:urllib3.connectionpool:https://<VCSA-FQDN>:443 "POST /analytics/telemetry/ph/api/hyper/send?_c=SVC.1_0_U1&_i=xxxxxxxxxxxxxxxxxxx HTTP/1.1" 400 0
YYYY-MM-DDTHH:MM:SS.464843712Z stderr F DEBUG:telemetry.telemetry:Pushing Telemetry data to VAC result: 400
YYYY-MM-DDTHH:MM:SS.46484818Z stderr F WARNING:telemetry.telemetry:Pushing Telemetry failed (400)! Error: 400 Client Error: Bad Request for url: https://<VCSA-FQDN>/analytics/telemetry/ph/api/hyper/send?_c=SVC.1_0_U1&_i=xxxxxxxxxxxxxxxxxxxxxx
YYYY-MM-DDTHH:MM:SS.06511941Z stderr F DEBUG:server:[140246558733840] Request: b'GET' b'/wcp/loginbanner' 127.0.0.1
YYYY-MM-DDTHH:MM:SS.065404806Z stderr F INFO:server:[140246558733840] "127.0.0.1" - - "GET /wcp/loginbanner HTTP/1.0" 200 1 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604" ""
YYYY-MM-DDTHH:MM:SS.324666893Z stderr F DEBUG:server:[140246558733840] Request: b'GET' b'/wcp/workloads' 127.0.0.1
YYYY-MM-DDTHH:MM:SS.810429889Z stderr F INFO:vclib.sso:[140246558733840] Invalid credentials for <username>.
YYYY-MM-DDTHH:MM:SS.810444756Z stderr F ERROR:vclib.sso:[140246558733840] Failed to obtain SAML token.
YYYY-MM-DDTHH:MM:SS.810449135Z stderr F Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.810452561Z stderr F   File "/usr/lib/python3.10/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
YYYY-MM-DDTHH:MM:SS.810455877Z stderr F     current.result = callback(  # type: ignore[misc]
YYYY-MM-DDTHH:MM:SS.810459063Z stderr F   File "/authproxy/vclib/sso.py", line 128, in get_bearer_saml_assertion_soap_eb
YYYY-MM-DDTHH:MM:SS.810462199Z stderr F     raise InvalidCredentials(username)
YYYY-MM-DDTHH:MM:SS.810465225Z stderr F auth.errors.InvalidCredentials: InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SS.810468401Z stderr F ERROR:auth.filters:[140246558733840] Failed to obtain SAML token.
YYYY-MM-DDTHH:MM:SS.810471507Z stderr F Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SS.810474603Z stderr F   File "/usr/lib/python3.10/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
YYYY-MM-DDTHH:MM:SS.810477698Z stderr F     current.result = callback(  # type: ignore[misc]
YYYY-MM-DDTHH:MM:SS.810480744Z stderr F   File "/authproxy/vclib/sso.py", line 128, in get_bearer_saml_assertion_soap_eb
YYYY-MM-DDTHH:MM:SS.81048378Z stderr F     raise InvalidCredentials(username)
YYYY-MM-DDTHH:MM:SS.810486956Z stderr F auth.errors.InvalidCredentials: InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SS.810490062Z stderr F WARNING:auth.resources:[140246558733840] Authentication failed with InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SS.810493167Z stderr F DEBUG:telemetry.telemetry_object:Adding 1 failed auth request caused by InvalidCredentials.




 

 

Environment

vSphere with Tanzu

Cause

The vSphere CLI login failure is caused by the expiration of SSL/TLS certificates used by the Pinniped Concierge service on the Supervisor Cluster.

In environments configured with Dex and Pinniped for OIDC (OpenID Connect) authentication, the login workflow relies on valid certificates to establish a secure trust chain between the client, the Identity Provider (Dex), and the Kubernetes API server.

Resolution

1. Verify Port Communication and Network Connectivity


If the login process is failing or timing out, use verbose logging to trace the HTTP requests and identify potential port blockages or network drops.

 

Run the kubectl vsphere login command with the -v (verbosity) flag. 

kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>   --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify -v6

kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>   --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify -v10

 

2. Verify Certificate Validity (OIDC Authentication)


For environments utilizing Dex and Pinniped for OIDC user authentication, login failures may stem from expired certificates on the Supervisor Cluster.


3. SSH into the Supervisor Cluster 

Run the following openssl command to retrieve the certificate chain and expiry dates for the target endpoint.

Bash

openssl s_client -connect <Target-IP>:443

 

Check for the Pinniped documentation: https://pinniped.dev/docs/background/architecture/

 

Additional Information

Supervisor supports the external IDP: Configure an External IDP for Use with TKG Service Clusters

External IDP Configuration
Okta Example OIDC Configuration Using Okta
See also Configure Okta as an OIDC provider for Pinniped
Workspace ONE Configure Workspace ONE Access as an OIDC provider for Pinniped
Dex Configure Dex as an OIDC provider for Pinniped
GitLab Configure GitLab as an OIDC provider for Pinniped
Google OAuth Using Google OAuth 2

 

For the pinniped configuration: Configure the Pinniped Supervisor as an OIDC issuer

Powered by Wolken Software