Kubectl vSphere Login Fails with 'Invalid or Missing Credentials' Error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Authenticating the user account to Supervisor with 'kubectl vsphere login' fails:
  
  kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>  --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify
Username: <username>
Password:
FATA Error while getting list of workloads: invalid or missing credentials
  
  
• From the Supervisor cluster logs, following errors are seen:
  
  /storage/cluster-info/kube-system/kubectl-plugin-vsphere-####/logs.txt

YYYY-MM-DDTHH:MM:SS [error] 6#0: *598049 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: <client-IP>, server: default, request: "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0", upstream: "https://<IP>:12001/wcp/pinniped/.well-known/openid-configuration", host: "<host-ip-address>"
<client-IP> - - [YYYY-MM-DDTHH:MM:SS +0000] "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0" 502 157 "-" "pinniped-concierge/v0.0.0 (linux/amd64) kubernetes/$Format"
YYYY-MM-DDTHH:MM:SS [error] 6#0: *598051 upstream SSL certificate verify error: (10:certificate has expired) while SSL handshaking to upstream, client: <client-IP>, server: default, request: "GET /wcp/pinniped/.well-known/openid-configuration HTTP/2.0", upstream: "https://<IP>:12001/wcp/pinniped/.well-known/openid-configuration", host: "<host-ip-address>"


/var/log/pods/kube-system_wcp-authproxy-####/wcp-authproxy/1.log

YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:auth.cache:self-clean of cache begins.
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:telemetry.telemetry_object:AuthProxy Telemetry Object data reported, resetting.
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:telemetry.telemetry_object:AuthProxy Telemetry Object data reset.
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): <VCSA-FQDN>:443
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:urllib3.connectionpool:https://<VCSA-FQDN>:443 "POST /analytics/telemetry/ph/api/hyper/send?_c=SVC.1_0_U1&_i=#### HTTP/1.1" 400 0
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:telemetry.telemetry:Pushing Telemetry data to VAC result: 400
YYYY-MM-DDTHH:MM:SSZ stderr F WARNING:telemetry.telemetry:Pushing Telemetry failed (400)! Error: 400 Client Error: Bad Request for url: https://<VCSA-FQDN>/analytics/telemetry/ph/api/hyper/send?_c=SVC.1_0_U1&_i=####
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:server:[####] Request: b'GET' b'/wcp/loginbanner' 127.0.0.1
YYYY-MM-DDTHH:MM:SSZ stderr F INFO:server:[####] "127.0.0.1" - - "GET /wcp/loginbanner HTTP/1.0" 200 1 "-" "kube-plugin-vsphere bld 24795027 - cln 15520604" ""
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:server:[####] Request: b'GET' b'/wcp/workloads' 127.0.0.1
YYYY-MM-DDTHH:MM:SSZ stderr F INFO:vclib.sso:[####] Invalid credentials for <username>.
YYYY-MM-DDTHH:MM:SSZ stderr F ERROR:vclib.sso:[####] Failed to obtain SAML token.
YYYY-MM-DDTHH:MM:SSZ stderr F Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSZ stderr F   File "/usr/lib/python3.10/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
YYYY-MM-DDTHH:MM:SSZ stderr F     current.result = callback(  # type: ignore[misc]
YYYY-MM-DDTHH:MM:SSZ stderr F   File "/authproxy/vclib/sso.py", line 128, in get_bearer_saml_assertion_soap_eb
YYYY-MM-DDTHH:MM:SSZ stderr F     raise InvalidCredentials(username)
YYYY-MM-DDTHH:MM:SSZ stderr F auth.errors.InvalidCredentials: InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SSZ stderr F ERROR:auth.filters:[####] Failed to obtain SAML token.
YYYY-MM-DDTHH:MM:SSZ stderr F Traceback (most recent call last):
YYYY-MM-DDTHH:MM:SSZ stderr F   File "/usr/lib/python3.10/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
YYYY-MM-DDTHH:MM:SSZ stderr F     current.result = callback(  # type: ignore[misc]
YYYY-MM-DDTHH:MM:SSZ stderr F   File "/authproxy/vclib/sso.py", line 128, in get_bearer_saml_assertion_soap_eb
YYYY-MM-DDTHH:MM:SSZ stderr F     raise InvalidCredentials(username)
YYYY-MM-DDTHH:MM:SSZ stderr F auth.errors.InvalidCredentials: InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SSZ stderr F WARNING:auth.resources:[140246558733840] Authentication failed with InvalidCredentials: <username>
YYYY-MM-DDTHH:MM:SSZ stderr F DEBUG:telemetry.telemetry_object:Adding 1 failed auth request caused by InvalidCredentials.

Environment

vSphere Kubernetes Service

Cause

• The vSphere CLI login failure is caused by the expiration of SSL/TLS certificates used by the Pinniped Concierge service on the Supervisor Cluster.
• In environments configured with Dex and Pinniped for OIDC (OpenID Connect) authentication, the login workflow relies on valid certificates to establish a secure trust chain between the client, the Identity Provider (Dex), and the Kubernetes API server.

Resolution

1. Verify Port Communication and Network Connectivity
   • If the login process is failing or timing out, use verbose logging to trace the HTTP requests and identify potential port blockages or network drops.
     Run the kubectl vsphere login command with the -v (verbosity) flag. 
     
     kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>   --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify -v6

kubectl vsphere login   --server=https://IP   --tanzu-kubernetes-cluster-namespace <cluster-namespace>   --tanzu-kubernetes-cluster-name <cluster-name>   --insecure-skip-tls-verify -v10
     
     
2. Verify Certificate Validity (OIDC Authentication)
   • For environments utilizing Dex and Pinniped for OIDC user authentication, login failures may stem from expired certificates on the Supervisor Cluster.
     
     
3. SSH into the Supervisor Cluster
   • Run the following openssl command to retrieve the certificate chain and expiry dates for the target endpoint.
     openssl s_client -connect <Target-IP>:443
     
     
4. Check the Pinniped documentation.

Additional Information

Supervisor supports the external IDP: Configure an External IDP for Use with TKG Service Clusters

External IDP	Configuration
Okta	Example OIDC Configuration Using Okta See also Configure Okta as an OIDC provider for Pinniped
Workspace ONE	Configure Workspace ONE Access as an OIDC provider for Pinniped
Dex	Configure Dex as an OIDC provider for Pinniped
GitLab	Configure GitLab as an OIDC provider for Pinniped
Google OAuth	Using Google OAuth 2

 

For the pinniped configuration: Configure the Pinniped Supervisor as an OIDC issuer