Unable to login to a Guest Cluster via kubectl vSphere login after STS Certificate renewal on vCenter

Products

• VMware Tanzu Mission Control - SM

Issue/Introduction

• Attempting to log in to a VKS guest cluster using the kubectl vsphere login command, the login fails with the following error:
  
  kubectl vsphere login --server <Control_plane_IP> --vsphere-username <User> --insecure-skip-tls-verify --tanzu-kubernetes-cluster-name <Cluster_Name> --tanzu-kubernetes-cluster-namespace <Namespace> -v10
...
INFO[0000] Using <User> as username.
DEBU[0000] Env variable KUBECTL_VSPHERE_PASSWORD is present
DEBU[0000] Error while getting list of workloads: invalid or missing credentials
FATA[0000] Failed to get available workloads, response from the server was invalid.
  
  
• The following error entries can be observed in the /var/log/pods/kube-system_wcp-authproxy logs on the Supervisor node:
  
  DEBUG:auth.authentication_sso:Certificate failed verification
WARNING:auth.authentication_sso:Could not find a signing certificate to validate certificates present in the token: <cert>
ERROR:auth.filters: SAML validation failed.
<date><time> stderr F ERROR:auth.filters:[140182351438528] SAML 
<date><time> stderr F ERROR:auth.filters:[140182351438528] SAML validation failed.
<date><time> stderr F Traceback (most recent call last):
<date><time> stderr F   File "/authproxy/auth/filters.py", line 201, in validate
<date><time> stderr F     saml_token.validate()
<date><time> stderr F   File "/authproxy/auth/authentication_sso.py", line 564, in validate
<date><time> stderr F     signing_chain = self.validate_certificate()
<date><time> stderr F   File "/authproxy/auth/authentication_sso.py", line 756, in validate_certificate
<date><time> stderr F     raise AuthenticationError(
<date><time> stderr F auth.authentication_sso.AuthenticationError: No valid certificates.
<date><time> stderr F ERROR:auth.filters:[140182351438528] Failed to obtain SAML token.
<date><time> stderr F Traceback (most recent call last):
<date><time> stderr F   File "/authproxy/auth/filters.py", line 201, in validate
<date><time> stderr F     saml_token.validate()
<date><time> stderr F   File "/authproxy/auth/authentication_sso.py", line 564, in validate
<date><time> stderr F     signing_chain = self.validate_certificate()
<date><time> stderr F   File "/authproxy/auth/authentication_sso.py", line 756, in validate_certificate
<date><time> stderr F     raise AuthenticationError(
<date><time> stderr F auth.authentication_sso.AuthenticationError: No valid certificates.

Environment

vSphere Kubernetes Service

Cause

This issue occurs when the vCenter Security Token Service (STS) certificate is recently renewed, but the Supervisor clusters are not synchronized with the updated certificates.

Resolution

Restart the Workload Control Plane (WCP) service on the vCenter Server to force a synchronization of the new certificates with the Supervisor clusters:

1. Connect to the vCenter Server via SSH as the root user.
2. Restart the WCP service using the following command:
   
   service-control --restart wcp
   
   
3. Once the WCP service restarts, wait for 5-10 minutes, retry the kubectl vsphere login command to confirm access to the guest cluster.