LDAP/OIDC-authentication-based kubectl commands fail after restarting all nodes in a Tanzu Kubernetes Grid 1.3 installation

Products

• VMware Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x

Issue/Introduction

Symptoms:

• You have restarted all of the nodes (control plane and worker) in a Tanzu Kubernetes Grid 1.3 installation.
• You see the following message when you attempt to issue kubectl commands while using a kubeconfig file that relies on LDAP/OIDC authentication:

Error: could not complete concierge credential exchange: login failed: authentication failed
Error: pinniped-auth login failed: exit status 1
Error: exit status 1
Unable to connect to the server: getting credentials: exec: executable tanzu failed with exit code 1

• The pinniped-concierge-kube-cert-agent pod in the pinniped-concierge namespace has a Status of Unknown.

Environment

VMware Tanzu Kubernetes Grid Plus 1.x
VMware Tanzu Kubernetes Grid 1.x

Resolution

This is a known issue affecting Tanzu Kubernetes Grid 1.3. There is currently no resolution. 

Workaround:
To workaround this issue, delete the pinniped-concierge-kube-cert-agent pod so that it will automatically be recreated.

kubectl -n pinniped-concierge delete pod --selector=kube-cert-agent.pinniped.dev=true