Unable to login to management clusters via Pinniped

Products

• VMware Telco Cloud Automation

Issue/Introduction

When installing Pinniped as an addon to TCA deployed clusters, authentication will work in the workload cluster but fails in the management cluster with the following error:

$ kubectl get pod
Error: could not complete concierge credential exchange: could not login: the server could not find the requested resource
Error: pinniped-auth login failed: exit status 1
Unable to connect to the server: getting credentials: exec: executable tanzu failed with exit code 1

In the Pinniped Concierge pod logs on the management cluster, you see entries like below:

{"level":"info","timestamp":"2024-11-13T20:28:58.660232Z","caller":k8s.io/[email protected]/trace/trace.go:236$trace.(*Trace).logTrace,"message":"Trace[1365036503]: \"create\" kind:TokenCredentialRequest (13-Nov-2024 20:28:58.659) (total time: 0ms):\nTrace[1365036503]: ---\"failure\" failureType:token authentication,msg:oidc: verify token: oidc: expected audience \https://<CLUSTER_VIP>:31234\ got [\"<CLUSTER_NAME>\"] 0ms (20:28:58.660)\nTrace[1365036503]: [146.812µs] [146.812µs] END\n"}

Environment

2.x
3.x

Cause

The configuration created for the management cluster uses the cluster's VIP as the 'audience' in the JWTAuthenticator resource, but Pinniped expects the cluster's name.

Resolution

To resolve this issue and enable Pinniped authentication for your management cluster, edit the JWTAuthenticator resource in the management cluster so the 'audience' value reflects the cluster's name, and not the cluster's VIP. See below for an example configuration:

apiVersion: authentication.concierge.pinniped.dev/v1alpha1
kind: JWTAuthenticator
metadata:
  annotations:
    kapp.k14s.io/identity: v1;/authentication.concierge.pinniped.dev/JWTAuthenticator/tkg-jwt-authenticator;authentication.concierge.pinniped.dev/v1alpha1
  creationTimestamp: "2024-07-09T00:08:44Z"
  generation: 4
  labels:
    kapp.k14s.io/app: "#####"
    kapp.k14s.io/association: v1.#####
  name: tkg-jwt-authenticator
  resourceVersion: "#########"
  uid: ########-####-####-####-############
spec:
  audience: <NAME_OF_MGMT_CLUSTER>
  issuer:https://###.###.###.###:31234
  tls:
    certificateAuthorityData: <BASE64_CERT>