TMC-SM login failure - "errcode: 3001 errmsg: Unauthorized requestid"

Products

VMware Tanzu Mission Control

Issue/Introduction

TMC-SM (Tanzu Mission Control Self-Managed) users cannot log in to the TMC-SM UI. The web browser displays the following error message.

errcode: 3001 errmsg: Unauthorized requestid: ########-####-####-####-########

Switch to the k8s context of the workload cluster that is running TMC-SM.

# kubectl -n tmc-local logs deployment/landing-service-server | grep "id token not found"
{"component":"server-serve-http","http.host":"landing.<TMC_DOMAIN>","http.proto_major":2,...,"level":"error","msg":"errcode: 2004 errmsg: Unauthorized requestid: ####-####-####-######### cause: id token not found",...}

# kubectl -n tmc-local logs deployment/auth-manager-server | grep "the value is too long"
level=error msg="Could not save idtokenCookie" X-Request-ID=########-####-####-########## error="securecookie: the value is too long" ...

Environment

All versions earlier than TMC-SM v1.4.3

Cause

During authentication, the browser receives data that exceeds the 4 KB cookie size limit.
As a result, the login-id-token cannot be stored in the cookie, leading to authentication failure.

Resolution

Permanent Solution

Upgrade to TMC-SM v1.4.3 (requires Kubernetes v1.30 or later).

Check the release note.

Temporary Solution

Reduce cookie usage by applying filtering on the Pinniped FederationDomain.

1. Set the value

DISPLAY_NAME="TMC-SM" # Displayed title in the login page
# Select the IdP backend for your environment
KIND="ActiveDirectoryIdentityProvider" # "LDAPIdentityProvider" or "OIDCIdentityProvider" or "ActiveDirectoryIdentityProvider"
KIND_OBJECT_NAME="<DOMAIN>" # NAME of "kubectl -n tmc-local get $KIND"
TMC_GROUP_ADMIN="tmc-admin"   # Value of idpGroupRoles.admin in the values.yaml
TMC_GROUP_MEMBER="tmc-member" # Value of idpGroupRoles.member in the values.yaml

2. Create a Secret for overlay

cat > federationdomain-overlay.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: pinniped-federationdomain-overlay
  namespace: tmc-local
stringData:
  overlay.yml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"apiVersion":"config.supervisor.pinniped.dev/v1alpha1", "kind":"FederationDomain"}),expects="1+"
    ---
    #@overlay/match expects="1+"
    spec:
      #@overlay/match expects="0+"
      identityProviders:
        - displayName: ${DISPLAY_NAME}
          objectRef:
            apiGroup: idp.supervisor.pinniped.dev
            kind: ${KIND}
            name: ${KIND_OBJECT_NAME}
          transforms:
            expressions:
              - expression: 'groups.filter(g, g in ["${TMC_GROUP_ADMIN}", "${TMC_GROUP_MEMBER}"])'
                type: groups/v1
EOF
kubectl apply -f federationdomain-overlay.yaml

3. Create a secret to include the above overlay in the tmc-local-stack package

cat > tmc-package-overlay.yaml <<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: tmc-package-overlay
  namespace: tmc-local
stringData:
  tmc-pkgi-overlay.yml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"apiVersion":"packaging.carvel.dev/v1alpha1", "kind":"PackageInstall", "metadata": {"name": "tmc-local-stack"}}),expects="1+"
    ---
    metadata:
      #@overlay/match missing_ok=True
      annotations:
        #@overlay/match missing_ok=True
        ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: pinniped-federationdomain-overlay
EOF
kubectl apply -f tmc-package-overlay.yaml

4. Patch the tanzu-mission-control meta package to include the overlay for tmc-local-stack package.

kubectl -n tmc-local patch pkgi tanzu-mission-control --type='merge' -p '{"metadata": {"annotations": {"ext.packaging.carvel.dev/ytt-paths-from-secret-name.0": "tmc-package-overlay"}}}'

5. Confirm that the federationdomain custom resource was updated as expected (Added identityProviders: section)

Wait up to 5 minutes for kapp-controller to update the resource during the reconcile process.

kubectl -n tmc-local get federationdomain pinniped -oyaml | yq .spec

6. Restart pinniped-supervisor

kubectl -n tmc-local rollout restart deployment/pinniped-supervisor

7. Confirm that the reconciliation of all packages in tmc-local namespace

kubectl -n tmc-local get pkgi
kubectl -n tmc-local get pods

8. Try logging in to TMC-SM through the browser.

Rollback Operation

If the workaround does not work, or if you no longer need it after upgrading to TMC-SM v1.4.3 or later, purge the configuration.

# 1. Pause tmc-local package
tanzu -n tmc-local package installed pause tmc-local-stack

# 2. Delete the Annotation and Secrets
kubectl -n tmc-local annotate pkgi tanzu-mission-control ext.packaging.carvel.dev/ytt-paths-from-secret-name.0-
kubectl -n tmc-local delete secret tmc-package-overlay
kubectl -n tmc-local delete secret pinniped-federationdomain-overlay

# 3. Unpause to reconcile the package
tanzu -n tmc-local package installed kick tmc-local-stack

# 4. Restart pinniped-supervisor
kubectl -n tmc-local rollout restart deployment/pinniped-supervisor

# 5. Check the setting is rollbacked correctly (identityProviders: section has been removed)
kubectl -n tmc-local get federationdomain pinniped -oyaml | yq .spec

# 6. Confirm that the reconciliation of all packages in tmc-local namespace
kubectl -n tmc-local get pkgi
kubectl -n tmc-local get pods