Administrative access revoked in TMC Self-Managed due to unintended changes to access policies

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

After making a change to access policies within Tanzu Mission Control, the following is observed:

• Users can log into Tanzu Mission Control Self-Managed (TMC-SM), but find that none of the inventory (guest clusters, management clusters) is viewable.
• The access policies tab is either not present or it returns an error.
• Re-authentication does not resolve the issue.
• Administrative tasks such as registering a new management cluster throw a "Permission denied" message.

Environment

Tanzu Mission Control Self-Managed

Cause

This issue occurs due to a string mismatch between the user principal defined in the TMC Access Policies and the actual identity string provided by the Identity Provider (IDP) during authentication.

In TMC-SM, authorization requires an exact, case-sensitive match. The identity is extracted from the grpc-metadata-x-user-id header, within the username field of the JWT token.

 

Common Mismatch Scenarios:

• Case-Sensitivity: The policy specifies [email protected], but the token contains Test.User.
• Missing Prefixes: The policy is updated to include a standard email address, but the IDP requires an sso: prefix (e.g., sso:[email protected]).
• Domain Suffixes: The IDP provides a short name (e.g., Test.User), but the access policy expects a Fully Qualified Domain Name (FQDN).

When these strings do not match perfectly, the organization.admin role's privileges cannot be granted to the session, effectively locking the user out of administrative functions. Without administrative privileges, access policies cannot be changed and reverted via the TMC SM GUI.

Resolution

To resolve this issue, you must ensure the principle name in the Access Policy matches the exact value of the identity username in the authentication token. If the UI is inaccessible due to loss of administrative control, manual database intervention is required.

 

Step 1: Identify the Exact Principal String

1. Log into TMC-SM and capture a network trace or inspect the session headers.
2. Locate the grpc-metadata-x-user-id header.
3. Note the exact string value, including capitalization and prefixes (e.g., Test.User)

 

Step 2: Manually Update the Database (If UI Locked)

Please contact Broadcom technical support for performing the database updates.

 

Step 3: Verify Access

1. Log out of all active TMC-SM sessions.
2. Log back in with the corrected administrative account.
3. Verify that the cluster inventory and Access Policies tab are now visible.

Additional Information