"Error from server (Forbidden): pods is forbidden: User "sso:[email protected]" cannot list resource "pods" in API group" when running kubectl commands on VKS Guest Clusters managed by TMC

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Running kubectl commands returns a Forbidden error indicating the user cannot list resources on the VKS Guest Clusters which are managed by TMC. 
  Example: 
  

  Error from server (Forbidden): pods is forbidden: User "sso:[email protected]" cannot list resource "pods" in API group "" in the namespace "default"

• Authentication via Pinniped is successful and healthy as the API server correctly identifies the user.
• Assigning the user the "Owner" role directly within the vCenter vSphere Namespace permissions resolves the issue, confirming the local cluster RBAC engine is functional.
• Tanzu Mission Control (TMC) cluster status and policy sync extensions report as healthy.

Environment

VMware Tanzu Mission Control Self-Managed
VMware vSphere Kubernetes Service

Cause

This issue occurs due to a string mismatch between the identity provided by the Identity Provider (IDP) and the identity defined in the TMC Access Policy.

Kubernetes Role-Based Access Control (RBAC) authorization requires an exact, case-sensitive string match to grant permissions. When users authenticate, the IDP via Pinniped passes a specific identity string to the Kubernetes API server, which often includes a predefined prefix such as sso: (e.g., sso:[email protected]).

If the TMC Access Policy is configured with just the email address (e.g., [email protected]) or uses different capitalization (e.g., [email protected]), Kubernetes will treat the authenticated user and the policy subject as two completely different entities, resulting in a Forbidden authorization denial.

Resolution

To permanently resolve the authorization failure, the TMC Access Policies must be updated to match the exact identity string passed by the IDP. 

1. Identify the Exact String: Instruct an affected user to run a kubectl command and trigger the Forbidden error. Copy the exact user string displayed in the error message (e.g., sso:[email protected]). This string reveals the exact prefix and capitalization required.
   
2. Remove Invalid Policies: Log into the Tanzu Mission Control UI, navigate to the affected cluster or organization, and open the Access Policies tab. Delete the failing user identities from both inherited and direct access policies.
3. Re-add User Identities: Re-add the users or groups, ensuring the identity field perfectly matches the string captured in Step 1. The sso: prefix must be manually typed, and case-sensitivity must be strictly observed.
   
4. Validation: Once TMC syncs the updated policies down to the guest clusters, affected users will be able to run kubectl commands successfully.