After upgrading to vSphere 8.0 U3, SSO users in Azure AD groups unable use kubectl get error "namespaces is forbidden: User"

Products

• VMware vSphere with Tanzu

Issue/Introduction

Symptoms

Permission issues with SSO users in Azure AD groups  after upgrading to vSphere 8.0 U3

kubectl shows "namespaces is forbidden: User"

Example:

kubectl get ns --v 9

I0723 12:35:03.332092 2419655 loader.go:395] Config loaded from file:  /home/*****/.kube/config
I0723 12:35:03.340228 2419655 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.28.3+vmware.wcp.1 (linux/amd64) kubernetes/9c6580b" -H "Authorization: Bearer <masked>" 'https://************************/api/v1/namespaces?limit=500'
I0723 12:35:03.342685 2419655 round_trippers.go:510] HTTP Trace: Dial to tcp:172.19.12.29:6443 succeed
I0723 12:35:03.347350 2419655 round_trippers.go:553] GET https://*********************/api/v1/namespaces?limit=500 403 Forbidden in 7 milliseconds
I0723 12:35:03.347378 2419655 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 2 ms TLSHandshake 2 ms ServerProcessing 1 ms Duration 7 ms
I0723 12:35:03.347389 2419655 round_trippers.go:577] Response Headers:
I0723 12:35:03.347402 2419655 round_trippers.go:580]     Content-Type: application/json
I0723 12:35:03.347413 2419655 round_trippers.go:580]     X-Content-Type-Options: nosniff
I0723 12:35:03.347422 2419655 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: ***********************************************
I0723 12:35:03.347433 2419655 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: ***********************************************
I0723 12:35:03.347443 2419655 round_trippers.go:580]     Content-Length: 279
I0723 12:35:03.347455 2419655 round_trippers.go:580]     Date: Tue, 23 Jul 2024 12:35:03 GMT
I0723 12:35:03.347465 2419655 round_trippers.go:580]     Audit-Id: ***********************************************
I0723 12:35:03.347475 2419655 round_trippers.go:580]     Cache-Control: no-cache, private
I0723 12:35:03.347511 2419655 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"sso:********\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}
I0723 12:35:03.347745 2419655 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "namespaces is forbidden: User \"sso:user@domain\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "namespaces"
  },
  "code": 403
}]

 

 

wcpsvc.log shows that group name in the token has the domain name added twice

/var/log/vmare/wcp/wcpsvc.log 

2024-07-23T12:33:14.161Z debug wcp [namespace/authz.go:66] [opID=wcp-AuthzFilter] Checking privileges for username: <domain\user>, groupnames: [******@domain@domain *******], original resources: [{Type:PermissionFolder ID:global-permission}], privs: [System.View]

 

 

 

Environment

vSphere 8.0 U3

Cause

Group name in token issued by STS has the domain name added twice

Resolution

Workaround:

add permission to all namespaces via dcli

 

SSH to VCenter

dcli +i

namespaces access create --namespace <name of namespace> --domain <domain name> --subject "<group name>" --type GROUP --role OWNER

 

Permanent solution:

Engineering identified a permanent fix that will be provided in a future release.