Symptoms
Permission issues with SSO users in Azure AD groups after upgrading to vSphere 8.0 U3
kubectl shows "namespaces is forbidden: User"
Example:
kubectl get ns --v 9
I0723 12:35:03.332092 2419655 loader.go:395] Config loaded from file: /home/*****/.kube/config
I0723 12:35:03.340228 2419655 round_trippers.go:466] curl -v -XGET -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.28.3+vmware.wcp.1 (linux/amd64) kubernetes/9c6580b" -H "Authorization: Bearer <masked>" 'https://************************/api/v1/namespaces?limit=500'
I0723 12:35:03.342685 2419655 round_trippers.go:510] HTTP Trace: Dial to tcp:172.19.12.29:6443 succeed
I0723 12:35:03.347350 2419655 round_trippers.go:553] GET https://*********************/api/v1/namespaces?limit=500 403 Forbidden in 7 milliseconds
I0723 12:35:03.347378 2419655 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 2 ms TLSHandshake 2 ms ServerProcessing 1 ms Duration 7 ms
I0723 12:35:03.347389 2419655 round_trippers.go:577] Response Headers:
I0723 12:35:03.347402 2419655 round_trippers.go:580] Content-Type: application/json
I0723 12:35:03.347413 2419655 round_trippers.go:580] X-Content-Type-Options: nosniff
I0723 12:35:03.347422 2419655 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: ***********************************************
I0723 12:35:03.347433 2419655 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: ***********************************************
I0723 12:35:03.347443 2419655 round_trippers.go:580] Content-Length: 279
I0723 12:35:03.347455 2419655 round_trippers.go:580] Date: Tue, 23 Jul 2024 12:35:03 GMT
I0723 12:35:03.347465 2419655 round_trippers.go:580] Audit-Id: ***********************************************
I0723 12:35:03.347475 2419655 round_trippers.go:580] Cache-Control: no-cache, private
I0723 12:35:03.347511 2419655 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"namespaces is forbidden: User \"sso:********\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope","reason":"Forbidden","details":{"kind":"namespaces"},"code":403}
I0723 12:35:03.347745 2419655 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "namespaces is forbidden: User \"sso:user@domain\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "namespaces"
},
"code": 403
}]
wcpsvc.log shows that group name in the token has the domain name added twice
/var/log/vmare/wcp/wcpsvc.log
2024-07-23T12:33:14.161Z debug wcp [namespace/authz.go:66] [opID=wcp-AuthzFilter] Checking privileges for username: <domain\user>, groupnames: [******@domain@domain *******], original resources: [{Type:PermissionFolder ID:global-permission}], privs: [System.View]
vSphere 8.0 U3
Group name in token issued by STS has the domain name added twice
Workaround:
add permission to all namespaces via dcli
SSH to VCenter
dcli +i
namespaces access create --namespace <name of namespace> --domain <domain name> --subject "<group name>" --type GROUP --role OWNER
Permanent solution:
Engineering identified a permanent fix that will be provided in a future release.