Error: admission webhook admission.vmware.com denied the request in vSphere Namespaces

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• After upgrading vCenter Server to 8.0 Update 3b or higher, or following a vSphere HA failover event, vSphere Namespaces may become stuck in a Configuring state.
• Namespaces are stuck in a "Configuring" state in the vSphere UI.
• Users receive the following error when creating RoleBindings:

Failed to create RoleBinding for ### in namespace #######. API server returned error 'admission webhook "admission.vmware.com" denied the request: Users are allowed to create role bindings only for service accounts.'. This operation will be retried.

• In vCenter server wcpsvc logs, below errors are noticed.

/var/log/vmware/wcp/wcpsvc.log
[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:###] [opID=svc-velero-domain-####-workload=svc-velero-domain-####] Reconcile role bindings done map[] [{Severity:ERROR Details:0x#######0}]
[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:###] [opID=svc-velero-domain-####-workload=svc-velero-domain-####] Reconcile role bindings done map[] [{Severity:ERROR Details:0x#######0}]
[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:###] [opID=svc-velero-domain-####-workload=svc-velero-domain-####] Reconcile role bindings done map[] [{Severity:ERROR Details:0x#######0}]

• When attempting to list objects in a guest cluster, error below is prompted.

Error from server (Forbidden): pods is forbidden: User "sso:[email protected]" cannot list resource "pods" in API group "" at the cluster scope

• The file  /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist on Supervisor control plane nodes is empty or truncated.
  
  
• This issue prevents rollout of TKG components.

Environment

• VMware Cloud Foundation (VCF)
• vCenter Server 8.0 Update 3b and higher
• vSphere Namespaces / Tanzu Kubernetes Grid (TKG)

Cause

This issue occurs because the   "/etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist"  file on one or more Supervisor control plane nodes is truncated or empty

Resolution

Resolution:

Fixed in release 8.0U3e and higher. See Download Broadcom products, patches and software to download this release.

Workaround: For each Supervisor Control Plane with an empty wcp-schedext-admission-controller-user-whitelist file, perform the following steps:

1. Retrieve the required values by running these commands on the Supervisor node:

   • Get the <machine_id> from the output of: grep MACHINE_ID /var/lib/node.cfg
   • Get the <sso_domain> from the output of: grep SSO_DOMAIN /var/lib/node.cfg


2. Run the following command to update the file content
   
   cat << EOL > /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist
# List of user-prefixes whitelisted by schedext admission controller for
# creating or updating resources modifying secure annotations or tolerating
# master/control plane taint.
kubernetes-admin
kubeadm
system:
sso:wcp-#####@####
vmware-system-
EOL

(Note: Replace wcp-#####@#### with the machine_id and sso_domain retrieved in Step 1.)

3. Restart the wcp-schedext container on the Supervisor Control Plane:

   • Locate the container ID: crictl ps -a | grep schedext
   • Stop the container (it will auto-start): crictl stop <container_id>
   • The container should auto-start.

4. Wait 10-15 minutes for TKG components to reconcile and return to a healthy state.