Namespaces got stuck in configuring state after vCenter Server is upgraded to 8.0 Update 3b and above

Products

• VMware vSphere with Tanzu

Issue/Introduction

• Error message in GUI on namespace:

Failed to create RoleBinding for xxx in namespace xxxxxxx. API server returned error 'admission webhook "admission.vmware.com" denied the request: Users are allowed to create role bindings only for service accounts.'. This operation will be retried.

• In vCenter server wcpsvc logs, below errors are noticed.

/var/log/vmware/wcp/wcpsvc.log

[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:906] [opID=svc-velero-domain-*****-workload=svc-velero-domain-*****] Reconcile role bindings done map[] [{Severity:ERROR Details:0xc025cb90e0}]
[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:906] [opID=svc-velero-domain-*****-workload=svc-velero-domain-*****] Reconcile role bindings done map[] [{Severity:ERROR Details:0xc025e29db0}]
[YYYY-MM-DDTHH:MM:SS] debug wcp [workload/controller.go:906] [opID=svc-velero-domain-*****-workload=svc-velero-domain-*****] Reconcile role bindings done map[] [{Severity:ERROR Details:0xc025edd090}]

 

• On one or more Supervisor control planes the file /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist is empty
  
  
• This issue prevents rollout of TKG components.

Environment

 vCenter Server is 8.0 Update 3b and above

Cause

VMware Engineering is aware of this issue, and it is currently under investigation.

Please subscribe to the KB article to get notified of changes.

Resolution

 

Workaround:

For each Supervisor Control Plane with an empty wcp-schedext-admission-controller-user-whitelist file, perform the following steps:

1. Retrieve the required values:

   • Get the <machine_id> from the output of:
      

     grep MACHINE_ID /var/lib/node.cfg

      
   • Get the <sso_domain> from the output of:
      

     grep SSO_DOMAIN /var/lib/node.cfg

2. Add the following content to the file /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist:

    

   # List of user-prefixes whitelisted by schedext admission controller for
# creating or updating resources modifying secure annotations or tolerating
# master/control plane taint.

   kubernetes-admin
kubeadm
system:
sso:wcp-<machine_id>@<sso_domain>
vmware-system-

3. Restart the wcp-schedext pod on the Supervisor Control Plane:

   • First, locate the pod:
      

     root@******************* [ ~ ]# crictl ps -a | grep schedext
b93dfeb4bf980       ed05c0dd2aa27       9 minutes ago  

      
   • Then, stop the pod:

     root@******************* [ ~ ]# crictl stop b93dfeb4bf980
b93dfeb4bf980

   • The pod should auto-start.

4. Wait 10-15 minutes for TKG components to reconcile and return to a healthy state.