vSphere pods stuck at PodVMAnnotationsMissing or PodVMCreationFailed in vSphere 8.0u3b+

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pods are stuck in PodVMAnnotationsMissing or PodVMCreationFailed state.
  
  
• Verifying the kube-scheduler logs, following errors are seen:
  
  Failed to add podVM's annotations to the pod <pod-name> in namespace <namespace name>.
Error: admission webhook "admission.vmware.com" denied the request: Cannot change VMware system annotation 'vmware-system-vm-uuid'. Will retry
  
  
• Verifying the wcp-schedext logs on particular Supervisor node shows the following errors:
  
  Could not login to vCenter. Error: ServerFaultCode: The object 'vim.VirtualMachine:<vm-####>' has already been deleted or has not been completely created


• The file /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist on that Supervisor node is empty.

Environment

vSphere Kubernetes Service

Cause

 When a fault occurs such as a power off or IO write failure, the /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist could be truncated.

Resolution

Workaround:

1. Add the following content to the /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist on every impacted Supervisor Control Plane node:
   
   cat <<EOL > /etc/vmware/wcp/wcp-schedext-admission-controller-user-whitelist
# List of user-prefixes whitelisted by schedext admission controller for
# creating or updating resources modifying secure annotations or tolerating
# master/control plane taint.
kubernetes-admin
kubeadm
system:
sso:wcp-<machine_id>@<sso_domain>
vmware-system-
EOL
   
   Replace <machine_id> with the machine ID of the vCenter. This should be a UUID.
   
   Run the following command on the Supervisor VM to gather MACHINE_ID:
   grep MACHINE_ID /var/lib/node.cfg

   Replace <sso_domain> with the domain being used by VC's SSO (such as vsphere.local).
   
   To gather this grep the same file.
   grep SSO_DOMAIN /var/lib/node.cfg


2. Restart wcp-schedext pod running on that Supervisor Control Plane node after the file has been updated:
   
   root [ ~ ]# crictl ps -a | grep schedext
<Pod PID>       9 minutes ago        Running      wcp-schedext      10     kube-scheduler-<UUID>

root [ ~ ]# crictl stop <Pod PID> 
<Pod PID> 
   
   
3. Wait for a few minutes and observe the following message appear in wcp-schedext log:
   
   root [ ~ ]# crictl logs <Pod PID> | grep vCenter
Successfully connected to vCenter https://<VC-FQDN>:443/sdk
   
   
4. Redeploy the vSphere pods.

Additional Information

For other admission webhook failures, please refer to below KB articles:

• Mitigate error condition 'admission webhook "admission.vmware.com" denied the request'
• After VC upgrade to 8.0U3 Build 24262322 namespaces got stuck in configuring state