TKGS Cluster stuck updating with: "cannot create VirtualMachine with its resource policy in DELETING state"

Products

• VMware vCenter Server
• VMware vSphere ESXi
• VMware vSphere Kubernetes Service

Issue/Introduction

To allow the workload cluster to continue updating.

When encountering this issue the following symptoms should be expected:

• The workload cluster in question will be stuck in an "updating" phase.
• A virtualmachine object will be spawned on the Supervisor cluster context but no VM will be created on vSphere
• The virtualmachinesetresourcepolicy object that corresponds to the workload cluster is stuck in a "deleting" phase
• The wcpcluster object which corresponds to the workload cluster is still in a "running" phase
• The namespace in which the workload cluster exists is in an "active" state
• The following log messages are found within the manager container of the vmware-system-vmop-controller-manager pod on the supervisor context:

 

E1010 15:43:06.861908       1 virtualmachine_controller.go:290] VirtualMachine "msg"="Failed to reconcile VirtualMachine" "error"="cannot create VirtualMachine when its resource policy is being deleted" "name"="<ns>/<cluster-name>"
E1010 15:43:06.862649       1 controller.go:329]  "msg"="Reconciler error" "error"="cannot create VirtualMachine when its resource policy is being deleted" "VirtualMachine"={"name":"<cluster-name>","namespace":"<ns>"} "controller"="virtualmachine" "controllerGroup"="vmoperator.vmware.com" "controllerKind"="VirtualMachine" "name"="<cluster-name>" "namespace"="<ns>" "reconcileID"="########-####-####-####-###########"

Environment

VMware vSphere 8.0 with Tanzu
VMware vSphere 7.0 with Tanzu

Cause

The virtualmachinesetresourcepolicy was deleted at some point in the past. This resource policy is used to align vSphere with Tanzu Guest Cluster objects with the vSphere Resource Pool and Cluster inventory objects. The virtualmachinesetresourcepolicy defines the ResourcePool name and Guest Cluster Folder name from vSphere in order for the VMOP pod to place virtualmachine objects in the correct location during creation.

Resolution

The recommended resolution is to remove the workload cluster and redeploy it. 

If you cannot remove the workload cluster and redeploy, contact GS support.

Additional Information

The workload cluster will continue to exist in this state and will not update. However, applications on the affected workload cluster should continue to run.