Disabling Workload Management fails with "Failed to find resource pool for workload" after DRS is disabled on the cluster running Supervisor.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Disabling a vSphere Supervisor cluster fails if vSphere DRS was disabled beforehand.. 
• Following error is seen within /var/log/vmware/wcp/wcpsvc.log on vCenter Server: 
  
  YYYY-MM-DDTHH:MM:SS error wcp [workload/vcinvnt.go:151] [opID=WCP-#######] Failed to find resource pool for workload ServerFaultCode: The object 'vim.ResourcePool:resgroup-####' has already been deleted or has not been completely created. Err %!v(MISSING)
YYYY-MM-DDTHH:MM:SS error wcp [workload/workload_impl.go:1498] Workload <cluster_name> is already being removed 

• Additionally, the following ManagedObjectNotFound stack trace will be logged in /var/log/vmware/vpxd/vpxd.log: 

YYYY-MM-DDTHH:MM:SS info vpxd[06272] [Originator@#### sub=Default opID=wcp-######] [VpxLRO] -- ERROR lro-577005 -- AuthorizationManager -- vim.AuthorizationManager.setEntityPermissions: vmodl.fault.ManagedObjectNotFound:
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> obj = 'vim.ResourcePool:resgroup-####'
--> msg = ""
--> }
--> Args:
-->
--> Arg entity:
--> 'vim.ResourcePool:resgroup-####' 

• NOTE: This specific error indicates that the backend vSphere objects (the resource pools) were deleted out of order. Please verify that the wcpsvc.log matches the exact ManagedObjectNotFound error shown above before contacting Broadcom Support to clear the stale database entries.

Environment

VMware vSphere Kubernetes Service

Cause

This issue occurs because vSphere Supervisor Namespaces are backed by DRS-managed Resource Pools. Disabling DRS automatically deletes these resource pools from the inventory. When Workload Management subsequently attempts to disable the cluster, it fails because it cannot locate the associated resource pool objects to cleanly remove them.

Resolution

Contact Broadcom Support for assistance in resolving this issue as the workaround involves steps to remove the object from database.