2022-10-10T13:26.586Z error wcp [workload/vcinvnt.go:151] [opID=CLUSTER_NAME] Failed to find resource pool for workload ServerFaultCode: The object 'vim.ResourcePool:resgroup-1234' has already been deleted or has not been completely created. Err %!v(MISSING)
2022-10-10T13:38:31.581Z error wcp [workload/workload_impl.go:1498] Workload CLUSTER_NAME is already being removed
Users will see the following, or similar, logging in the /var/log/vmware/vpxd/vpxd.log
2022-10-10T13:26.586Z info vpxd[06272] [Originator@6876 sub=Default opID=wcp-6245a1a1-68] [VpxLRO] -- ERROR lro-577005 -- AuthorizationManager -- vim.AuthorizationManager.setEntityPermissions: vmodl.fault.ManagedObjectNotFound:
--> Result:
--> (vmodl.fault.ManagedObjectNotFound) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> obj = 'vim.ResourcePool:resgroup-1234'
--> msg = ""
--> }
--> Args:
-->
--> Arg entity:
--> 'vim.ResourcePool:resgroup-1234'
This failure occurs because the resource pools backing the Namespace objects in vSphere inventory are provided by DRS. When DRS is disabled, the association to the resource pool can not longer be found, leading to a failure to remove the Namespace object.
Caution:
python3 /usr/lib/vmware-wcp/nsx_policy_cleanup.py --cluster domain-c11040:035xxxe8-eb20-41xx-a7xx-972068xxxxxx -u <nsx admin user> -p '<nsx mgr admin pass>' --mgr-ip=<nsx mgr ip> --no-warning --top-tier-router-id=domain-c11040:035xxxe8-eb20-41xx-a7xx-972068xxxxxx --all-res -r
Preparation:
# /usr/lib/vmware-wcp/decryptK8Pwd.py
Example output:
Read key from file
Connected to PSQL
Cluster: domain-c8:4f67k834-5436-7456-b307-467g109j5xxx
IP: <CLUSTER_IP>
PWD: <ROOT_PASSWORD>
Cluster: domain-c20:4f67k834-5436-7456-b307-467g109j5xxx
IP: <CLUSTER_IP>
PWD: <ROOT_PASSWORD>
# vmon-cli -k wcp
Completed Stop service request
# PGPASSFILE=/etc/vmware/wcp/.pgpass psql -U wcpuser -h localhost VCDB
<snip>
VCDB=>
VCDB=> select cluster from cluster_db_configs ;
count
-------
domain-c8:4f67xxxx-5436-7456-b307-467g10xxxxxx
domain-c20:4f67xxxx-5436-7456-b307-467g10xxxxxx
(2 row)
VCDB=> select cluster from workload_configs ;
count
-------
domain-c8:4f67kxxx-5436-7456-b307-467g109j5xxx
domain-c20:4f67kxxx-5436-7456-b307-467g109j5xxx
(2 row)
VCDB=> delete from workload_configs where cluster = 'domain-c8:4f67k834-5436-7456-b307-467g109j5xxx';
DELETE 1
VCDB=> \q
# vmon-cli -i wcp
Completed Start service request.