Supervisor deployment fails with Authorization Manager vpxd object not found error

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

When attempting to redeploy a VMware Cloud Foundation (VCF) Workload Domain Supervisor after an unsuccessful shutdown and removal, the deployment fails almost immediately and the Supervisor Control Plane VM is not created.

The vSphere UI displays the following error: Authorization Manager (vpxd) failed — object or item referred to could not be found

The /var/log/vmware/vpxd/vpxd.log on the vCenter Server contains errors similar to the following:

error vpxd[...] [Originator@6876 sub=Default opID=...] [VpxLRO] -- ERROR lro-780948 -- ... -- AuthorizationManager -- vim.AuthorizationManager.setEntityPermissions: :vim.fault.NotFound 
--> Result: 
--> (vim.fault.NotFound) { 
--> faultCause = (vmodl.MethodFault) null, 
--> faultMessage = <unset> 
--> msg = "" 
--> } 
--> Args: 
--> 
--> Arg entity: 
--> 'vim.Folder:group-d1' 
--> Arg permission: 
--> (vim.AuthorizationManager.Permission) [ 
--> (vim.AuthorizationManager.Permission) { 
--> entity = <unset>, 
--> principal = "XXXX\\wcp-observabilityop-user-...", 
--> group = false, 
--> roleId = 1107, 
--> propagate = true 
--> } 
--> ]

The /var/log/vmware/wcp/wcpsvc.log contains API errors related to vpxd:

warning wcp [kubelifecycle/kube_instance.go:1726] [opID=...] Failed to check if cpvm resize is supported for cluster domain-c10. Err API call to VMware vCenter Server (vpxd) failed. Details 'object references is empty' 

Querying the vCenter Server database (VCDB) vpx_authz_roles table confirms role_id=1107 is missing:

VCDB=> select * from vpx_authz_roles where role_id=1107;  
 role_id | role_name | role_version | role_description  
---------+-----------+--------------+------------------  
(0 rows)

Environment

VMware Cloud Foundation 9.0.0
vCenter Server 9.0.0

Cause

The Workload Control Plane roles (specifically the ObservabilityOperator and ZoneOperator roles) are inadvertently deleted from the vCenter Server database. This occurs because other product components (e.g., IMS and DR backup admin roles) improperly share or override these WCP roles. When automated product deregistration workflows are executed, the cleanup scripts permanently delete the shared roles.

Resolution

This is a known issue that is resolved in vCenter Server 9.1.

To work around this issue and allow the WCP service to recreate the missing roles on vCenter Server 9.0.0:

1. Connect to the affected vCenter Server Appliance via SSH and log in as root.

2. Delete the WCP configured marker file by running the following command: rm /etc/vmware/wcp/.configured_marker

3. Restart the WCP service to trigger the initial configuration routines: vmon-cli --restart wcp

4. Return to the vSphere UI and retry the Supervisor redeployment.