vCLS VM power on fails with the error as Agent VM is deleted on host

Products

• VMware vCenter Server

Issue/Introduction

• vCLS VM power on operation fails on the cluster during the VM reconfigure
• VCLS VMs are deployed under ESX Agents VM folder instead of vCLS folder
• /var/log/vmware/wcp/wcpsvc.log

YYYY:MM:DDTHH:MM:SS info wcp [eamagency/resolve.go:65] [opID=vCLS] Agency (Agency:dc1b0a86-6fd1-####-a6dc-#######) status is yellow on cluster (domain-c2001)
YYYY:MM:DDTHH:MM:SS info wcp [eamagency/resolve.go:98] [opID=vCLS] Checking for agent Agent:437732fe-66c2-####-b220-######
YYYY:MM:DDTHH:MM:SS debug wcp [eamagency/resolve.go:114] [opID=vCLS] Agent status is yellow
YYYY:MM:DDTHH:MM:SS error wcp [eamagent/evcconfig.go:98] [opID=vCLS] Failed to reconfigure VM VirtualMachine:vm-###### to set the CPU Masks for error: ServerFaultCode: Permission to perform this operation was denied.
YYYY:MM:DDTHH:MM:SS error wcp [eamagency/resolve.go:126] [opID=vCLS] setting EVC failed with error ServerFaultCode: Permission to perform this operation was denied.

• journalctl -xe

MM DD HH:MM:SS <vCenter FQDN> vpxd[3107422]: Event [8851348] [1-1] [2024-07-12T09:29:52.641455Z] [vim.event.EventEx] [warning] [] [<Cluster Name>] [8851348] [Privilege check failed for user VSPHERE.LOCAL\vpxd-extension-<machine ID> for missing permission VirtualMachine.Config.AdvancedConfig. Session user performing the check: ]
MM DD HH:MM:SS <vCenter FQDN> vpxd[3107422]: Event [8851350] [1-1] [2024-07-12T09:30:22.687739Z] [vim.event.EventEx] [warning] [] [<Cluster Name>] [8851350] [Privilege check failed for user VSPHERE.LOCAL\vpxd-extension-<machine ID> for missing permission VirtualMachine.Config.AdvancedConfig. Session user performing the check: ]

 

Cause

This issue is seen if vpxd-extension solution user or group containing the same user is added to the Builtin Administrator group of vmdir

Resolution

In order to identify the mismatch execute "authz-doctor" on the vCenter server, refer to Using the "authz-doctor" tool to identify vCenter permission issues

Sample Command:

./usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py all

Running authz_manager command...
No method chosen. Exiting...
Running solution_users command...

Following users are direct or indirect members of Administrators group and should be fixed
vpxd-<machine ID>: ActAsUsers => Administrators
vpxd-svc-acct-<machine ID>: ActAsUsers => Administrators
vpxd-extension-<machine ID>: ActAsUsers => Administrators