vCLS VMs deployment fails. vCenter reports error "vSphere DRS functionality was impacted due to unhealthy state Cluster Services caused by the unavailability of vSphere Cluster Service VMs."

Products

• VMware vCenter Server

Issue/Introduction

vCLS deployment fails, and vCenter reports, "vSphere DRS functionality was impacted due to unhealthy state Cluster Services caused by the unavailability of vSphere Cluster Service VMs."

/var/log/vmware/wcp/wcpsvc.log contains the following entries:

YYYY-MM--DD error wcp [eamlib/lister.go:84] [opID=EAMAgent] Failed to get EAM agencies. Err ServerFaultCode: EAM is still loading from database. Please try again later. 
YYYY-MM--DD error wcp [informer/informer.go:129] [opID=EAMAgent] Failed to list EAMAgent. Err ServerFaultCode: EAM is still loading from database. Please try again later. 
 
YYYY-MM--DD info wcp [eamagency/permissions.go:184] [opID=vCLS] vCLSAdmin role is already present with roleID 540460992. Updating role to ensure desired privileges are present 
YYYY-MM--DD warning wcp [eamagency/permissions.go:189] [opID=vCLS] Failed to update the role, newer privileges might not be present: ServerFaultCode: Permission to perform this operation was denied. 

Environment

VMware vCenter Server 

Cause

EAM service requires its users to have "EAM.View" and "EAM.Modify" on VPXD's root folder to allow modifications. Without these permissions, vCLS won't be able to instruct EAM to create or destroy vCLS VMs.
Run the authz-doctor to identify the permission assigned to the VPXD EXTENSION user - Using the "authz-doctor" tool to identify vCenter permission

Output from the authz-doctor - 

Incorrect Output :

VSPHERE.LOCAL\vpxd-extension-#######    | False | -1391227607 | Admin | True | Global |

Correct Output :

VSPHERE.LOCAL\vpxd-extension-#######    | False | -1 | Admin | True | Global |

Resolution

VPXD EXTENSION user must have global permission with the Administrator role. 

1. Take a non-memory snapshot of the vCenter VM. If the vCenter is in linked mode, take offline snapshots of all vCenters. 

vSphere UI Method: Change the permission and role from vSphere UI- 

• Log in to the vCenter by using the vSphere Client.
• Select Administration and click Global Permissions in the Access Control area.
• Under the domain, check for user "vpxd-extension-#####" 
• Click Edit > Role 
• The Role should be Administrator 
• The checkbox should be marked Propagate to children 

 

If the role assigned to the user is not getting fixed from the vSphere UI, change the role from the vCenter MOB. 

vCenter MOB Method: Change the permission and role from vCenter MOB - 

• Connect to the MOB by using the fully-qualified domain name (or the IP address) of the vCenter Server system to view the current role  -
  https://vc_fqdn/invsvc/mob3/?moid=authorizationService&method=AuthorizationService.GetGlobalAccessControlList
  
  
  
  
• To change the role, connect to the MOB page on the link below - 
  https://vc_fqdn/invsvc/mob3/?moid=authorizationService&method=AuthorizationService.AddGlobalAccessControlList
  
  
• Use the data below and hit Invoke - 
  

  <permissions>
     <principal>
        <name>vpxd-extension-xxxx</name>
        <group>false</group>
     </principal>
     <roles>-1</roles>
     <propagate>true</propagate>
     <version>-1</version>
  </permissions>

   

• Now, the Role should reflect Administrator (-1)
  
  https://vc_fqdn/invsvc/mob3/?moid=authorizationService&method=AuthorizationService.GetGlobalAccessControlList