Upgrade of vCenter fails with error "WCP service installation failed"

Products

• VMware vCenter Server

Issue/Introduction

• vCenter server upgrade fails with error "WCP service installation failed":
• vCenter installer UI would show similar error.

  
WCP service installation failed : Traceback (most recent call last): File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py",
line 67, in proxy return func(*args, **kwargs) File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 87, in configure wcpconfigure.configure_service()
File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 622, in configure_service register_extension_with_vc() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py",
line 557, in register_extension_with_vc vc_ext.register_extension() File "/usr/lib/vmware-wcp/py-modules/vc_extension_utils.py", line 124,
in register_extension self.em.UpdateExtension(extension=ext_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 618, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 391, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File
"/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1607, in InvokeMethod raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vim.fault.NoPermission: (vim.fault.NoPermission) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'Permission to perform this operation was denied.', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [], object = 'vim.Folder:group-d1',
privilegeId = 'Extension.Update', missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) [ (vim.fault.NoPermission.EntityPrivileges)
{ dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], entity = 'vim.ExtensionManager:ExtensionManager', privilegeIds = (str) [ 'Extension.Update' ] } ] }

• You will see the log entries similar to the below snippet in /var/log/firstboot/wcp-firstboot.py_XXXXX_stdout.log
  
  

YYYY-MM-DDTHH:MM:SS ERROR wcp-firstboot Failed to register extension 'com.vmware.vcenter.wcp' during 'updating extension'. Exception: (vim.fault.NoPermission) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Permission to perform this operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   object = 'vim.Folder:group-d1',
   privilegeId = 'Extension.Update',
   missingPrivileges = (vim.fault.NoPermission.EntityPrivileges) [
      (vim.fault.NoPermission.EntityPrivileges) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         entity = 'vim.ExtensionManager:ExtensionManager',
         privilegeIds = (str) [
            'Extension.Update'

• To validate the Duplicate "Admin" role using Authz-doctor. Refer: Using the "authz-doctor" tool to identify vCenter permission issues.
  
  
  • /usr/lib/vmware-vpx/scripts/authz-doctor/authz-doctor.py permission_check


    • Output will show following finding.
    • 

       OR

• Manually Validate duplicate "Admin" roles using vcdb.
  
  
  • SSH to vCenter as Root user and run following cmd.
    /opt/vmware/vpostgres/current/bin/psql -U postgres VCDB


  • Once connected to vcdb run following query, 
select * from vpx_authz_roles;
role_id   |    role_name                   | role_version | role_description
-1        |     Admin                      |            0 | Admin
-5        |    NoAccess                    |            0 | NoAccess
2081475472|     admin                      |            0 | 

  • From output of above query validate if you have multiple "Admin" or "Administrator" roles
    • Default role would have role id "-1" 

Environment

vCenter Server 7.x
vCenter Server 8.x

Resolution

Ensure that the vCenter Server does not contain duplicate roles named "Administrator" or "Admin" before the upgrade .

Note: Ensure there is valid backup/offline snapshot of the VCSA prior to implementing the workaround. Refer to VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

• • Log in to the vCenter Server by using the vSphere Client.
  • Navigate to Administration > Access Control > Roles tab
    • Should not have multiple role named "Administrator" or "Admin"
      
      
      
  • Validate and remove any duplicate "Administrator" roles by selecting the duplicate and clicking Delete.
  • Proceed with the vCenter Server upgrade.