vCenter Server may fail in the post install hook phase with the error message : "wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s."

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

The vCenter Server(VCSA) patching fails in the post install hook after 90%.
Log file : /var/log/vmware/applmgmt/Patchrunner.log

YYYY-MM-DDTHH:MM:SS.###Z wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1026', 'name': 'PodVmLifeCycleManager', 'description': 'This role entitles the permission required by PodVM LCM controller to be granted in the datacenter.', 'priv_ids': ['Network.Assign', 'Resource.AssignVMToPool', 'System.Read', 'System.Anonymous', 'System.View', 'VirtualMachine.Config.AddExistingDisk', 'VirtualMachine.Config.AddNewDisk', 'VirtualMachine.Config.AddRemoveDevice', 'VirtualMachine.Config.RemoveDisk', 'VirtualMachine.Config.Settings', 'VirtualMachine.Inventory.Create', 'VirtualMachine.Inventory.Delete', 'Datastore.AllocateSpace', 'Datastore.FileManagement', 'StorageProfile.View', 'EAM.Modify', 'Cns.Searchable', 'Resource.ColdMigrate', 'Host.Config.Storage']}
YYYY-MM-DDTHH:MM:SS.###Z wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
YYYY-MM-DDTHH:MM:SS.###Z wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
YYYY-MM-DDTHH:MM:SS.###Z wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
File "/storage/seat/software-update5vpph267/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
executionResult = systemExtension(args)
File "/storage/seat/software-update5vpph267/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
result = self.extension(*args)
File "/storage/seat/software-update5vpph267/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
return func(*args)
File "/storage/seat/software-update5vpph267/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 225, in doPatching
doIncrementalPatching(current_version)
File "/storage/seat/software-update5vpph267/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role PodVmLifeCycleManager (id: 1026) not found in VC..

Cause

PodVmLifeCycleManager role is missing, resulting in error "not found" during privilege check.
If the following command returns no output, it confirms the PodVmLifeCycleManager role is missing from the directory :
ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W > /tmp/roles.ldif && grep "PodVmLifeCycleManager" /tmp/roles.ldif

The upgrade installer uses this property to identify the roles to use them when re-registering the services during the upgrade, and will fail when it cannot find a specific role.

Resolution

Manually create the missing role on VCSA :

Step 1 : Extract the Role from a Working VCSA

Connect via ssh (as root) to a working VCSA.
Export the missing role's data to a new file by specifying the exact Role ID identified in logs (using 1026 as the example):
ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=1026,cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W >/tmp/working_1026.ldif

NOTE : The commands are for the SSO domain vsphere.local, modify domain name in case of custom SSO domain.

Step 2 : Import the role to the affected VCSA

Connect via ssh (as root) to the affected VCSA.
Copy the 1026.ldif file from the working VCSA to affected VCSA.
Edit the file 1026.ldif (using text editor such as vi or nano).

NOTE : Locate the line within the file that begins with nTSecurityDescriptor. Delete that line and replace it with changetype: add

The following is the sample content for the modified 1026.ldif :

dn: cn=1026,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
changetype: add
objectClass: top
objectClass: vmwAuthzRole
cn: 1026
vmwAuthzRolePrivilegeId: System.Anonymous
vmwAuthzRolePrivilegeId: System.Read
vmwAuthzRolePrivilegeId: System.View
vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddRemoveDevice
vmwAuthzRolePrivilegeId: VirtualMachine.Config.RemoveDisk
vmwAuthzRolePrivilegeId: EAM.Modify
vmwAuthzRolePrivilegeId: Host.Config.Storage
vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Delete
vmwAuthzRolePrivilegeId: StorageProfile.View
vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddExistingDisk
vmwAuthzRolePrivilegeId: VirtualMachine.Config.Settings
vmwAuthzRolePrivilegeId: VirtualMachine.Inventory.Create
vmwAuthzRolePrivilegeId: VirtualMachine.Config.AddNewDisk
vmwAuthzRolePrivilegeId: Resource.ColdMigrate
vmwAuthzRolePrivilegeId: Cns.Searchable
vmwAuthzRolePrivilegeId: Datastore.FileManagement
vmwAuthzRolePrivilegeId: Datastore.AllocateSpace
vmwAuthzRolePrivilegeId: Network.Assign
vmwAuthzRolePrivilegeId: InventoryService.Tagging.ObjectAttachable
vmwAuthzRolePrivilegeId: Resource.AssignVMToPool
vmwAuthzRoleVersion: 2
vmwAuthzRoleName: PodVmLifeCycleManager
vmwAuthzRoleDescription: This role entitles the permission required by PodVM LCM controller to be granted in the datacenter.
Import the role using the following command:

/opt/likewise/bin/ldapmodify -f /tmp/1026.ldif -h localhost -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W
Resume the update.