Patching vCenter Server to 8.0 U3 fails at 80% and 95% with error "Exception occurred in postInstallHook for wcp:Patch"

Products

VMware vCenter Server 8.0

Issue/Introduction

Patching vCenter Server to 8.0 U3 fails during PostInstallHook phase with error "Exception occurred in postInstallHook for wcp:Patch"

In /var/log/vmware/applmgmt/PatchRunner.log shows the error message

<YYYY-MM-DDTHH:MM:SS> wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '####', 'name': 'CNS-Datastore', 'description': 'Allows performing read, write, delete, and rename operations in the datastore browser. ', 'priv_ids': ['System.Read', 'System.Anonymous', 'System.View', 'Datastore.AllocateSpace', 'Datastore.FileManagement']}
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updateu46go3kh/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 329, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role CNS-Datastore (id: ####) not found in VC..
<YYYY-MM-DDTHH:MM:SS> ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
  File "/storage/seat/software-updateu46go###/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
<YYYY-MM-DDTHH:MM:SS> WARNING root stopping status aggregation...
<YYYY-MM-DDTHH:MM:SS> ERROR __main__ Patch vCSA failed

Upgrading vCenter from 7.0 U3 to 8.0 U3 also fails with the error : Failed to start wcp service at 95%.

In /var/log/vmware/applmgmt/PatchRunner.log shows the error message

<YYYY-MM-DDTHH:MM:SS> In(05) host-#### <wcp-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-wcp/wcpsvc-prestart.py
<YYYY-MM-DDTHH:MM:SS> Wa(03) host-#### <wcp> Service pre-start command's stderr: Patching error: Traceback (most recent call last):
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 98, in doIncrementalPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     mod.doPatchingWithDependencies()
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 405, in doPatchingWithDependencies
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     patch_authz(featureState)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 379, in patch_authz
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     authz_patch.setup_roles("/usr/lib/vmware-wcp/csi-roles.xml")
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 208, in setup_roles
<YYYY-MM-DDTHH:MM:SS>Wa(03)+ host-####     raise Exception("Role %s (id: %d) not found in VC." % (expected_role["name"], expected_role_id))
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-#### Exception: Role CNS-Datastore (id: ##) not found in VC.
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-#### During handling of the above exception, another exception occurred:
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####
<YYYY-MM-DDTHH:MM:SS>Wa(03)+ host-#### Traceback (most recent call last):
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 332, in <module>
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     doPatching(patching_in_post_rpm)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 64, in doPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     doIncrementalPatching(patching_in_post_rpm)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 102, in doIncrementalPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####     raise Exception(err_msg % (module_path, str(e)))
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-#### Exception: Failed to patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC.
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-####
<YYYY-MM-DDTHH:MM:SS> Er(02) host-#### <wcp> Service pre-start command failed with exit code 1.

Environment

vCenter Server 7.0.x

vCenter Server 8.0.x

Cause

This issue is caused due to the role "CNS-Datastore" or "CNS-SEARCH-AND-SPBM" already available on the vCenter Server. This role is created when vCenter Server is integrated with Vanilla CSI+Kubernetes and 8.0 U3 patching process tries to create the role with same name and results in a conflict.

Resolution

Workaround:

Follow below steps to workaround the issue :

If VC patching is already in failed state, click on "Rollback to the last vCenter Server backup version" to rollback the patch operation OR as an alternate option revert vCenter Server VM to the snapshot which was taken prior to the patching.
Locate existing CNS-Datastore role in the VC. Found under Administration -> Roles.
Select the CNS-Datastore role and note the number of rows in the 'Usage' tab.
Click EDIT at the top.
Rename role name from CNS-Datastore to CNS-Vanilla-Datastore without changing anything else. Hit SAVE.
Check the 'Usage' tab again for this CNS-Vanilla-Datastore. It should match the previous rows as we are only changing the name.
Perform the same rename operation for role name "CNS-SEARCH-AND-SPBM" (rename CNS-SEARCH-AND-SPBM to CNS-Vanilla-SEARCH-AND-SPBM)
Restart the patching to 8.0 U3.