Patching vCenter Server to 8.0 U3 fails at 95% with error "Exception occured in postInstallHook for wcp:Patch"

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Patching vCenter Server to 8.0 U3 fails during PostInstallHook phase.
  
  
  
  
• Patchrunner.log shows the error message "Failed to apply patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC"
  
  <Timestamp> wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1090', 'name': 'CNS-Datastore', 'description': 'Allows performing read, write, delete, and rename operations in the datastore browser. ', 'priv_ids': ['System.Read', 'System.Anonymous', 'System.View', 'Datastore.AllocateSpace', 'Datastore.FileManagement']}
<Timestamp> wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
<Timestamp> wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
<Timestamp> wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updateu46go3kh/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 329, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC..
2024-07-02T11:14:33.677Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
  File "/storage/seat/software-updateu46go3kh/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
2024-07-02T11:14:34.681Z WARNING root stopping status aggregation...
2024-07-02T11:14:34.682Z ERROR __main__ Patch vCSA failed

Environment

vCenter Server 8.0 U3

Cause

This issue is caused due to the role "CNS-Datastore" or "CNS-SEARCH-AND-SPBM" already available on the vCenter Server. This role is created when vCenter Server is integrated with Vanilla CSI+Kubernetes and 8.0 U3 patching process tries to create the role with same name and results in a conflict.

Resolution

This is a known issue with vCenter Server 8.0 U3, engineering is actively working towards fixing this issue.

Workaround:

Follow below steps to workaround the issue :

1. If VC patching is already in failed state, click on "Rollback to the last vCenter Server backup version" to rollback the patch operation OR as an alternate option revert vCenter Server VM to the snapshot which was taken prior to the patching. 
2. Locate existing CNS-Datastore role in the VC. Found under Administration -> Roles.
3. Select the CNS-Datastore role and note the number of rows in the 'Usage' tab. 
4. Click EDIT at the top. 
5. Rename role name from CNS-Datastore to CNS-Vanilla-Datastore without changing anything else. Hit SAVE. 
6. Check the 'Usage' tab again for this CNS-Vanilla-Datastore. It should match the previous rows as we are only changing the name. 
7. Perform the same rename operation for role name "CNS-SEARCH-AND-SPBM" (rename CNS-SEARCH-AND-SPBM to CNS-Vanilla-SEARCH-AND-SPBM)
8. Restart the patching to 8.0 U3.