Patching vCenter Server to 8.0 U3 fails at 95% with error "Exception occured in postInstallHook for wcp:Patch"

Products

• VMware vCenter Server 8.0

Issue/Introduction

• Patching vCenter Server to 8.0 U3 fails during PostInstallHook phase with error "Exception occured in postInstallHook for wcp:Patch"
  
  
  
  
• Patchrunner.log shows the error message "Failed to apply patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC"
  
  <YYYY-MM-DDTHH:MM:SS> wcp:Patch INFO roles_groups_users Checking if privileges should be updated for role {'id': '1090', 'name': 'CNS-Datastore', 'description': 'Allows performing read, write, delete, and rename operations in the datastore browser. ', 'priv_ids': ['System.Read', 'System.Anonymous', 'System.View', 'Datastore.AllocateSpace', 'Datastore.FileManagement']}
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
<YYYY-MM-DDTHH:MM:SS> wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/seat/software-updateu46go3kh/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 329, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC..
<YYYY-MM-DDTHH:MM:SS> ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
  File "/storage/seat/software-updateu46go3kh/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
    raise ex
patch_errors.ComponentError
<YYYY-MM-DDTHH:MM:SS> WARNING root stopping status aggregation...
<YYYY-MM-DDTHH:MM:SS> ERROR __main__ Patch vCSA failed

Upgrading vCenter from 7.0 U3 to 8.0 U3 also fails with the error : Failed to start wcp service at 95%.

 

<YYYY-MM-DDTHH:MM:SS> In(05) host-18727 <wcp-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-wcp/wcpsvc-prestart.py
<YYYY-MM-DDTHH:MM:SS> Wa(03) host-18727 <wcp> Service pre-start command's stderr: Patching error: Traceback (most recent call last):
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 98, in doIncrementalPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     mod.doPatchingWithDependencies()
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 405, in doPatchingWithDependencies
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     patch_authz(featureState)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 379, in patch_authz
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     authz_patch.setup_roles("/usr/lib/vmware-wcp/csi-roles.xml")
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 208, in setup_roles
<YYYY-MM-DDTHH:MM:SS>Wa(03)+ host-18727     raise Exception("Role %s (id: %d) not found in VC." % (expected_role["name"], expected_role_id))
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727 Exception: Role CNS-Datastore (id: 1090) not found in VC.
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727 During handling of the above exception, another exception occurred:
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727
<YYYY-MM-DDTHH:MM:SS>Wa(03)+ host-18727 Traceback (most recent call last):
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 332, in <module>
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     doPatching(patching_in_post_rpm)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 64, in doPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     doIncrementalPatching(patching_in_post_rpm)
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 102, in doIncrementalPatching
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727     raise Exception(err_msg % (module_path, str(e)))
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727 Exception: Failed to patch roles_groups_users! Error: Role CNS-Datastore (id: 1090) not found in VC.
<YYYY-MM-DDTHH:MM:SS> Wa(03)+ host-18727
<YYYY-MM-DDTHH:MM:SS> Er(02) host-18727 <wcp> Service pre-start command failed with exit code 1.

 

 

Environment

vCenter Server 8.0 U3

Cause

This issue is caused due to the role "CNS-Datastore" or "CNS-SEARCH-AND-SPBM" already available on the vCenter Server. This role is created when vCenter Server is integrated with Vanilla CSI+Kubernetes and 8.0 U3 patching process tries to create the role with same name and results in a conflict.

Resolution

This issue is resolved in vCenter Server 8.0 U3b (Build Number - 24262322).

Workaround:

Follow below steps to workaround the issue :

1. If VC patching is already in failed state, click on "Rollback to the last vCenter Server backup version" to rollback the patch operation OR as an alternate option revert vCenter Server VM to the snapshot which was taken prior to the patching. 
2. Locate existing CNS-Datastore role in the VC. Found under Administration -> Roles.
3. Select the CNS-Datastore role and note the number of rows in the 'Usage' tab. 
4. Click EDIT at the top. 
5. Rename role name from CNS-Datastore to CNS-Vanilla-Datastore without changing anything else. Hit SAVE. 
6. Check the 'Usage' tab again for this CNS-Vanilla-Datastore. It should match the previous rows as we are only changing the name. 
7. Perform the same rename operation for role name "CNS-SEARCH-AND-SPBM" (rename CNS-SEARCH-AND-SPBM to CNS-Vanilla-SEARCH-AND-SPBM)
8. Restart the patching to 8.0 U3.