vCenter Server Patching fails with Error: Data conversion/Post install hook failed

Products

• VMware vCenter Server

Issue/Introduction

• vCenter Patching via VAMI console fails with Error as follows:
  
  
  
  
• Under vCenter's /var/log/vmware/applmgmt/Patchrunner.log, below error log snips are found:
  
  YYYY-MM-DDTHH:MM:SS wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s.
YYYY-MM-DDTHH:MM:SS wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1
YYYY-MM-DDTHH:MM:SS wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
  File "/storage/updatemgr/software-updatepw8okahb/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
    executionResult = systemExtension(args)
    return func(*args)
  File "/storage/updatemgr/software-updatepw8okahb/stage/scripts/patches/payload/components-script/wcp/__init__.py", line 343, in doIncrementalPatching
    raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: (dataservice.fault.AlreadyExistsFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [ ]
  
  
• Under vCenter's /var/log/vmware/upgrade/import.json, below error log snips are found:
  
  
  "id": "ur.start.services.failed.text",
  "translatable": "Failed to start service %(0)s.",
   "args": [
  "wcp"
   ],
   "localized": "Failed to start service wcp."
   }
   ],
   "componentKey": "upgrade_framework",
   "problemId": null,
   "resolution": null
  
  
  
• Under vCenter's /var/log/vmware/vmon/vmon.log, below error log snips are found:
  
  <wcp-prestart> Constructed command: /usr/bin/python /usr/lib/vmware-wcp/wcpsvc-prestart.py
<wcp> Service pre-start command's stderr: Patching error: Traceback (most recent call last):
   File "/usr/lib/vmware-wcp/./day0_patching/post_rpm/wcp/patchingOrchestrator.py", line 98, in doIncrementalPatching
     mod.doPatchingWithDependencies()
   File "/usr/lib/vmware-wcp/day0_patching/post_rpm/wcp/patches/roles_groups_users.py", line 478, in doPatchingWithDependencies
     patch_authz(featureState)
  
  
• Under vCenter's /var/log/vmware/vpxd-svcs/vpxd-svcs.log, below error log snips are found:
  

  VirtualMachinertualMachine. Config.AddExistingDisk', 'System. Read', 'System. View' , 'Resource. HotMigrate' , 'Host. Config. Storage' , 'VirtualMachine. Config. AddRemoveDevice' , 'Resource. ColdMigrate', 'VirtualMachine. Config.RemoveDisk' ' System. Anonymous' } ]
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users No new privileges to grant to CNS-SUPERVISOR-SEARCH-AND-SPBM
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users Successfully loaded authz roles, added 0 new roles
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users Checking if role names should be updated for CSI role {'id' : '1090','name' : 'CNS-SUPERVISOR-SEARCH-AND-SPBM' , 'description' : 'Allows performing read, write, delete, and rename operations in the datastore browser. ', 'priv_ids' : [ 'System. Read', 'System. Anonymous','System. View', 'Datastore. AllocateSpace', 'Datastore. FileManagement']}
VirtualMachinertualMachine. Config.AddExistingDisk', 'System. Read', 'System. View' , 'Resource. HotMigrate' , 'Host. Config. Storage' , 'VirtualMachine. Config. AddRemoveDevice' , 'Resource. ColdMigrate', 'VirtualMachine. Config.RemoveDisk' ' System. Anonymous' } ]
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users No new privileges to grant to CNS-HOST-CONFIG-STORAGE-AND-CNS-VM
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users Successfully loaded authz roles, added 0 new roles
YYYY-MM-DDTHH:MM:SS wcp: Patch INFO roles groups users Checking if role names should be updated for CSI role {'id' : '1090','name' : 'CNS-SUPERVISOR-DATASTORE' , 'description' : 'Allows performing read, write, delete, and rename operations in the datastore browser. ', 'priv_ids' : [ 'System. Read', 'System. Anonymous','System. View', 'Datastore. AllocateSpace', 'Datastore. FileManagement']}
[authz-service-3 [] INFO com.vmware.cis.core.authz.accesscontrol.impl.AuthzServiceBaseImpl opId=########-####-####-####-############] [updateRoleByUser] Updating role with ID: 1090
[authz-service-3 [] ERROR com.vmware.cis.core.authz.accesscontrol.impl.AuthzServiceBaseImpl opId=########-####-####-####-############] [updateRoleByUser] Role with name CNS-SUPERVISOR-DATASTORE already exists!

Environment

• vCenter Server 8.x
• vCenter Server 9.x

Cause

A issue occurs when there is a Duplicate Role Names under under the vCenter, that has caused due to a previous upgrade/patching attempt failure.

Resolution

To resolve this issue, follow below steps:

Option 1:

1. Login to the vCenter UI, with SSO credentials.
2. Navigate to Administration > Roles.
3. Delete below Roles.
   
   CNS-SUPERVISOR-DATASTORE
CNS-SUPERVISOR-HOST-CONFIG-STORAGE-AND-CNS-VM
CNS-SUPERVISOR-SEARCH-AND-SPBM
   
   Note: Additionally, search for any Duplicate Roles and Delete one of them, which is not assigned with any USAGE.
   
   
4. Access the vCenter via SSH with root credentials.
5. And restart the vpxd-svcs service, using below command:
   
   service-control --restart vpxd-svcs
   
   
6. Go to VAMI Page and Resume vCenter update.

Option 2:

1. Revert the vCenter back to the Snapshot, that was taken before the Patching attempt or restore from last good backup and make the vCenter healthy.
2. Follow the steps from Option 1 (point a to e).
3. Login to VAMI and start the patching.