Patching an ELM vCenter 7.x/8x fails with an error "Failed to apply patch roles_groups

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

when patching a vCenter Server 7.x/8.x, which is part of an Enhanced Linked Mode replication, using the command line, you see the below error in the Shell:

[YYYY-MM-DDTHH:MM:SS] : Patch command patch failed 
[YYYY-MM-DDTHH:MM:SS] :
    Mismatch:
        summary: Failed to apply patch roles_groups_users! Error: (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  dynamicProperty = (vmodl.DynamicProperty) [],
  msg = 'java.lang.NumberFormatException',
  faultCause = <unset>,
  faultMessage = (vmodl.LocalizableMessage) [],
  reason = 'Invalid fault'
}.

Patchrunner.log, located in /var/log/vmware/applmgmt/, has the following entries:

[YYYY-MM-DDTHH:MM:SS] wcp:Patch INFO root Updating user "wcp" in SSO 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch INFO root Removing certificate for user "wcp" from the filesystem 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch INFO root Adding "wcp" solution user to "ActAsUsers" group. 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch INFO root Adding "wcp" solution user to "NsxAdministrators" group. 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Failed to apply patch %s! Error: %s. 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR wcp Not all patches were applied. Latest applied patch is 1 
[YYYY-MM-DDTHH:MM:SS] wcp:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'wcp:Patch' failed.
Traceback (most recent call last):
 File "/storage/core/software-packages/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in execute
Hook
  executionResult = systemExtension(args)
 File "/storage/core/software-packages/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
  result = self.extension(*args)
 File "/storage/core/software-packages/scripts/patches/libs/sdk/extensions.py", line 123, in _func
  return func(*args)
 File "/storage/core/software-packages/scripts/patches/payload/components-script/wcp/__init__.py", line 213, in doPatching
  doIncrementalPatching(current_version)
 File "/storage/core/software-packages/scripts/patches/payload/components-script/wcp/__init__.py", line 340, in doIncrementalPatching
  raise user_error
patch_errors.UserError: Failed to apply patch roles_groups_users! Error: (vmodl.fault.SystemError) {
  dynamicType = <unset>,
  dynamicProperty = (vmodl.DynamicProperty) [],
  msg = 'java.lang.NumberFormatException',
  faultCause = <unset>,
  faultMessage = (vmodl.LocalizableMessage) [],
  reason = 'Invalid fault'
}.

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

This issue occurs when one of the following namespaces in VMDir does not contain the vmwAuthzPrivilegeVersion:

cn=Namespaces.Backup,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
cn=Namespaces.ManageCapabilities,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
cn=Namespaces.SelfServiceManage,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local

Those Namespaces after they are created and never modified during the patching process due to the missing vmwAuthzPrivilege entries.

Those entries may differ from one case to another

Resolution

Install JXplorer using JXplorer download page.
- Disclaimer: VMware is not responsible for the reliability of any data, opinions, advice, or statements made on third-party websites. Inclusion of such links does not imply that VMware endorses, recommends, or accepts any responsibility for the content of such sites.
Open the JXplorer utility
Click File > Connect and enter the following information:

Host: FQDN_of_SSO_Server
Protocol: LDAP v3
Base DN: dc=vsphere,dc=local
Security Level: User + Password
Security User DN: cn=administrator,cn=users,dc=vsphere,dc=local
Security Password: administrator password for the identity or SSO server.

Check the below entries in the path Services > VmwAuthz > PrivilegeModel
- Check Namespaces.Backup, Namespaces.ManageCapabilities and Namespaces.SelfServiceManage
- Check if any of them is missing vmwAuthzPrivilegeVersion value
- To Verify information from vCenter SSH if Jxplorer is not available, use commands (Replace the 'administrator_password' in the below commands with the correct password for the SSO administrator (e.g. [email protected]) )

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Namespaces.SelfServiceManage,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'administrator_password'

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Namespaces.ManageCapabilities,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'administrator_password'

ldapsearch -o ldif-wrap=no -LLL -h localhost -b "cn=Namespaces.Backup,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'administrator_password'

Normally they should look like this from Jxplorer

If those Attributes are missing any of the vmwAuthzPrivilege values
Delete the attribute using Jxplorer

To delete attribute from vCenter SSH if Jxplorer is not available execute command (Change the cn to attribute which you want to delete. Replace the 'administrator_password' with the correct password for SSO administrator)

ldapdelete -H ldap://localhost:389 -x -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'administrator_password' "cn=Namespaces.SelfServiceManage,cn=PrivilegeModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local"

Re-add the attribute using the vCenter SSH Session, by running the below commands (Replace the 'administrator_password' with the correct password for the SSO administrator (e.g. [email protected]) ).

For Namespaces.Backup

/opt/likewise/bin/ldapadd -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -w 'administrator_password' <<EOF
version: 1
dn: cn=Namespaces.Backup,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
objectClass: vmwAuthzPrivilege
objectClass: top
cn: Namespaces.Backup
vmwAuthzPrivilegeVersion: 1
vmwAuthzPrivilegeIsOnParent: FALSE
vmwAuthzPrivilegeGroupName: Namespaces
vmwAuthzPrivilegeDescription:: IA==
vmwAuthzPrivilegeId: Namespaces.Backup
EOF

For Namespaces.ManageCapabilities

/opt/likewise/bin/ldapadd -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -w 'administrator_password' <<EOF
version: 1
dn: cn=Namespaces.ManageCapabilities,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
objectClass: vmwAuthzPrivilege
objectClass: top
cn: Namespaces.ManageCapabilities
vmwAuthzPrivilegeVersion: 1
vmwAuthzPrivilegeIsOnParent: FALSE
vmwAuthzPrivilegeGroupName: Namespaces
vmwAuthzPrivilegeDescription:: IA==
vmwAuthzPrivilegeId: Namespaces.ManageCapabilities
EOF

For Namespaces.SelfServiceManage

/opt/likewise/bin/ldapadd -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -w 'administrator_password' <<EOF
version: 1
dn: cn=Namespaces.SelfServiceManage,cn=PrivilegeModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
objectClass: vmwAuthzPrivilege
objectClass: top
cn: Namespaces.SelfServiceManage
vmwAuthzPrivilegeVersion: 0
vmwAuthzPrivilegeIsOnParent: FALSE
vmwAuthzPrivilegeGroupName: Namespaces
vmwAuthzPrivilegeDescription:: IA==
vmwAuthzPrivilegeId: Namespaces.SelfServiceManage
EOF

Verify that all the attributes have the vmwAuthzPrivilege values
Try again to patch the vCenter Server.