Minor update to vCenter 7.0Update 3c/3d/3e/3f from an older 7.0 version or 8.0U1 to U3 fails with error "invalid type expected string instead got none type"

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

Attempts to update vCenter to 7.0 Update 3c, Update 3d, Update 3e or Update 3f fail with "invalid type expected string instead got none type"

Patchrunner.log, located in /var/log/vmware/applmgmt/, contains the following error stack:

2022-07-18 06:39:31,586.586Z content-library:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'content-library:Patch' failed.
Traceback (most recent call last):
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook
  executionResult = systemExtension(args)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/libs/sdk/extensions.py", line 106, in __call__
  result = self.extension(*args)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/libs/sdk/extensions.py", line 123, in _func
  return func(*args)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/__init__.py", line 279, in execute_patch
  register_cls()
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/patches/add_new_cls_privileges.py", line 151, in register_cls
  reg_info.registerAll(solution_user_name, solution_user_id, service_id=service_id)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/patches/cis_register.py", line 387, in registerAll
  self.registerUserAndService(user_name, user_id, service, service_id=service_id)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/patches/cis_register.py", line 422, in registerUserAndService
  create_sso_groups(service_spec)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/patches/cis_register.py", line 623, in create_sso_groups
  authz_patch.assign_groups_to_roles(service_spec['group-role'])
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/payload/components-script/content-library/patches/cis_register.py", line 591, in assign_groups_to_roles
  self.domain_name, role, group, True)
 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 706, in set_permission
  self._authz_service)
 File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 685, in create_access_control
  role_ids.append(role_objs_dict[rolename])
KeyError: 'com.vmware.Content.Registry.Admin'
2022-07-18T06:39:31.597Z ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError.
Traceback (most recent call last):
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch
  _patchComponents(ctx, userData, statusAggregator.reportingQueue)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 85, in _patchComponents
  executeComponentHook(Hook.Patch, ctx, c, userData, reportingQueue)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 98, in executeComponentHook
  reportQueue, identifier, expectedResultType)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/executor/execution_facade.py", line 53, in executeHook
  result = executor.executeHook(scriptFile, hook, args, reportQueue, reportIdentifier)
 File "/storage/updatemgr/software-update9r0_rc2_/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor_process.py", line 119, in executeHook
  raise ex
patch_errors.ComponentError
2022-07-18T06:39:32.601Z WARNING root stopping status aggregation...
2022-07-18T06:39:32.602Z ERROR __main__ Patch vCSA failed

Environment

VMware vCenter Server 7.0.3

VMware vCenter Server 8.0.3

Cause

This error can occur when one of more of the default roles in vCenter have been modified, resulting in their vmwAuthzRoleName property being different from what it should be.
The update installer uses this property to identify the roles to use them when reregistering the services during the update, and will fail when it cannot find a specific role.

It can also happen if the role is missing it's vmwAuthzRoleName value

Resolution

To resolve this problem apply the following steps:

Restore the vCenter backup which you created earlier, before proceeding.
Download the script repair_roles.sh attached to this KB.
Upload the script to the vCenter Server Appliance to the /tmp/ directory.
Conenct via SSH, go to the /tmp/ folder

Note: while most default roles have fixed role IDs, the ID for the role "com.vmware.Content.Admin" can vary. Therefore you need to edit the script and change this ID to the correct one for the specific environment.

To find the role ID for your vCenter, run the following command (If this command is blank check the bottom of the KB for corrected steps):

# /opt/likewise/bin/ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W | grep -A 4 com.vmware.Content.Admin

When asked for, provide the password for the SSO administrator. The output will look like this:

vmwAuthzRoleName: com.vmware.Content.Admin
vmwAuthzRoleDescription: Administrator user for Content Library
objectClass: top
objectClass: vmwAuthzRole
cn: 1111111111

Note the value of "cn:" highlighted in green. This is the role ID.

Edit the repair_roles.sh script and find the following block:

/opt/likewise/bin/ldapmodify -x -h localhost -p 389 -D 'cn=Administrator,cn=Users,dc=vsphere,dc=local' -W << EOF
dn: cn=-1615445737,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
changetype: modify
replace: vmwAuthzRoleName
vmwAuthzRoleName: com.vmware.Content.Admin
EOF

Change the numeric value for the first cn to the one from your vCenter. Example:

/opt/likewise/bin/ldapmodify -x -h localhost -p 389 -D 'cn=Administrator,cn=Users,dc=vsphere,dc=local' -W << EOF
dn: cn=1111111111,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
changetype: modify
replace: vmwAuthzRoleName
vmwAuthzRoleName: com.vmware.Content.Admin
EOF

Save the script and change the permissions of the file to make it executable:
```
# chmod +x repair_roles.sh
```
Run the script:
```
# ./repair_roles.sh
```

Note: While running, the script will ask you for the LDAP password multiple times (as many as eight times). Each time, enter the password for the [email protected] account, or of the administrator account of your SSO domain.

10. Restart services on VCSA to ensure change is applied:

service-control --stop --all && service-control --start --all

11. Retry the update.

If the command from Step 5 is blank, use the following steps:

1. Collect the correct ID for the com.vmware.Content.Admin role:

cat /var/log/firstboot/content-library-firstboot.py_YYY_stdout.log | grep -i "com.vmware.Content.Admin, role.id"

(replace YYY with the number from the log file)

2. Use the following command to add the missing com.vmware.Content.Admin

/opt/likewise/bin/ldapmodify -x -h localhost -p 389 -D 'cn=Administrator,cn=Users,dc=vsphere,dc=local' -W << EOF
dn: cn=ZZZ,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
changetype: modify
replace: vmwAuthzRoleName
vmwAuthzRoleName: com.vmware.Content.Admin
EOF

The above section of commands should be copied and pasted as a single block of command. Replace ZZZ with the ID found in step one

Attachments

repair_roles get_app