vCenter Server Upgrade from 7.x to 8.x fails in Stage 2 while starting HVC Service

Products

VMware vCenter Server 7.0

Issue/Introduction

The upgrade process halts in Stage 2 when the HVC service fails to start.

Error message in the Installer

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 153, in register_with_LS cloudvm_sso_cm_register(keystore, File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 728, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1240, in do_lsauthz_operation authz_client.set_permission(domain_name, File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 748, in set_permission ac = self.create_access_control(principal, rolenames, File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 728, in create_access_control role_ids.append(role_objs_dict[rolename]) KeyError: 'SyncUsers' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 186, in Main hvcsvc_fb.register_with_LS() File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 160, in register_with_LS raise createBaseInstallException( NameError: name 'createBaseInstallException' is not defined
Resolution
This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

From /var/log/firstboot/hvc_firstboot.py_14802_stderr.log

Traceback (most recent call last):
  File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 159, in register_with_LS
    dynVars=dynVars)
  File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 706, in cloudvm_sso_cm_register
    serviceId = do_lsauthz_operation(cisreg_opts_dict)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1102, in do_lsauthz_operation
    True)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 661, in set_permission
    self._authz_service)
  File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 640, in create_access_control
    role_ids.append(role_objs_dict[rolename])
KeyError: 'SyncUsers'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 192, in Main
    hvcsvc_fb.register_with_LS()
  File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 162, in register_with_LS
    raise createBaseInstallException(
NameError: name 'createBaseInstallException' is not defined

Cause

HVC firstboot uses a specific function to register the service, add the SyncUsers role, and add access control (global permission) using the role SyncUsers (roleId=1002). This function adds the permission based on the role name (NOT the role ID). If the vmwAuthzRoleName is changed, the HVC service will not register successfully.

Resolution

Note: If the affected vCenter is standalone, take a valid snapshot of the vCenter Server. If the affected vCenter is in Enhanced Linked Mode (ELM), take a power-off snapshot of all the vCenters.

Method #1: Using JXplorer to Connect to vSphere Single Sign-On

Refer to the article on Using JXplorer to Connect to the vSphere Single Sign-On.
Navigate to Services > VmwAuthz > RoleModel and expand role-id 1002.

Sample Output:
If the vmwAuthzRoleName attribute is not "SyncUsers," edit the name.
- Click "Table Editor" and change the vmwAuthzRoleName to "SyncUsers."
- Click the "Submit" button to save the entry.
Stop and start all vCenter services using the following command:
```
service-control --stop --all && service-control --start --all
```

Method #2: Using Command-Line Interface

If the JXplorer cannot be installed due to security policies, use the following steps:

SSH into the vCenter:
- Use root credentials to SSH into the affected vCenter server.

Find the Current Role Name:

Run the following command to find the information on role ID 1002:

/opt/likewise/bin/ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W | grep -A 32 1002

Sample Output: -

root@<FQDN> [ ~ ]# /opt/likewise/bin/ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W | grep -A 32 1002
Enter LDAP Password:
# 1002, RoleModel, VmwAuthz, services, vsphere.local
dn: cn=1002,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
cn: 1002
objectClass: top
objectClass: vmwAuthzRole
nTSecurityDescriptor:: AQAHhBQAAAA0AAAAAAAAAFQAAAABBgAAAAAABxUAAAC9C8XNvsq5u+k
 hRXXosilu9AEAAAEGAAAAAAAHFQAAAL0Lxc2+yrm76SFFdeiyKW4gAgAAAgDAAAUAAAAAEygAMwAG
 IAEGAAAAAAAHFQAAAL0Lxc2+yrm76SFFdeiyKW70AQAAABMoADMABiABBgAAAAAABxUAAAC9C8XNv
 sq5u+khRXXosiluIAIAAAATKAAzAAYgAQYAAAAAAAcVAAAAvQvFzb7KubvpIUV16LIpbgACAAAAEy
 gAMwAGAAEGAAAAAAAHFQAAAL0Lxc2+yrm76SFFdeiyKW4DAgAAABMYADAAAAABAgAAAAAAByAAAAC
 aAgAA
vmwAuthzRolePrivilegeId: System.Anonymous
vmwAuthzRolePrivilegeId: System.Read
vmwAuthzRolePrivilegeId: System.View
vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditTag
vmwAuthzRolePrivilegeId: InventoryService.Tagging.AttachTag
vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateCategory
vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForCategory
vmwAuthzRolePrivilegeId: HLM.Manage
vmwAuthzRolePrivilegeId: IntercomNamespace.Read
vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateTag
vmwAuthzRolePrivilegeId: IntercomNamespace.Write
vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteTag
vmwAuthzRolePrivilegeId: SettingsStore.Manage
vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditCategory
vmwAuthzRolePrivilegeId: CertificateManagement.Manage
vmwAuthzRolePrivilegeId: ReplicationService.Administer
vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteCategory
vmwAuthzRolePrivilegeId: Trust.Manage
vmwAuthzRolePrivilegeId: HLM.Create
vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForTag
vmwAuthzRoleVersion: 3
vmwAuthzRoleName: SyncUsers
vmwAuthzRoleDescription: This role entitles you to perform operations required
  for sync

Edit the Role Name:

If the "vmwAuthzRoleName" name is not "SyncUsers", update it using the command below:

/opt/likewise/bin/ldapmodify -x -h localhost -p 389 -D 'cn=Administrator,cn=Users,dc=vsphere,dc=local' -W << EOF
dn: cn=-1002,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local
changetype: modify
replace: vmwAuthzRoleName
vmwAuthzRoleName: SyncUsers
EOF

Restart vCenter Services:

Stop and start all vCenter services using the following command:
```
service-control --stop --all && service-control --start --all
```