vCenter upgrade from 7.x to 8.x fails at 82% in Stage 2 with the error "An error occurred while starting service 'hvc''"

Products

VMware vCenter Server

Issue/Introduction

During an upgrade from vCenter Server 7.x to 8.x, the upgrade process may fail at 82% with the following error:

The following entries are seen in /var/log/vmware/hvc/hvc-svcs.log:

[YYYY-MM-DDTHH:MM:SS] [main [] WARN org.springframework.context.support.ClassPathXmlApplicationContext opId=] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'isAccessLogCreated' defined in class path resource [vlsi-server.xml]: Cannot resolve reference to bean 'authzFilter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authzFilter' defined in class path resource [vlsi-server.xml]; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.hvc.synccontroller.Controller]: Constructor threw exception; nested exception is com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
defaultMessage = Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError,
args = [com.vmware.vapi.std.errors.InternalServerError],
}],
errorType = INTERNAL_SERVER_ERROR
}

[YYYY-MM-DDTHH:MM:SS]][main [] ERROR com.vmware.hvc.service.Main opId=] start: Hybrid VC Service failed to start

The following entries are seen in /var/log/firstboot/hvc_firstboot.py_XXXX_stderr.log

[YYYY-MM-DDTHH:MM:SS] ERROR starting hvc rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
Traceback (most recent call last):
File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 217, in Main
hvcsvc_fb.start_hvcsvc()
File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 54, in start_hvcsvc
self.start_service()
File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 266, in start_service
service_start(self.get_eff_service_name())
File "/usr/lib/vmware/site-packages/cis/utils.py", line 1087, in service_start
raise ServiceStartException(svc_name)
cis.exceptions.ServiceStartException: {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"hvc"
],
"localized": "An error occurred while starting service 'hvc'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
[YYYY-MM-DDTHH:MM:SS] HybridVC Service firstboot failed due to <cis.componentStatus.ErrorInfo object at 0x#########650>

Environment

VMware vCenter Server 7.x

Cause

The Hybrid VC (HVC) service requires the SyncUsers role to be assigned the role ID 1002. In the affected vCenter Server, the SyncUsers role is present but mapped to an incorrect role ID (for example, -71#####28). Because of this mismatch, the HVC service is unable to identify the correct role, resulting in the service failing to start and the upgrade process failing.

Resolution

Note: If the affected vCenter is standalone, take a valid snapshot of the vCenter Server. If the affected vCenter is in Enhanced Linked Mode (ELM), take a power-off snapshot of all the vCenters.

Power off the failed vCenter 8.x appliance.
Delete the vCenter 8.x VM.
Power on the original vCenter 7.x appliance and run the following steps:
1. Identify the Existing SyncUsers Role running the below command, (Note: Replace dc=vsphere,dc=local with the correct vSphere domain name)
  - - /opt/likewise/bin/ldapsearch -b "cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local" \ -s sub -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -W \ > /tmp/ldap_output.txt
    - less /tmp/ldap_output.txt
    - Look for vmwAuthzRoleName: SyncUsers and verify the CN ID.
2. Delete the incorrect SyncUsers role: (Replace <incorrect_cn_id_present> with the CN identified from the LDAP search)
  - /opt/likewise/bin/ldapdelete -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W "cn=<incorrect_cn_id_present>,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local"
3. Recreate the SyncUsers role with ID 1002
  - /opt/likewise/bin/ldapadd -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
    version: 1
    dn: cn=1002,cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
    objectClass: vmwAuthzRole
    objectClass: top
    cn: 1002
    vmwAuthzRoleDescription: This role entitles you to perform operations required for sync
    vmwAuthzRoleName: SyncUsers
    vmwAuthzRolePrivilegeId: System.Anonymous
    vmwAuthzRolePrivilegeId: System.Read
    vmwAuthzRolePrivilegeId: System.View
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditTag
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.AttachTag
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateCategory
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForCategory
    vmwAuthzRolePrivilegeId: HLM.Manage
    vmwAuthzRolePrivilegeId: IntercomNamespace.Read
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateTag
    vmwAuthzRolePrivilegeId: IntercomNamespace.Write
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteTag
    vmwAuthzRolePrivilegeId: SettingsStore.Manage
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditCategory
    vmwAuthzRolePrivilegeId: CertificateManagement.Manage
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteCategory
    vmwAuthzRolePrivilegeId: Trust.Manage
    vmwAuthzRolePrivilegeId: HLM.Create
    vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForTag
    vmwAuthzRoleVersion: 7
    EOF
    
    Example:
4. Provide the SSO Administrator/LDAP password when prompted.
5. Restart all vCenter services
  - service-control --stop --all && service-control --start --all
6. Reinitiate the vCenter Upgrade.

Additional Information

For more information, review PR 3006863