HVC service fails to start causing 'Exception occurred in PostInstallHook' on patching vCenter or SSL Certificate Install Failing VCSA 7.x

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

When patching vCenter 7.0.3, it stops on 80% with the error message "Exception occured in postInstallHook"
SSL certificate install fails on vCSA 7.x
hvc service fails to start even after resetting STS certificate or all machine and solution user certificates
Rebuilding the solution user using lsdoctor -u or rebuilding the hvc registration with lsdoctor -r does not resolve the issue.
/var/log/vmware/applmgmt/PatchRunner.log

YYYY-MM-DD HH:MM:SS last_component:Patch INFO last_component Starting all VMware services... The immediate command stdout is redirected to file /var/log/vmware/applmgmt/startAllServices.out.log and stderr is redirected to file /var/log/vmware/applmgmt/startAllServices.err.log YYYY-MM-DD HH:MM:SS last_component:Patch INFO last_component Start All VMware services: command=['/bin/service-control', '--start', '--all'], exit-code=1, stdout=Operation not cancellable. Please wait for it to finish... Performing start operation on service lwsmd... Successfully started service lwsmd Performing start operation on service vmafdd... Successfully started service vmafdd Performing start operation on service vmdird... Successfully started service vmdird Performing start operation on service vmcad... Successfully started service vmcad Performing start operation on profile: ALL... , stderr=Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start hvc services. Error: Operation timed out ... ... YYYY-MM-DD HH:MM:SS last_component:Patch ERROR vmware_b2b.patching.executor.hook_executor Patch hook 'last_component:Patch' failed. Traceback (most recent call last): File "/storage/seat/software-updatektk1cx54/stage/scripts/patches/payload/components-script/last_component/__init__.py", line 242, in _perfromStartAllVmwareServices ... raise UserError(FAILED_TO_START_SERVICES_TEXT) patch_errors.UserError: Failed to start all services after successful patching. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/storage/seat/software-updatektk1cx54/stage/scripts/patches/py/vmware_b2b/patching/executor/hook_executor.py", line 74, in executeHook executionResult = systemExtension(args) ... patch_errors.UserError: Failed to start all services after successful patching. ... YYYY-MM-DD HH:MM:SS ERROR vmware_b2b.patching.phases.patcher Patch hook Patch got ComponentWrapperError. Traceback (most recent call last): File "/storage/seat/software-updatektk1cx54/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 203, in patch _patchComponents(ctx, userData, statusAggregator.reportingQueue) ... YYYY-MM-DD HH:MM:SS WARNING root stopping status aggregation... YYYY-MM-DD HH:MM:SS ERROR __main__ Patch vCSA failed

/var/log/vmware/hvcs/hvc-svcs.log

YYYY-MM-DD HH:MM:SS [main ERROR com.vmware.hvc.service.Main opId=] start: Hybrid VC Service failed to start ... com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => { messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => { id = vapi.bindings.method.impl.unexpected, defaultMessage = Provider method implementation threw unexpected exception: com.vmware.vapi.std.errors.InternalServerError, args = [com.vmware.vapi.std.errors.InternalServerError], params = <null>, localized = <null> }], data = <null>, errorType = INTERNAL_SERVER_ERROR } ...

/var/log/vmware/vmon/vmon.log

YYYY-MM-DD HH:MM:SS In(05) host-4041 <hvc> Service start operation timed out. YYYY-MM-DD HH:MM:SS Wa(03) host-4041 <hvc> Found empty StopSignal parameter in config file. Defaulting to SIGTERM YYYY-MM-DD HH:MM:SS In(05) host-4041 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata hvc,UNHEALTHY,UNKNOWN,0 YYYY-MM-DD HH:MM:SS Wa(03) host-4041 <hvc> Service exited. Exit code 143 YYYY-MM-DD HH:MM:SS Er(02) host-4041 Service batch op START failed. Failed services: 'hvc'

Environment

vCenter Server 7.0.3

Cause

The SyncUsers Role ID is incorrect or the SyncUsers Role is missing from the vCenter Server.

Resolution

1. Check if the hvc service is listed:
  
  /usr/lib/vmware-vmafd/bin/dir-cli service list
2. Confirm with jXplorer that hvc solution user is in the following locations:
  - Builtin-users
  - LicenseServer.Administator
  - ServicePrinciplals
  - Solution Users
  - SystemConfiguration.Administrators
  - SyncUsers
3. If the hvc service is not assigned to the SyncUsers role or there are no attributes assigned to the SyncUsers Role, proceed with the following:
- - Check SyncUsers role cn=1002 in vpxd-svcs/authz-ldapdump.ldif
  - Valid Sync Users role:
    
    dn: cn=1002,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local objectClass: top objectClass: vmwAuthzRole cn: 1002 vmwAuthzRolePrivilegeId: System.Anonymous vmwAuthzRolePrivilegeId: System.Read vmwAuthzRolePrivilegeId: System.View vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditTag vmwAuthzRolePrivilegeId: InventoryService.Tagging.AttachTag vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateCategory vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForCategory vmwAuthzRolePrivilegeId: HLM.Manage vmwAuthzRolePrivilegeId: IntercomNamespace.Read vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateTag vmwAuthzRolePrivilegeId: IntercomNamespace.Write vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteTag vmwAuthzRolePrivilegeId: SettingsStore.Manage vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditCategory vmwAuthzRolePrivilegeId: CertificateManagement.Manage vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteCategory vmwAuthzRolePrivilegeId: Trust.Manage vmwAuthzRolePrivilegeId: HLM.Create vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForTag vmwAuthzRoleVersion: 6 vmwAuthzRoleName: SyncUsers vmwAuthzRoleDescription: This role entitles you to perform operations required for sync
Note: The SyncUsers role may be present but the cn=1002 required for hvc may be incorrect.

WARNING: Please take offline snapshots of all vCenters in ELM prior to attempting the remediation steps.

Issue 1
- The SyncUsers Role is present but the incorrect cn is assigned
- Since the hvc service looks for the role with an ID of 1002 it will fail to find the role.
Resolution
1. Remove the existing SyncUsers Role with the wrong ID and recreate it with the correct ID using these commands:
  
  /opt/likewise/bin/ldapdelete -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W "cn=<incorrect_cn_id_present>,cn=RoleModel,cn=VmwAuthz,cn=services,dc=vsphere,dc=local"
  
  Note: If deleting the role using the CLI doesn't work. Connect to jXplorer Services > VmwAuthz > RoleModel delete the incorrect role ID for the hvc service.
2. Add the correct role with the correct ID:
Issue 2
- The SyncUsers role doesn't exist
Resolution
1. Add the SyncUsers Role by running the following command:
```
/opt/likewise/bin/ldapadd -x -D cn=Administrator,cn=Users,dc=vsphere,dc=local -W <<EOF
version: 1
dn: cn=1002,cn=RoleModel,cn=VmwAuthz,cn=Services,dc=vsphere,dc=local
objectClass: vmwAuthzRole
objectClass: top
cn: 1002
vmwAuthzRoleDescription: This role entitles you to perform operations required for sync
vmwAuthzRoleName: SyncUsers
vmwAuthzRolePrivilegeId: System.Anonymous
vmwAuthzRolePrivilegeId: System.Read
vmwAuthzRolePrivilegeId: System.View
vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditTag
vmwAuthzRolePrivilegeId: InventoryService.Tagging.AttachTag
vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateCategory
vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForCategory
vmwAuthzRolePrivilegeId: HLM.Manage
vmwAuthzRolePrivilegeId: IntercomNamespace.Read
vmwAuthzRolePrivilegeId: InventoryService.Tagging.CreateTag
vmwAuthzRolePrivilegeId: IntercomNamespace.Write
vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteTag
vmwAuthzRolePrivilegeId: SettingsStore.Manage
vmwAuthzRolePrivilegeId: InventoryService.Tagging.EditCategory
vmwAuthzRolePrivilegeId: CertificateManagement.Manage
vmwAuthzRolePrivilegeId: InventoryService.Tagging.DeleteCategory
vmwAuthzRolePrivilegeId: Trust.Manage
vmwAuthzRolePrivilegeId: HLM.Create
vmwAuthzRolePrivilegeId: InventoryService.Tagging.ModifyUsedByForTag
vmwAuthzRoleVersion: 7
EOF
```
2. Provide the SSO Administrator/LDAP password when prompted
3. Restart all vCenter services

Additional Information

Impact/Risks

Unable to proceed with patching vCenter
All services will not start after replacing SSL certificates

HVC service fails to start causing 'Exception occurred in PostInstallHook' on patching vCenter or SSL Certificate Install Failing VCSA 7.x

Article ID: 322292

Updated On:

Products

Issue/Introduction

Environment

Cause

Resolution

WARNING: Please take offline snapshots of all vCenters in ELM prior to attempting the remediation steps.

Issue 1

Resolution

Issue 2

Resolution

Additional Information

Feedback