Error: Operation timed out when starting Hybrid vCenter Service (HVC)

Products

• VMware vCenter Server

Issue/Introduction

• The vmware-hvc (Hybrid vCenter Service) fails to start with a timeout error.
• Attempting to start the service manually via service-control --start vmware-hvc fails with install.ciscommon.service.failstart.
  # service-control --start vmware-hvc
Operation not cancellable. Please wait for it to finish...
Performing start operation on service hvc...
Error executing start on service hvc. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "hvc"
            ],
            "localized": "An error occurred while starting service 'hvc'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
• The /var/log/vmware/hvc/hvc-svcs.log contains: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authzFilter'
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authzFilter' defined in class path resource [vlsi-server.xml]: Cannot resolve reference to bean 'authzFilter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: ...
• Listing solution users via /usr/lib/vmware-vmafd/bin/dir-cli service list confirms the hvc solution user is completely missing from the SSO domain
• Running python lsdoctor.py -u fails with the following errors: vecs-cli failed. Error 4312: Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312) dir-cli failed. Error 2: Win Error: Operation failed with error ERROR_FILE_NOT_FOUND (2)
# python lsdoctor.py -u
...
2026-06-18T10:25:01 INFO checkAndFix: Deleting and recreating user hvc
2026-06-18T10:25:01 WARNING delete_service: Failed to delete hvc-########-####-####-#####-#########.  This is because it's already missing.
...
Stderr = vecs-cli failed. Error 4312: Possible errors:
LDAP error: Unknown (extension) error
Win Error: Operation failed with error ERROR_OBJECT_NOT_FOUND (4312)
...
Stderr = dir-cli failed. Error 2: Possible errors:
LDAP error: Protocol error
Win Error: Operation failed with error ERROR_FILE_NOT_FOUND (2)
• Checking the certificate status using vCert utility shows the hvc Solutions User certificate as "NOT FOUND":
  
  

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The hvc solution user is missing from the SSO domain, and the hvc VMware Endpoint Certificate Store (VECS) entry is missing its certificate. Because the store is empty, lsdoctor cannot read a certificate to bind to the directory, causing the recreation to fail.

Resolution

Note: Before using lsdoctor to make any changes, ensure to take offline snapshots of the vCenters in ELM. Shut down all vCenters that are in the SSO domain at the same time, then snapshot them, and power them on again.  If it is required to revert to one of these snapshots, shut all the vCenter servers down, and revert all of them to the snapshot, and then power them on. Failure to perform these steps will lead to replication problems across the PSC databases.

To resolve this issue of hvc service failing to start, renew the solution user certificates to repopulate the VECS store, then use lsdoctorto recreate the solution user.

1. Renew Solution User Certificates:
   • Navigate to the directory containing the vCert utility (e.g., /tmp/vCert-####).
   • Execute the script: python vCert.py
   • Select Option 3 (Manage certificates).
   • Select Option 2 (Solution User certificates) to renew certificates for all solution users.
2. Rebuild HVC Service:
   • Navigate to the lsdoctor directory.
   • Run the update command: python lsdoctor.py -u
   • Verify the hvc user is now successfully recreated.
   • The script deletes and rebuilds the primary solution user accounts. After executing the script renew the Solution User certificates again, following step 1, to fix the certificate mismatch for other solution users. As part of the certificate renewal, restart the vCenter services. Verify hvc service starts successfully.

Additional Information

For instructions on using lsdoctor, refer: Using the 'lsdoctor' Tool (KB 320837)