HybridVC (HVC) Service Startup Failure due to cis.componentStatus.ErrorInfo Object at 0x7f1dc9006650

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

Upon restarting the services, the vmware-hvc service fails to start. The following errors are observed:

 

• From /var/log/firstboot/hvc_firstboot.py_XXXX_stderr.log.
  

  YYYY-MM-DDThh:mm:ss HTTP Error 412: Precondition Failed
  YYYY-MM-DDThh:mm:ss ERROR starting hvc rc: 1, stdout: , stderr: Start service request failed. Error: Operation timed out
  Traceback (most recent call last):
    File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 217, in Main
      hvcsvc_fb.start_hvcsvc()
    File "/usr/lib/vmware-hvc/firstboot/hvc_firstboot.py", line 54, in start_hvcsvc
      self.start_service()
    File "/usr/lib/vmware/site-packages/cis/firstboot.py", line 266, in start_service
      service_start(self.get_eff_service_name())
    File "/usr/lib/vmware/site-packages/cis/utils.py", line 1087, in service_start
      raise ServiceStartException(svc_name)
  cis.exceptions.ServiceStartException: {
      "detail": [
          {
              "id": "install.ciscommon.service.failstart",
              "translatable": "An error occurred while starting service '%(0)s'",
              "args": [
                  "hvc"
              ],
              "localized": "An error occurred while starting service 'hvc'"
          }
      ],
      "componentKey": null,
      "problemId": null,
      "resolution": null
  }
  YYYY-MM-DDThh:mm:ss HybridVC Service firstboot failed due to <cis.componentStatus.ErrorInfo object at 0x7f1dc9006650>

• From /var/log/vmware/hvc/hvc-svcs.log, the following error is reported: 

  YYYY-MM-DDThh:mm:ss [main  ERROR com.vmware.sync.interceptors.AuthnUtils  opId=] Failed to create AuthZ connection
  com.vmware.vapi.std.errors.InternalServerError: InternalServerError (com.vmware.vapi.std.errors.internal_server_error) => {
      messages = [LocalizableMessage (com.vmware.vapi.std.localizable_message) => {
      id = vapi.bindings.method.impl.unexpected,
      defaultMessage = Provider method implementation threw unexpected exception: null,
      args = [null],
      params = <null>,
      localized = <null>
  }],
      data = <null>,
      errorType = INTERNAL_SERVER_ERROR
  }
   at com.vmware.vapi.std.errors.InternalServerError._newInstance2(InternalServerError.java:183)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

• From /var/log/vmware/vapi/endpoint/endpoint.log, further errors can be seen:

  YYYY-MM-DDThh:mm:ss  | WARN  | state-manager1            | ApiInterfacesFactory           | Retrieving interfaces for service XXXXXX-XXXX-XXXX-XXXXX-XXXXX\com.vmware.vcenter.hvc.vapi has failed.
  com.vmware.vapi.client.exception.ConnectionException: http://localhost:10090/hvc invocation failed with "java.net.ConnectException: Connection refused"
   at com.vmware.vapi.internal.protocol.client.rpc.http.ApacheHttpAsyncClientTransport$FutureCallbackImpl.failed(ApacheHttpAsyncClientTransport.java:210)
   at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137)
   at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101)
   at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426)
   at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.connectionRequestFailed(AbstractClientExchangeHandler.java:348)

 

Cause

This issue is caused by an improper snapshot revert for a vCenter in Linked Mode, leading to inconsistent service registration.

Resolution

Important: If the affected vCenter is standalone, take a valid snapshot of the vCenter Server. If the affected vCenter is in Enhanced Linked Mode (ELM), take a power-off snapshot of all the vCenters.

1. SSH into the vCenter Server Appliance (VCSA):

   • Use root credentials to SSH into the VCSA.

2. Download the lsdoctor Tool:

   • Download the lsdoctor tool from the VMware Knowledge Base article Using the 'lsdoctor' Tool
   • Follow the instructions provided in the KB to unpack the lsdoctor tool.

3. Upload the lsdoctor Tool to VCSA:

   • Upload the lsdoctor tool to the /tmp directory of the VCSA with Embedded PSC.
   • You may use WinSCP for this. If you encounter the error Received too large (1433299822 B) SFTP packet, refer to the KB for instructions on resolving this issue.Connecting to vCenter Server Virtual Appliance using WinSCP fails with the error: Received too large (1433299822 B) SFTP packet. Max supported packet size is 1024000 B

4. Rebuild All Services using lsdoctor:

   • Run the following commands to navigate to the lsdoctor directory and run the tool:
      

     cd /tmp
cd lsdoctor
python lsdoctor -r
   • The following option will appear:
     • Option #2: "Replace all services with new services."
     • Choose this option to replace all the affected services.

5. Restart All vCenter Services:

   • Restart all services on the vCenter server by running the following commands:
      

     service-control --stop --all && service-control --start --all

After following these steps, the HybridVC (HVC) service should start successfully.