HVC service fails to start in vCenter version 7.0 due to missing solution user

Products

• VMware vCenter Server
• VMware vCenter Server 7.0

Issue/Introduction

HVC service fails to start and hvc-svcs.log file located in /var/log/vmware/hvc provides the following insights:

YYYY-MM-DDTHH:MM:SS [main WARN com.vmware.vim.sso.client.SecurityTokenSerrviceConfig$ConnectionConfig opId=] This configuration will establish untrusted connection with the STS server.It is acceptable for developing purposes only!
YYYY-MM-DDTHH:MM:SS [main ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] SOAP fault
com.sun.xml.internal.ws.fault.ServerSOAPFaultException: Client received SOAP Fault from server: Invalid credentials Please see the server log to find more detail regarding exact cause of the failure.
    at com.sun.xml.internal.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:178)
    at com.sun.xml.internal.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:116)
    at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.doInvoke(DispatchImpl.java:259)
    at com.sun.xml.internal.ws.client.dispatch.DispatchImpl.invoke(DispatchImpl.java:289)
    at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:208)
    at com.vmware.vim.sso.client.impl.SoapBindingImpl.sendMessage(SoapBindingImpl.java:138)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.sendRequest(SecurityTokenServiceImpl.java:983)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl$RequestResponseProcessor.executeRoundtrip(SecurityTokenServiceImpl.java:902)
    at com.vmware.vim.sso.client.impl.SecurityTokenServiceImpl.acquireTokenByCertificate(SecurityTokenServiceImpl.java:509)
    at com.vmware.sync.interceptors.AuthnUtils.acquireTokenByCertificate(AuthnUtils.java:505)
    at com.vmware.sync.interceptors.AuthnUtils.createVapiAuthzSession(AuthnUtils.java:178)
    at com.vmware.hvc.synccontroller.Controller.createPrivilegeUpdateRole(Controller.java:283)
    at com.vmware.hvc.synccontroller.Controller.init(Controller.java:320)
    at com.vmware.hvc.synccontroller.Controller.<init>(Controller.java:216)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

Cause

To help diagnose the issue, please execute the following command which will display a list of solution users:

• /usr/lib/vmware-vmafd/bin/dir-cli service list
  
  

Please check if the HVC solution user is listed.

Solution users follow a specific format: <solution_user>-<machine_id>.
For example: hvc- ########-####-####-####-#########.

To obtain the machine ID for your vCenter server, run the following command:

• /usr/lib/vmware-vmafd/bin/vmafd-cli get-machine-id --server-name localhost

Comparing this machine ID to the expected format of the HVC solution user will help us determine if there are any inconsistencies.

Resolution

NOTE: Take powered-off snapshots of the vCenter Server Appliance and any vCenter Servers linked to it via ELM.

• To address the missing HVC solution user issue, you can utilize the lsdoctor tool to recreate solution users within the vCenter environment: 
  • https://knowledge.broadcom.com/external/article/320837/using-the-lsdoctor-tool.html
• Below are the steps for the same:
  
  • Copy and extract the lsdoctor tool to the filesystem of the affected vCenter node.
  • Execute the command python lsdoctor.py -u to analyze the system.
  • Verify that you have created appropriate offline snapshots of your vCenter environment before proceeding.
  • Provide the password for your vCenter Single Sign-On (SSO) administrator account when prompted by the script.
  • After the script completes, restart all services on the affected node.