Unable to start vCenter service.Failed to start sca, vpxd, vpxd-svcs, vapi-endpoint services. Error: A system error occurred

Products

VMware vCenter Server VMware vCenter Server 7.0

Issue/Introduction

Unable to access the vCenter UI and while starting all the services it fails to start.

service-control --start --all 

Performing start operation on service vmdird...
Successfully started service vmdird 
Performing start operation on service vmcad... 
Successfully started service vmcad 
Performing start operation on service vmware-sts-idmd... 
Successfully started service vmware-sts-idmd 
Performing start operation on service vmware-stsd... 
Successfully started service vmware-stsd 
Performing start operation on service vmdnsd... 
Successfully started service vmdnsd 
Performing start operation on profile: ALL...
Successfully started service vmware-vmon 
Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start sca, vpxd, vpxd-svcs, vapi-endpoint services. Error: A system error occurred. Check logs for details

From /var/log/vmware/vapi/endpoint/endpoint.log, we see below mentioned error

    YYYY-MM-DDTHH:MM:SS.955Z | INFO  | state-manager1            | DefaultStateManager            | Invoking configuration-utilities
    YYYY-MM-DDTHH:MM:SS.234Z | INFO  | state-manager1            | DefaultStateManager            | Invoking http-server   
    YYYY-MM-DDTHH:MM:SS.235Z | INFO  | state-manager1            | BaseServerBuilder              | Creating endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
    YYYY-MM-DDTHH:MM:SS.345Z | INFO  | state-manager1            | DefaultJettyServer             | Logging initialized @13986ms to com.vmware.vapi.endpoint.http.DefaultJettyServer$JettyLogWrapper
    YYYY-MM-DDTHH:MM:SS.729Z | WARN  | state-manager1            | BaseServerBuilder              | Failed to bind /0:0:0:0:0:0:0:1:12346 while testing the endpoint validity
     java.net.SocketException: Protocol family unavailable
        at java.net.PlainSocketImpl.socketBind(Native Method)
        at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513)
        at java.net.Socket.bind(Socket.java:661)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.isPortAccessible(BaseServerBuilder.java:172)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.trimInvalidEndpoints(BaseServerBuilder.java:148)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.populateEndpointSettings(BaseServerBuilder.java:184)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.createServer(BaseServerBuilder.java:233)
        at com.vmware.vapi.endpoint.http.BaseServerBuilder.buildInitial(BaseServerBuilder.java:75)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
    YYYY-MM-DDTHH:MM:SS.769Z | WARN  | state-manager1            | BaseServerBuilder              | Hostname ::1 was found to be invalid and removed from the configuration
    YYYY-MM-DDTHH:MM:SS.509Z | INFO  | state-manager1            | BaseServerBuilder              | Starting endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346
    YYYY-MM-DDTHH:MM:SS.786Z | INFO  | state-manager1            | DefaultJettyServer             | Starting jetty server.
    YYYY-MM-DDTHH:MM:SS.261Z | INFO  | state-manager1            | BaseServerBuilder              | Started endpoint with name 'default' on address(es): 127.0.0.1, ::1 with port: 12346.
    YYYY-MM-DDTHH:MM:SS.261Z | INFO  | state-manager1            | DefaultStateManager            | Invoking cis-sso-settings-builder
    YYYY-MM-DDTHH:MM:SS.433Z | INFO  | state-manager1            | CertificateUtil                | Creating anonymous SSO Admin Client for URI https://<IPaddress of VCenter>/sso-adminserver/sdk/vsphere.local
    YYYY-MM-DDTHH:MM:SS.641Z | INFO  | state-manager1            | DefaultStateManager            | Invoking sts-builder
    YYYY-MM-DDTHH:MM:SS.311Z | INFO  | shutdown-hook             | ApiEndpointServer              | Start shutting down...
    YYYY-MM-DDTHH:MM:SS.312Z | INFO  | shutdown-hook             | DefaultStateManager            | shutdown
    YYYY-MM-DDTHH:MM:SS.321Z | INFO  | shutdown-hook             | ApiEndpointServer              | Shutdown.

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 6.7.x

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

Protocol family unavailable error indicates that vCenter is unable to bind to specific IP or specific network family due to inccorrect or missing entry in hosts file.

Resolution

Note: If the affected vCenter is standalone, take a valid snapshot of the vCenter Server. If the affected vCenter is in ELM, take a power-off snapshot of all the vCenters.

SSH into the vCenter
- Use root credentials to SSH into the affected vCenter server.

Check the hosts File

Run the following command to view the current hosts file:
```
cat /etc/hosts
```

Example output:

# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 localhost.localdomain localhost
127.0.0.1 photon-machine
# VAMI EDIT END

Edit the Hosts File
- If the hosts file resembles the example above, it needs modification.
- Use the following command to edit the hosts file:
```
vi /etc/hosts
```
- Enter insert mode by pressing the i key.
- Add the following line at the end of the file, replacing <IP address of the vCenter> with the affected vCenter server’s IP address:
```
<IP address of the vCenter> localhost
```
- After the modification, the file should look like this:
```
# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 localhost.localdomain localhost
127.0.0.1 photon-machine
<IP address of the vCenter> localhost
# VAMI EDIT END
```
Save the Changes
- Press the ESC key to exit insert mode.
- Save and exit the file by typing:
```
:wq!
```
Restart vCenter Services
- Stop and start all vCenter services using the following command:
```
service-control --stop --all && service-control --start --all
```

Additional Information

In some scenarios, the issue can also be caused by incorrect entry in /etc/hosts file.

If the reason to keep the problematic entry cannot be specified or the entry is confirmed not in use, the entry needs to be removed/commented out.

For example, below ipv6 entry can be causing issue because vapi-endpoint service is not able to bind to this ipv6 address to test validity.

# VAMI EDIT BEGIN
# Generated by Studio VAMI service. Do not modify manually.
127.0.0.1 <vcenter name> localhost
::1 <vcenter name> localhost ipv6-localhost ipv6-loopback <---------------------------
# VAMI EDIT END