vCenter services vapi-endpoint and vpxd-svcs fail to start with "Unexpected status code: 404"

Products

• VMware vCenter Server

Issue/Introduction

Impact/Risks:
Warning: Before running the attached script, ensure proper snapshots have been taken of the SSO domain. This means all vCenters or PSCs that are in the SSO domain must be shutdown at the same time, then perform a snapshot of all machines, and power them on again. Should it be required to revert to one of these snapshots, shut down all nodes, and revert all nodes to the snapshot. Failure to perform these steps will lead to replication problems across the PSC databases. For more information, refer to KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

The purpose of the KB article is to update the vmwSTSConnectionStrings attribute in the vmdir database.

• vCenter service fails to start multiple services such as vapi-endpoint, vpxd-svcs
• vCenter - /var/log/vmware/vapi/endpoint.log shows similar to below messages 

[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from Lookup Service.
[YYYY-MM-DDTHH:MM:SS]  | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:301)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:211)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
        ... 10 more
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
        at com.sun.proxy.$Proxy38.list(Unknown Source)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:297)
        ... 13 more
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between [YYYY-MM-DDTHH:MM:SS] and [YYYY-MM-DDTHH:MM:SS].
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE    

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The vmwSTSConnectionStrings attribute is set at install of a vCenter Server, but may not be changed back to the correct value of 'ldap://localhost:389'.  The issue will cause a problem after the incorrectly referenced node is shutdown or decommissioned.

Resolution

To correct the value:

1. Download the attached script 'fix_sts_attrs.py'.
2. Using WinSCP or similar, upload the script to any vCenter server in ELM (it is recommended to place the script in /tmp). 
3. Change into the directory where the script is located:
   
   cd /<location of script>
   
   
4. Run the python scrip:
   
   python fix_sts_attrs.py

4. Restart all vCenter services. For more information on service restart, refer to Stopping, Starting or Restarting VMware vCenter Server Appliance 6.x & above services.
   
   service-control --stop --all && service-control --start --all

Additional Information