vCenter services vapi-endpoint and vpxd-svcs fail to start with "Unexpected status code: 404"

Products

• VMware vCenter Server

Issue/Introduction

The purpose of the KB article is to update the vmwSTSConnectionStrings attribute in the vmdir database.

Symptoms:

  

• vCenter service fails to start multiple services such as vapi-endpoint, vpxd-svcs
• /var/log/vmware/vapi/endpoint.log shows similar to below messages 

2023-03-17T19:24:51.769Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from Lookup Service.
2023-03-17T19:24:51.770Z | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:301)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:211)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
        ... 10 more
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
        at com.sun.proxy.$Proxy38.list(Unknown Source)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:297)
        ... 13 more
2023-03-17T19:24:51.770Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
2023-03-17T19:24:51.770Z | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between 2023-03-17T19:24:51UTC and 2023-03-17T19:24:51UTC.
2023-03-17T19:24:51.770Z | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE


      

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This attribute is set on install of a vCenter, but may not be changed back to the correct value of 'ldap://localhost:389'.  The issue will cause a problem after the incorrectly referenced node is shutdown or decommissioned.

Resolution

To correct the value:

1. Download the attach script 'fix_sts_attrs.py'.
2. Using WinSCP or the utility of your choice, upload the script to any vCenter server in ELM (it is recommended to place the script in /tmp). 
3. Run the python script by typing:  

cd /<location of script>
python fix_sts_attrs.py

4. Restart all the services (https://kb.vmware.com/s/article/2109887):

service-control --stop --all && service-control --start --all

Additional Information

Impact/Risks:
Warning: Before running the attached script, ensure you have taken proper snapshots of your SSO domain. This means that you must shut down all vCenters or PSCs that are in the SSO domain at the same time, then snapshot them, and power them on again.  If you need to revert to one of these snapshots, shut all the nodes down, and revert all nodes to the snapshot. Failure to perform these steps will lead to replication problems across the PSC databases.  For more information, see https://kb.vmware.com/s/article/85662