vCenter services vapi-endpoint and vpxd-svcs fail to start with "Unexpected status code: 404"

Products

• VMware vCenter Server

Issue/Introduction

Impact/Risks:
Warning: Before running the attached script, ensure proper snapshots have been taken of the SSO domain. This means all vCenters or PSCs that are in the SSO domain must be shutdown at the same time, then perform a snapshot of all machines, and power them on again. Should it be required to revert to one of these snapshots, shut down all nodes, and revert all nodes to the snapshot. Failure to perform these steps will lead to replication problems across the PSC databases. For more information, refer to KB VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice.

The purpose of the KB article is to update the vmwSTSConnectionStrings attribute in the vmdir database.

• vCenter service fails to start multiple services such as vapi-endpoint, vpxd-svcs
• vCenter - /var/log/vmware/vapi/endpoint/endpoint.log shows similar to below messages 

[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Failed to retrieve SSO settings from Lookup Service.
[YYYY-MM-DDTHH:MM:SS]  | ERROR | state-manager1            | DefaultStateManager            | Could not initialize endpoint runtime state.
com.vmware.vapi.endpoint.config.ConfigurationException: Failed to retrieve SSO settings.
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:48)
        at com.vmware.vapi.state.impl.DefaultStateManager.build(DefaultStateManager.java:353)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.doInitialConfig(DefaultStateManager.java:167)
        at com.vmware.vapi.state.impl.DefaultStateManager$1.run(DefaultStateManager.java:150)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.vmware.vapi.endpoint.config.ConfigurationException: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:301)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookupSso(LookupServiceClientWrapper.java:211)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.ssoSettings(SsoSettingsBuilder.java:107)
        at com.vmware.vapi.endpoint.cis.SsoSettingsBuilder.buildInitial(SsoSettingsBuilder.java:41)
        ... 10 more
Caused by: com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 404
        at com.vmware.vim.vmomi.client.common.Response$Status.getStatus(Response.java:58)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:156)
        at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:53)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
        at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:106)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:629)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:610)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:360)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:311)
        at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:184)
        at com.sun.proxy.$Proxy38.list(Unknown Source)
        at com.vmware.vapi.endpoint.cis.ls.LookupServiceClientWrapper.lookup(LookupServiceClientWrapper.java:297)
        ... 13 more
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH ORANGE Application error has occurred. Please check log files for more information.
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthStatusCollectorImpl      | HEALTH GREEN Current vApi Endpoint health status is created between [YYYY-MM-DDTHH:MM:SS] and [YYYY-MM-DDTHH:MM:SS].
[YYYY-MM-DDTHH:MM:SS] | INFO  | state-manager1            | HealthConfigurationEventListener | Computed health status is = ORANGE    

vSphere Diagnostic Tool reports:

   IDENTITY SOURCE CHECKS   

              [FAIL]    STS connection string is incorrect (ldap://my.vcenter.com:389)
                    Note:     This could prevent services from starting after a recent decommission of another vCenter
                    Documentation:     https://knowledge.broadcom.com/external/article?articleNumber=323195

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

The vmwSTSConnectionStrings attribute is set at the installation of a vCenter Server, but may not be changed back to the correct value of 'ldap://localhost:389'.  The issue will cause a problem after the incorrectly referenced node is shut down or decommissioned.

Resolution

To correct the value:

1. Confirm the  vmwSTSConnectionStrings value is invalid by running the command below (Note: This assumes vsphere.local.  Change the domain name to match)

   ldapsearch -o ldif-wrap=no -LLL -H ldap://localhost -x -W -D "cn=administrator,cn=users,dc=vsphere,dc=local" -b "cn=vsphere.local,cn=IdentityProviders,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local" -s sub "(&(objectclass=vmwSTSIdentityStore)(vmwSTSProviderType=IDENTITY_STORE_TYPE_VMWARE_DIRECTORY))" vmwSTSConnectionStrings

2. Once confirmed, download the attached script 'fix_sts_attrs.py'.
      
3. Using WinSCP or any similar file transfer application, upload the script to any vCenter server in ELM (it is recommended to place the script in /tmp). 
   
   
4. Navigate into the directory where the script is located:
   

   cd /<location of script>

5. Run the Python script:
   

   python fix_sts_attrs.py

   root@vcenter [ ~ ]# python fix_sts_attrs.py
   This tool checks for issues with the built in identity source attributes and fixes them.

       WARNING!  This script makes permanent changes.
       Please ensure you have a backup or supported snapshot of this vCenter and all other
       vCenters in the ELM group (see VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice).

       Would you like to continue?[Yy|Nn]: Y

   Provide password for [email protected]:

   [WARNING] Detected that the connection string (ldap://my.vcenter.com:389) is incorrect.  Setting to ldap://localhost:389

   [DONE] Connection string is now: ldap://localhost:389

6. Restart all vCenter services. For more information on service restart, refer to Stopping, Starting or Restarting VMware vCenter Server Appliance services.
   

   service-control --stop --all && service-control --start --all