vCenter Server 5.5 fails to start after system reboot with the vpxd.log error: com.vmware.identity.interop.ldap.ServerDownLdapException

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

vCenter Server 5.5 fails to start after system reboot.
vCenter Server is under high load
In the C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpxd.log file, you see entries similar to:

YYYY-MM-DDT<time>Z [06096 error '[SSO][SsoCertificateManagerImpl]'] [RetryOnConnectionFailure] TimeoutException while trying to connect to SSO Admin server: class Vmacore::TimeoutException(Operation timed out)
YYYY-MM-DDT<time>Z [06096 warning 'Default'] Closing Response processing in unexpected state: 3
YYYY-MM-DDT<time>Z [06096 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: Invalid response code: 404 Not Found.
YYYY-MM-DDT<time>Z [06096 warning 'VpxProfiler'] Vpxd::ServerApp::Init [Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)] took 581059 ms
YYYY-MM-DDT<time>Z [06096 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
--> Backtrace:
--> backtrace[00] rip 000000018018b7fa
--> backtrace[01] rip 0000000180104c78
--> backtrace[02] rip 0000000180105f6e
--> backtrace[03] rip 000000018008ff08
--> backtrace[04] rip 0000000000675bac
--> backtrace[05] rip 0000000000696722
--> backtrace[06] rip 00007ff6b0fb550a
--> backtrace[07] rip 00007ff6b0faf06c
--> backtrace[08] rip 00007ff6b11e7c9b
--> backtrace[09] rip 00007ff8934d7f2a
--> backtrace[10] rip 00007ff893c913d2
--> backtrace[11] rip 00007ff893fc03c4
-->
YYYY-MM-DDT<time>Z [06096 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 586353 ms
YYYY-MM-DDT<time>Z [06096 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
</time></time></time></time></time></time></time>
In the C:\Program Files\VMware\Infrastructure\SSOServer\logs\ssoAdminServer.log file, you see entries similar to:

[YYYY-MM-DD <time> localhost-startStop-1 ERROR org.springframework.web.context.ContextLoader] Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfo' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Cannot create inner bean 'ssoServerAboutInfoFactory$created#41870e99' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAboutInfoFactory$created#41870e99' defined in ServletContext resource [/WEB-INF/sso-common-context.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public com.vmware.vim.sso.AboutInfo com.vmware.identity.admin.server.impl.AboutInfoFactory.createAboutInfo()] threw exception; nested exception is com.vmware.identity.admin.server.ims.AdminServerException: Failed to connect to IDM after [20] attempts
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:230)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:117)
at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:474)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:342)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:901)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:815)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)</time>
In the C:\Program Files\VMware\Infrastructure\SSOServer\logs\vmware-sts-idmd.log, you see entries similar to:

YYYY-MM-DD <time> WARN [LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.WinLdapClientLibrary, error code: 81
YYYY-MM-DD <time> ERROR [WinLdapClientLibrary] Failed ldap_bind_s().com.vmware.identity.interop.ldap.ServerDownLdapException: Can't contact LDAP server LDAP error [code: 81]
at com.vmware.identity.interop.ldap.LdapErrorChecker$43.RaiseLdapError(LdapErrorChecker.java:599)
at com.vmware.identity.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:826)
at com.vmware.identity.interop.ldap.WinLdapClientLibrary.CheckError(WinLdapClientLibrary.java:758)
at com.vmware.identity.interop.ldap.WinLdapClientLibrary.ldap_bind_s(WinLdapClientLibrary.java:247)
at com.vmware.identity.interop.ldap.LdapConnection.bindConnection(LdapConnection.java:132)
at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:306)
at com.vmware.identity.idm.server.ServerUtils.getLdapConnectionByURIs(ServerUtils.java:217)
at com.vmware.identity.idm.server.config.directory.DirectoryConfigStore.getConnection(DirectoryConfigStore.java:4041)
at com.vmware.identity.idm.server.config.directory.DirectoryConfigStore.getTenantAttributes(DirectoryConfigStore.java:1103)
at com.vmware.identity.idm.server.IdentityManager.buildTenantInformation(IdentityManager.java:6030)
at com.vmware.identity.idm.server.IdentityManager.loadTenant(IdentityManager.java:6435)
at com.vmware.identity.idm.server.IdentityManager.initializeTenantCache(IdentityManager.java:6282)
at com.vmware.identity.idm.server.IdentityManager.<init>(IdentityManager.java:270)
at com.vmware.identity.idm.server.IdmServer.startserver(IdmServer.java:67)
</time></time>
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server 5.5.x

Cause

This issue occurs because the VMware Identity Manager fails to contact VMware Directory Service. This is because, the high system load prevents the services to start in a reasonable time.

Resolution

To resolve the issue, determine and correct the cause of the high system load.

To work around this issue, manually restart the vCenter Server services in this order:

Stop the VMware Inventory Service.
Stop the VMware Secure Token Service.
Restart the VMware Identity Manager Service.
Start the VMware Secure Token Service.
Start the VMware Inventory Service.
Start the VMware VirtualCenter Server.

For more information on restarting vCenter Server services, see Stopping, starting, or restarting VMware vCenter Server services (1003895).

Additional Information

How to stop, start, or restart vCenter Server services
vCenter Server 5.5 fails to start after reboot with the error: Unable to create SSO facade: Invalid response code: 404 Not Found
vCenter Server 5.5 がシステム再起動後に次の vpxd.log エラーで起動に失敗する：com.vmware.identity.interop.ldap.ServerDownLdapException
vCenter Server 5.5 在系统重新引导后启动失败，并显示 vpxd.log 错误：com.vmware.identity.interop.ldap.ServerDownLdapException