lookupsvc or vpxd-svcs fails to start with error | Service crashed while starting

Products

VMware vCenter Server

Issue/Introduction

Symptoms:
While starting vCenter services, lookupsvc or vpxd-svcs fails to start.

/var/log/vmware/lookupsvc/lookupserver-default.log :

[YYYY-MM-DDTHH:MM:SS pool-2-thread-4                                                           WARN  com.vmware.vim.lookup.impl.LdapStorage] Failed search with base DN 'cn=ServiceRegistrations,cn=LookupService,cn=Granite,cn=Sites,cn=Configuration,dc=vsphere,dc=local'
com.vmware.sso.interop.ldap.NoSuchObjectLdapException: No such object
        at com.vmware.sso.interop.ldap.LdapErrorChecker$22.RaiseLdapError(LdapErrorChecker.java:336) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
        at com.vmware.sso.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:863) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]     
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_351]
        at java.lang.Thread.run(Thread.java:750) [?:1.8.0_351]
[YYYY-MM-DDTHH:MM:SS pool-2-thread-4                                                           WARN  com.vmware.vim.lookup.impl.LdapStorage] Empty list returned with filter: com.vmware.vim.lookup.ServiceRegistrationTypes$Filter@1332c25e[siteId=granite,nodeId=<null>,serviceProduct=com.vmware.cis,serviceType=cs.identity,endpointType=<null>,endpointProtocol=<null>,endpointTrustAnchor=<null>]
[YYYY-MM-DDTHH:MM:SS pool-2-thread-4                                                           ERROR com.vmware.vim.lookup.vlsi.util.VmodlEnhancer] SSO service record not found
java.lang.IllegalStateException: SSO service record not found

[YYYY-MM-DDTHH:MM:SS localhost-startStop-1                                                           ERROR com.vmware.sync.subscribe.SusbscribeDbStore] SusbscribeDbStore intialized with dbUserName: lookupsvc_sync_db
[YYYY-MM-DDTHH:MM:SS localhost-startStop-1                                                           ERROR com.vmware.vim.lookup.impl.LdapStorage] LDAP action failed; host=XX.XX.XX.XX, port=389
com.vmware.sso.interop.ldap.NoSuchObjectLdapException: No such object
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.lookup.impl.ServiceRegistrationImpl]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: No such object
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceRegistration' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.vim.lookup.impl.ServiceRegistrationImpl]: Constructor threw exception; nested exception is com.vmware.vim.lookup.exception.StorageException: No such object

/var/log/vmware/vpxd-svcs/vpxd-svcs.log :

[YYYY-MM-DDTHH:MM:SS] [cache-sync-task [] WARN  com.vmware.identity.interop.ldap.LdapErrorChecker  opId=] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: -1
[YYYY-MM-DDTHH:MM:SS] [cache-sync-task [] ERROR com.vmware.identity.interop.ldap.OpenLdapClientLibrary  opId=] Exception when calling ldap_search_s: base=, scope=0, filter=(objectClass=*), attrs=[objectGUID, null], attrsonly=1
com.vmware.identity.interop.ldap.ServerDownLdapException: Can't contact LDAP server
[YYYY-MM-DDTHH:MM:SS] [cache-sync-task [] WARN  com.vmware.cis.core.tagging.internal.impl.SyncManagerImpl  opId=] Unable to get object guid of ldap root DSE entry
[YYYY-MM-DDTHH:MM:SS] [cache-sync-task [] ERROR com.vmware.cis.core.tagging.internal.impl.SyncManagerImpl  opId=] Failed to get deleted objects from Lotus:
[YYYY-MM-DDTHH:MM:SS] [tomcat-exec-52 [] INFO  com.vmware.vim.vcauthenticate.servlets.AuthenticationServlet  opId=] Sending security error because of: com.vmware.vim.vcauthenticate.exception.NotAuthenticatedException Msg: null
[YYYY-MM-DDTHH:MM:SS] [pool-14-thread-1 [] WARN  com.vmware.cis.lotus.LdapUtils  opId=] Error on ldapSearch:

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This happens when LookupService container is missing from the vmdird DB.

Resolution

To resolve this issue, first we need to confirm if the LookupService container is missing. SSH to the vCenter and run the below command.

Change the sitename/sso domain and password as per customer environment.

ldapsearch -x -h localhost -b "cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local" -s sub "(objectClass=container)" -D "cn=Administrator,cn=Users,dc=vsphere,dc=local" -w 'SSOPWD' dn

Expected Output :

# extended LDIF
#
# LDAPv3
# base <cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local> with scope subtree
# filter: (objectClass=container)
# requesting: dn
#

# Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# Servers, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Servers,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# Replication Agreements, XX.XX.XX.XX, Servers, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Replication Agreements,cn=XX.XX.XX.XX,cn=Servers,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# LookupService, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=LookupService,cn=Default-First-Site, cn=Sites,cn=Configuration,dc=vsphere,dc=local

# Configuration, LookupService, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Configuration,cn=LookupService,cn=Default-First-Site, cn=Sites,cn=Configuration,dc=vsphere,dc=local

# ServiceRegistrations, LookupService, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=ServiceRegistrations,cn=LookupService,cn=Default-First-Site, cn=Sites,cn=Configuration,dc=vsphere,dc=local

# search result
search: 2
result: 0 Success

# numResponses: 7
# numEntries: 6

If missing, the output will be like below :

# extended LDIF
#
# LDAPv3
# base <cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local> with scope subtree
# filter: (objectClass=container)
# requesting: dn
#
# Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# Servers, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Servers,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# Replication Agreements, 10.0.0.1, Servers, Default-First-Site, Sites, Configuration, vsphere.local
dn: cn=Replication Agreements,cn=10.0.0.1,cn=Servers,cn=Default-First-Site,cn=Sites,cn=Configuration,dc=vsphere,dc=local

# search result
search: 2
result: 0 Success

# numResponses: 4
# numEntries: 3

Workaround:

To recreate the missing Container, copy the attached script to vCenter Server Appliance using WinSCP or create the file on VCSA using the contents on to /var/tmp folder.

Run : python recreatecontainer.py

This script will automatically recreate the missing containers.

After the script runs successfully, stop and start the services.
Few services will fail to start since Service Registrations are missing. So take a duplicate session to vCenter and use lsdoc tool to recreate the service registrations.

LSDOCTOR Script : Using the 'lsdoctor' Tool

python lsdoctor.py -r and select option 2 : Replace all services with new services.

Once done, restart all services.

Additional Information

TSE Notes : 36018727

After replacing the machine SSL certificates, the vpxd-svcs service failed to start. In the vpxd-svcs logs, the error message observed was:

2024-12-19T01:49:09.296Z Wa(03) host-140208 <vpxd-svcs> Service pre-start command's stderr: pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 dynamicType = <unset>,
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 dynamicProperty = (vmodl.DynamicProperty) [],
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 msg = 'Internal server error',
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 faultCause = <unset>,
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 faultMessage = (vmodl.LocalizableMessage) [],
2024-12-19T01:49:09.296Z Wa(03)+ host-140208 reason = 'SSO service record not found'

Attempts were made to rebuild the services using the lsdoctor -r command, but this failed with the following error messages.

2024-12-19T01:55:09 ERROR unregister_service: Failed to unregister service 57cbb4e8-9637-4b35-91a5-43279fc406ed, esclate the error
2024-12-19T01:55:09 ERROR unregisterPnid: Failed to unregister service 57cbb4e8-9637-4b35-91a5-43279fc406ed.
2024-12-19T01:55:09 WARNING unregister_service: Failed to unregister_service [9b17bdb8-a6f6-4640-bb56-76c31999434c]: (vmodl.fault.SystemError)

Resolution:

To resolve the issue, the cs.identity and legacy SSO endpoints were reinstalled using the attached scripts in this KB 343793. After completing this step, we were successfully able to rebuild the services.

Attachments

recreatecontainer.py get_app