Lookupservice may fail to start automatically after reboot in vCenter Server 7.0 after upgrade or update, manual start works

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter was upgraded to 7.0 or updated to a newer 7.0 version
• after rebooting the vCenter, the Lookupservice fails to start automatically
• manually starting Lookupservice using vmon-cli or the service-control command (e.g. service-control --stop --all && service-control --start --all) works fine without any errors
• /var/log/vmware/lookupsvc/lookupserver-default.log contains multiple instances of the following error:

  [YYYY-MM-DDTHH:MM:SS.429Z localhost-startStop-1                                                           ERROR com.vmware.vim.lookup.impl.LdapStorage] LDAP action failed; host=<vcenter_fqdn>, port=389
  com.vmware.sso.interop.ldap.LdapException: Can't contact LDAP server
          at com.vmware.sso.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:869) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
          at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1219) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
          at com.vmware.sso.interop.ldap.OpenLdapClientLibrary.ldap_sasl_srp_bind_s(OpenLdapClientLibrary.java:765) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
          at com.vmware.sso.interop.ldap.LdapConnection.bindSaslSrpConnection(LdapConnection.java:670) ~[ldap-lib-0.0.1-SNAPSHOT.jar:?]
          at com.vmware.vim.lookup.impl.LdapStorage$LdapQuery.execute(LdapStorage.java:2283) ~[lookupservice-impl-0.0.1-SNAPSHOT.jar:?]
          (etc.)

Environment

VMware vCenter Server 8.0.2
VMware vCenter Server 7.0.3
VMware vCenter Server 7.0.x

Cause

Lookupservice depends on the vmdird service, but no start dependency is configured. In situations where the vmdird start takes longer, this can result in vmdird not be ready when Looupservice is starting, resulting in Lookupservice start failing

Resolution

This issue is resolved in vCenter Server 7.0 U3o and vCenter Server 8.0 U2.

Workaround:

As a workaround, you can configure a start dependency for the Lookupservice on vmdird, as outlined below.

Note: Please ensure you have created a fresh backup or an offline snapshot of the vCenter Server, before applying these steps:

1. Open an SSH connection to the vCenter Server Appliance and login with the root user
2. Create a backup copy of the file /etc/systemd/system/multi-user.target.wants/vmware-vmon.service in the /tmp directory using the following command:

   # cp /etc/systemd/system/multi-user.target.wants/vmware-vmon.service /tmp/vmware-vmon.service.bak

IMPORTANT: Do not put the backup file within the /etc/systemd/system/ directory. This will cause stability issues in future updates.

3. Edit this file using the vi editor:

   # vi /etc/systemd/system/multi-user.target.wants/vmware-vmon.service

4. In the [Unit] section of the file, find the entry starting with 'After=' and replace the dependency from vmafdd.service with vmdird.service.

For example, if the configuration is:

After=network.target vmafdd.service vmware-stsd.service rsyslogd.service

you can replace it with:

After=network.target vmdird.service vmware-stsd.service rsyslogd.service

NOTE: the other entries in the line may vary depending on the specific vCenter Version. There is no need to change them as well, just leave them as they are.

 

1. Update the system configuration using the following command:

   # systemctl daemon-reload

2. To verify that the workaround fixed the problem, reboot your vCenter Server Appliance once.