Lookupsvc fails to start and the vSphere client displays the error 'No healthy upstream'

Products

• VMware vCenter Server

Issue/Introduction

• Lookupsvc service fails to start after a vCenter Server reboot.
• lookupserver-default.log located at /var/log/vmware/lookupsvc/ shows the following log snippets: 

YYYY-MM-DDTHH:MM Thread-3   ERROR com.vmware.sync.GrpcUtils] *** shutting down gRPC server since JVM is shutting down
YYYY-MM-DDTHH:MM Thread-3   ERROR com.vmware.sync.GrpcUtils] *** server shut down

• vmon.log located at /var/log/vmware/vmon/ reports OSError while trying to use "/opt/likewise/bin/ldapsearch" executable file, and soon the lookupsvc fails to start with an exit code 143.

YYYY-MM-DDTHH:MM Wa(03) host-2256 <lookupsvc> Service api-health command's stderr: self._posix_spawn(args, executable, env, restore_signals,
YYYY-MM-DDTHH:MM Wa(03)+ host-2256  File "/usr/lib/python3.10/subprocess.py", line 1675, in _posix_spawn
YYYY-MM-DDTHH:MM Wa(03) host-2256 <lookupsvc> Service api-health command's stderr:     self.pid = os.posix_spawn(executable, args, env, **kwargs)
YYYY-MM-DDTHH:MM Wa(03)+ host-2256 OSError: [Errno 8] Exec format error: '/opt/likewise/bin/ldapsearch'
YYYY-MM-DDTHH:MM Wa(03) host-2256 <lookupsvc> Service api healthcheck command returned unknown exit code 1
YYYY-MM-DDTHH:MM In(05) host-2256 <lookupsvc> Re-check service health since it is still initializing.
YYYY-MM-DDTHH:MM In(05) host-2256 <lookupsvc> Service start operation timed out.
YYYY-MM-DDTHH:MM Wa(03) host-2256 <lookupsvc> Found empty StopSignal parameter in config file. Defaulting to SIGTERM
YYYY-MM-DDTHH:MM In(05) host-2256 <event-pub> Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonEventPublisher.py --eventdata lookupsvc,UNHEALTHY,UNKNOWN,0
YYYY-MM-DDTHH:MM Wa(03) host-2256 <lookupsvc> Service exited. Exit code 143

• The ldapsearch executable file is in an invalid state with the file size of 0 bytes and an invalid modified date.
    root@<hostname> [ /opt/likewise/bin ]# ls -lath
  -rwxr-xr-x 1  root root  0  MM DD 06:49 ldapsearch
  
  
• ldapsearch executable file should have a valid size and modified date as seen in the below example.
    root@<hostname> [ /opt/likewise/bin ]# ls -lath
  -rwxr-xr-x 1  root root  264k  MM DD YYYY ldapsearch

Environment

vCenter 8.x

Cause

The lookup service (lookupsvc) failed to start as it could not perform LDAP search queries using "/opt/likewise/bin/ldapsearch" executable file.

Resolution

    1. Connect to vCenter Server Appliance shell via SSH.
    2. Back up the ldapsearch executable file from the /opt/likewise/bin/ directory on the affected vCenter.
            cp /opt/likewise/bin/ldapsearch /storage/core/ldapsearch.bak 

    3. Copy the ldapsearch file located in the directory /opt/likewise/bin/ from a working vCenter Server of the same build to the corresponding directory on the affected vCenter Server, overwriting the existing file.
    4. If no other vCenter Server Appliance instances are present in the environment, deploy a new vCenter Server Appliance using the same build number, and then proceed with step 3.
    5. Provide the required permissions to the ldapsearch file to make it executable by running the command:
           chmod 755 /opt/likewise/bin/ldapsearch

    6. Restart all vCenter services by running the command:
           service-control --stop --all && service-control --start --all