Error: "Operation timed out" for Lookup service when restarting services on vCenter Server 7.0

Products

• VMware vCenter Server

Issue/Introduction

• The Lookup service (lookupsvc) fails to start in vCenter Server 7.0.x
• When reviewing /var/log/vmware/lookupsvc/lookupserver-default.log, errors similar to the following example can be seen:

  Cannot resolve reference to bean 'lsSubscribableImpl' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'lsSubscribableImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Invocation of init method failed; nested exception is java.sql.SQLException: unable to connect to database: localhost:5432/VCDB

• The vPostgres (vCenter database) service is running. Verify with the following command:

  # service-control --status vpostgres

• The vPostgres log at /var/log/vmware/vpostgres/postgresql.log contains similar errors as below:

  66a0e291.685c 0    26716 27 LOG:  Updating instance status...
  66a0e291.685c 0    26716 28 LOG:  Checking bloat of WAL caused by replication slots
  66a0e291.685c 0    26716 29 LOG:  Memory check: flag = rss, mem_used_bytes = 7680 kB, mem_avail = 715776 kB
  66a0e291.685c 0    26716 30 LOG:  Writing instance status...
  66a0e291.685c 0    26716 31 LOG:  Wrote instance status successfully.
  66a0e291.685c 0    26716 32 LOG:  Updated instance status successfully.
  66a0e2ed.69d5 0 [unknown] [unknown] 127.0.0.1(40650) 27093 1 LOG:  connection received: host=127.0.0.1 port=40650
  66a0e2ed.69d5 0 VCDB vc 127.0.0.1(40650) 27093 2 FATAL:  password authentication failed for user "vc"
  YYYY-MM-DD HH:MM:SS UTC 66a0e2ed.69d5 0 VCDB vc 127.0.0.1(40650) 27093 3 DETAIL:  Password does not match for user "vc".

• You cannot proceed with upgrading the vCenter to version 8

Environment

VMware vCenter Server 7.0.x

Cause

This can happen when there is a password inconstancy between the database configuration files and the vCenter database for the vCenter (vc) account. 

Resolution

NOTE: Before attempting the steps below, please ensure to take a snapshot of the vCenter server. If there are two or more vCenter Servers in Enhanced Linked Mode, all of the vCenter nodes will need to be shut down and cold /offline snapshots taken of them at the same time.

1. Log into the vCenter server machine and run the following command to reset the password in the configuration files: 

   # vpxd -p

   There will be a prompt for a password - choose one as needed. There will also be a prompt to confirm the password. Note that the password will not be echoed on screen:

   root@localhost [ ~ ]# vpxd -p
   Enter new DB password:
   again:

2. Connect to the vCenter database using the following command:

   # psql -U postgres -d VCDB

3. Run the following command to update the password in the database (replace <CHOSEN_PASSWORD_HERE> against the actual password used in step 1):

   ALTER ROLE vc WITH PASSWORD '<CHOSEN_PASSWORD_HERE>';

4. Run the following query to exit the psql database command line interface:

   \q

5. Finally restart all vCenter services with the following command:

   # service-control --stop --all && service-control --start --all