lookup service fails to start after modifying VCHA

Products

• VMware vCenter Server

Issue/Introduction

• vCenter High Availability (VCHA) fails to failover
• vCHA recently disabling/destroyed
• vpostgres service is up but lookupsvc does not start
• lookupsvc service logs shows:

ERROR: cannot execute CREATE TABLE in a read-only transaction
Cannot execute create table in a read-only transaction. Caused by: org.springframework.beans.factory.beancreationexception: Error creating bean with name 'ssoserveradapter' defined in servletcontext resource [WEB-INF/tomcat-ls.xml]: Cannot create inner bean'######'

• postgres service logs shows:

UTC ########.### #   FATAL:  could not connect to the primary server: could not connect to server: No route to host
Is the server running on host "#.#.#.#" and accepting
TCP/IP connections on port 5432?
UTC ########.### # root root LOG:  could not send data to client: Broken pipe

• Recently disabled VCHA (High Availability)

• Fail to update SSL certificates with errors similar to the following:

  • operation failed: Unable to update  com.vmware.rbd extension thumbprint in VCDB

  • "An error occurred while invoking external command : 'Error while updating the certificate for VC extension com.vmware.vim.eam'"

    
    

     

Environment

7.x, 8.x

Cause

• Failed VCHA failover
• VCHA was not properly destroyed/disabled.
• VCHA split brain situation

Resolution

1. Generate a file-based backup. See Manually Back up vCenter Server by Using the vCenter Server Management Interface

2. Log into the appliance as root.

3. Manually promote the database with the command:

   sudo -u vpostgres /opt/vmware/vpostgres/current/bin/pg_ctl promote -D /storage/db/vpostgres

4. Restart all the services:

   1. service-control --stop --all

   2. service-control --start --all

5. If that fails to resolve the issue, move the following standby files:

   1. cd /storage/db/vpostgres

   2. mv standby.conf /tmp

   3. mv standby.signal /tmp

6. Restart the vCenter Services:

1. 1. service-control --stop --all

   2. service-control --start --all


Additional Information

• vPostgres will keep reaching out to the other node that had the active role while it was already decommissioned.
• Lookup service will not start due to vPostgres in recovery/read-only mode.