SDDC Manager services fail to start with "Could not acquire change log lock" if the service or SDDC Manager is abruptly restarted during service initialization

Products

• VMware Cloud Foundation

Issue/Introduction

• In the /var/log/vmware/vcf/{service_name}/{service_name}.log,  below error messages are seen:

INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
WARN [0000000000000000,0000] [o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by 10.x.x.x (10.x.x.x) since yyyy-mm-dd
INFO [0000000000000000,0000] [o.a.catalina.core.StandardService,main] Stopping service [Tomcat]

Note: This log excerpt is an example. Environmental variables may vary depending on your environment.

• The SDDC Manager UI may not display any domain information
• SDDC UI displays "Failed to Poll" and "Failed to Fetch" messages 

Failed to poll domain info. Could not reach the server., possible reasons could be network connectivity issue or server going down.
Failed too fetch allocation. Could not reach the server., possible reasons could be network connectivity issue or server going down

• In the Common Services log, located at /var/log/vmware/vcf/commonsvcs, there are references to 502 Bad Gateway:

ERROR [pool-2-thread-5] [com.vmware.evo.sddc.taskaggregator.services.TaskCollector] collect() failedorg.springframework.web.client.HttpServerErrorException: 502 Bad Gateway

Environment

VMware Cloud Foundation 3.0.x
VMware Cloud Foundation 3.7.x
VMware Cloud Foundation 3.8.x
VMware Cloud Foundation 3.9.x
VMware Cloud Foundation 3.10.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.2.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 4.5.x
VMware Cloud Foundation 5.0.x
VMware Cloud Foundation 5.1.x
VMware Cloud Foundation 5.2.x

Cause

• A service acquires a database change log lock during initialization.
• If this service is restarted or the VM rebooted before the lock was released, it will result in race condition. Subsequent service restarts will continue to wait for this lock.

Resolution

This is a known issue. To workaround the issue, refer the below steps.

Workaround:

1. Take a backup/snapshot of the SDDC Manager VM before proceeding.
2. Open an SSH to the SDDC Manager VM, change to root:

su -

3. Run the following command to drop the databasechangeloglock table for the failing service:
   
   For VMware Cloud Foundation 3.7.x:
   psql -h /home/postgresql/ -U postgres -d {service_name} -c 'drop table databasechangeloglock;'

For VMware Cloud Foundation 3.8.x - 4.5.x:

psql -h localhost -U postgres -d {service-name} -c 'drop table databasechangeloglock;'

 

           For VMware Cloud Foundation 5.x and onward:

     /usr/pgsql/13/bin/psql -h localhost -U postgres -d {service-name} -c 'drop table databasechangeloglock;'

4. Reboot the SDDC Manager VM and ensure all services are up.