SDDC Manager service(s) may fail to start with "Could not acquire change log lock." if the service or SDDC Manager is abruptly restarted during service initialization

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• In the /var/log/vmware/vcf/{service_name}/{service_name}.log, you see errors messages similar to the below:

2019-07-26T00:00:09.007+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:00:19.017+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:00:29.019+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:00:39.021+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:00:49.022+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:00:59.023+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:01:09.025+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:01:19.026+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:01:29.029+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2019-07-26T00:01:39.033+0000 WARN [0000000000000000,0000] [o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by 10.x.x.x (10.x.x.x) since 7/22/19 8:02 PM
2019-07-26T00:01:39.036+0000 INFO [0000000000000000,0000] [o.a.catalina.core.StandardService,main] Stopping service [Tomcat]

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

• The SDDC Manager UI does may not display any domain information
• UI displays "Failed to Poll" and "Failed to Fetch" messages 

Failed to poll domain info. Could not reach the server., possible reasons could be network connectivity issue or server going down.
Failed too fetch allocation. Could not reach the server., possible reasons could be network connectivity issue or server going down

• In the Common Services log, located at /var/log/vmware/vcf/commonsvcs, there are references to 502 Bad Gateway:

2019-07-11 08:56:50.443 ERROR [pool-2-thread-5] [com.vmware.evo.sddc.taskaggregator.services.TaskCollector] collect() failedorg.springframework.web.client.HttpServerErrorException: 502 Bad Gateway

Environment

VMware Cloud Foundation 3.0.x
VMware Cloud Foundation 3.7.x
VMware Cloud Foundation 3.8.x
VMware Cloud Foundation 3.9.x
VMware Cloud Foundation 3.10.x
VMware Cloud Foundation 4.0.x
VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.2.x
VMware Cloud Foundation 4.3.x
VMware Cloud Foundation 4.5.x
VMware Cloud Foundation 5.0.x
VMware Cloud Foundation 5.1.x
VMware Cloud Foundation 5.2.x

Cause

• A service acquires a database change log lock during initialization.
• If this service is restarted or the VM rebooted before the lock was released, it will result in race condition. Subsequent service restarts will continue to wait for this lock.

Resolution

• This is a known issue, to workaround the problem please follow the steps below.

Workaround:

1. Take a backup/snapshot of the SDDC Manager VM before proceeding
2. Open an SSH to the SDDC Manager VM, change to root:

su -

3. Run the following command to drop the databasechangeloglock table for the failing service:

For VMware Cloud Foundation 3.7.x:

psql -h /home/postgresql/ -U postgres -d {service_name} -c 'drop table databasechangeloglock;'
 

For VMware Cloud Foundation 3.8.x - 4.5.x:

psql -h localhost -U postgres -d {service-name} -c 'drop table databasechangeloglock;'

 

           For VMware Cloud Foundation 5.x and onward:

     /usr/pgsql/13/bin/psql -h localhost -U postgres -d {service-name} -c 'drop table databasechangeloglock;'

 

4. Reboot the SDDC Manager VM to ensure all services come up successfully