Ebs-app pods constantly restarting and failing to start with liquibase errors

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Deployment & destroy operations are failing with EBS liquibase related errors. The /var/log/services-logs/prelude/ebs-app/file-logs/ebs-app.log file contains errors similar to:

2023-04-13T13:02:57.788Z ERROR event-broker [host='ebs-app-5fcfdc48f8-tngnz' thread='main' user='' org='' trace='' request-trace=''] o.s.boot.SpringApplication.reportFailure:824 - Application run failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by ebs-app-%id%

• The kubectl -n prelude get pods commands shows the EBS pods with a repeating status of 'CrashLoopBackoff'.

 

 

Environment

VMware vRealize Automation 8.x

Resolution

To resolve the issue follow the steps outlined below:

1. Snapshot the vRealize Automation appliance(s).

2. Scale down the Ebs pods:

kubectl -n prelude scale deployment --replicas=0 ebs-app

3. Reset the liquibase component:

vracli reset liquibase

4. Scale the Ebs pods back up:

- For single appliance installs run the command:

kubectl -n prelude scale deployment ebs-app --replicas=1

- For 3 appliance installs run the command:

kubectl -n prelude scale deployment ebs-app --replicas=3

5. Allow a moment for Ebs pods to initialize and check that they are now running:

kubectl -n prelude get pods