VMware Identity Manager fails to start with error: Could not acquire change log lock

Products

• VMware Aria Suite

Issue/Introduction

Accessing VIDM  UI in a browser returns a HTTP 404, or 502, or 503 “Bad Gateway” or red error banner. 

The Following error can be seen on Horizon.log,

 

ERROR (localhost-startStop-1) [;;] org.springframework.web.context.ContextLoader - Context initialization failedorg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by <ipAddressHoldingLock> (<ipAddressHoldingLock>)

Authentication to applications through WS1 VIDM fails

 

Environment

VMware Identity Manager 3.3.x

Cause

When performing a database migration performing a database migration, or if the application experiences an ungraceful shutdown, or if there are network disruptions between nodes and VIDM, it can result in a stale database lock record in the saas.databasechangeloglock.

Resolution

The following steps can be taken to manually remove the lock on the VIDM Database:

1. Take a valid Snapshot /backup of the VIDM cluster nodes. 
   
   
2.  Log in using root credentials to a console session on each VIDM appliance node and stop the horizon-workspace service by running the following command:
   

   # service horizon-workspace stop

    

3. Connect to the VIDM database on master node and run the following query to verify if the database is in a Locked state:
   

   export PGPASSWORD=`cat /usr/local/horizon/conf/db.pwd`
   /opt/vmware/vpostgres/current/bin/psql -U postgres saas
   
   saas# SELECT * FROM saas.DATABASECHANGELOGLOCK;
   
    If database locked will show t for true, the following steps can be taken to release the lock manually
    id | locked | lockgranted | lockedby
   ----+--------+-------------+----------
    1 | t            |   xxxxxxxxx  | xxxxxxx

    

4. Execute the following SQL command to release the lock on the primary node:
   

   UPDATE saas.DATABASECHANGELOGLOCK set locked=FALSE, lockgranted=NULL, lockedby=NULL where id=1;

   
   Re-run Step 3 to verify that the lock has been released (LOCKED should have the value f)
   
   
   
5. Start the horizon-workspace service by running the following command on each node. It is highly recommended that you wait 2 minutes between running this service on one node and the next.
   

   # service horizon-workspace start

   
   
   
6. Monitor the logfile horizon.log until the application initialization is complete, without logging further deadlock warnings or change log lock.

Additional Information

Verify new migrations have been recorded

1. Run this SQL command:

SELECT * FROM databasechangelog ORDER BY orderexecuted DESC LIMIT 1;

1. Verify that the listed migration is different to the migration noted above.
2. Verify that the orderexecuted number is higher in value.
3. Run this command to start Horizon Workspace on any remaining Service VAs:

service horizon-frontend start