VMware Identity Manager fails to start with error: Could not acquire change log lock

Products

• VMware Aria Suite

Issue/Introduction

Accessing VIDM  UI in a browser returns a HTTP 404, or 502, or 503 “Bad Gateway” or red error banner. 

The Following error can be seen on Horizon.log,

 

ERROR (localhost-startStop-1) [;;] org.springframework.web.context.ContextLoader - Context initialization failedorg.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [spring/datastore-wireup.xml]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by <ipAddressHoldingLock> (<ipAddressHoldingLock>)

Authentication to applications through WS1 VIDM fails

 

Environment

VMware Identity Manager 3.3.x

Cause

When performing a database migration performing a database migration, or if the application experiences an ungraceful shutdown, or if there are network disruptions between nodes and VIDM, it can result in a stale database lock record in the saas.databasechangeloglock.

Resolution

The following steps can be taken to manually remove the lock on the VIDM Database:

1. Take a valid Snapshot /backup of the VIDM cluster nodes. 
   
   
2.  Log in using root credentials to a console session on each VIDM appliance node and stop the horizon-workspace service by running the following command:
   

   # service horizon-workspace stop

    

3. Connect to the VIDM database on master node and run the following query to verify if the database is in a Locked state:
   

   export PGPASSWORD=`cat /usr/local/horizon/conf/db.pwd`
   /opt/vmware/vpostgres/current/bin/psql -U postgres saas
   
   saas# SELECT * FROM saas.DATABASECHANGELOGLOCK;
   
    If database locked will show t for true, the following steps can be taken to release the lock manually
    id | locked | lockgranted | lockedby
   ----+--------+-------------+----------
    1 | t            |   xxxxxxxxx  | xxxxxxx

    

4. Execute the following update to release the lock on the primary node:
   

   UPDATE saas.DATABASECHANGELOGLOCK set locked=FALSE, lockgranted=NULL, lockedby=NULL where id=1;

   
   Re-run Step 3 to verify that the lock has been released (LOCKED should have the value f)
   
   
   
5. Start the horizon-workspace service by running the following command on each node. It is highly recommended that you wait 2 minutes between running this service on one node and the next.
   

   # service horizon-workspace start