ACS Health - Application Deployment Status, has failure in the vIDM Health tab causing various errors at vIDM and Aria Automation layer

Products

• VMware Aria Suite

Issue/Introduction

Below are some symptoms that you can see because of the ACS Health not being available

1. Certain sections of the UI may not function properly without ACS being operational.

In System Diagnostics > ACS Health, the following error is displayed:

"Error when connecting to the application."

Due to this ACS health issue:

• • Inventory Sync for vIDM in Aria Suite Lifecycle fails with the following error:
    
    Error Code: LCMVIDM71059 or LCMVIDM71091
"Failed to promote default configuration user as an admin. Retry after providing necessary details."
"Failed to promote local user to tenant admin on vIDM host xxxxx.xxxx.xxxxx.com. Exception message: vIDM GET Association Ruleset failed with status code: 404."
    
    
  • Aria Automation UI may display a "Bad Gateway" error.

2. When attempting to access the 'Roles' tab within the admin UI the following error is shown:

"An unexpected error has occurred. Please try again later."

The accesscontrol-service.log file under /opt/vmware/horizon/workspace/logs shows the following error:

2024-08-21 20:25:17,972 GMT ERROR accesscontrol (localhost-startStop-8) [;;;] org.springframework.boot.SpringApplication - Application run failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [com/vmware/vidm/accesscontrol/db/DbDataStoreAutoConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock.  Currently locked by XX.XX.XX.XX (XX.XX.XX.XX) since 8/19/24 5:50 AM

2024-08-21 20:25:17,982 GMT INFO accesscontrol (localhost-startStop-8) [;;;] com.vmware.vidm.common.http.context.listener.AbstractBootstrapServletContextListener - Shutting down micro service

3. When attempting to update the certificate for vIDM through Aria Suite Life Cycle Manager you will see the below error:

"Ensure that ACSHealth is up for node vIDM_fqdn"

Environment

VMware Identity Manager 3.3.x

Cause

There is a lock on the Access Control Service database table preventing the service from starting properly.

Resolution

To resolve this issue we need to remove the ACS database change lock

NOTE - Please ensure valid backups / snapshots are taken prior to performing the steps below

1. Login to all vIDM nodes via SSH as root user.
2. Shutdown horizon-workspace service on all nodes
   
   service horizon-workspace stop

   
   
3. Fetch the password using below command
   
   cat /usr/local/horizon/conf/db.pwd
   
   
4. On the master node connect to Postgres by running the following command
   
   psql -U postgres saas
   
   It will request Password for user postgres:use the password from step 3
5. Run the following command to confirm the lock
   
   select * from saas.ACS_DATABASECHANGELOGLOCK;


6. If there is a lock present, release the lock with this command
   
   update saas.ACS_DATABASECHANGELOGLOCK set locked='f', lockgranted=NULL, lockedby=NULL where id=1;
   
   
7. Exit PostgreSQL:
   
   \q
   
   
8. Start the horizon-workspace service on master node. Wait 30 seconds and start horizon-workspace service on other two nodes
   
   service horizon-workspace start
   
   

Access the admin UI once again. The ACS health should now also show GREEN and the related issues should be resolved.

Additional Information

• Stopping and restarting the horizon-workspace service will cause a brief downtime of vIDM, including user logins.
• Interacting with the DB can be done by below command as well with the horizon user:
  /opt/vmware/vpostgres/current/bin/psql -U horizon saas
• Running the sql queries can be done skipping the schema name as well and it will still work, for example:
  select * from ACS_DATABASECHANGELOGLOCK;
update ACS_DATABASECHANGELOGLOCK set locked='f', lockgranted=NULL, lockedby=NULL where id=1;