CA API Developer Portal containers are not coming up

Products

• CA API Developer Portal
• CA API Gateway

Issue/Introduction

API Portal 4.x , 5.x  does not come up after a power outage and restart. 


The below errors are found in the portal logs:

| 2018-08-20 15:56:13.854  WARN [tenant-provisioning-service,,,] 9 --- [           main] c.c.a.p.tenant.config.LiquibaseConfig    : Failed to get DB connection on attempt 0 of 5 [An attempt by a client to checkout a Connection has timed out.], retrying in 5s
 | 2018-08-20 15:56:20.318  INFO [tenant-provisioning-service,,,] 9 --- [           main] liquibase                                : Waiting for changelog lock....
portal_tenant-provisioner.1.8g24zxxxxxxxx@xxxxxxxxxxxxxxxxx   | INFO: Waiting for changelog lock....

Cause

On a powerdown and restore or update  the database log lock status can be in set in an unavailable state.  

The portal documentation : 

https://techdocs.broadcom.com/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/5-2/install-configure-and-upgrade/install-portal-on-docker-swarm/troubleshoot-the-installation.html

 

Has some troubleshooting steps: 
 

UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;

 

But there are a number of databases in use, and the locked log setting may not be in the "portal" database.

Resolution

The complete resolution means checking ALL the databases to see if any of them have DATABASECHANGELOGLOCK entries locking access to the database. 

This includes the following  databases :

portal
tenant_provisioning
otk_db
rbac
analytics
druid

For example TENANT_PROVISIONING

use TENANT_PROVISIONING ;
SELECT * FROM DATABASECHANGELOGLOCK; UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;

All databases should be checked for LOGLOCK entries.