CA API Developer Portal containers are not coming up

Products

• CA API Developer Portal
• CA API Gateway

Issue/Introduction

API Portal 4.x , 5.x  does not come up after a power outage and restart.


The below errors are found in the portal logs:

| 2018-08-20 15:56:13.854  WARN [tenant-provisioning-service,,,] 9 --- [           main] c.c.a.p.tenant.config.LiquibaseConfig    : Failed to get DB connection on attempt 0 of 5 [An attempt by a client to checkout a Connection has timed out.], retrying in 5s
 | 2018-08-20 15:56:20.318  INFO [tenant-provisioning-service,,,] 9 --- [           main] liquibase                                : Waiting for changelog lock....
[email protected]   | INFO: Waiting for changelog lock....

Cause

On powerdown and restore the database log lock status can be in set in an unavailable state.  

The portal documentation : 

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/layer7-api-management/api-developer-portal/4-4/install-configure-and-upgrade/install-portal-on-docker-swarm/troubleshoot-the-installation.html

Has some troubleshooting steps: 
 

UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;

 

But there are a number of databases in use, and the locked log setting may not be in the "portal" database.

Resolution

The complete resolution means checking ALL the databases to see if any of them have DATABASECHANGELOGLOCK entries locking access to the database. 

This includes these databases
portal
tenant_provisioning
otk_db
lddsdb

rbac

analytics

druid

ie:
use TENANT_PROVISIONING ;
SELECT * FROM DATABASECHANGELOGLOCK; UPDATE DATABASECHANGELOGLOCK SET locked=0, lockgranted=null, lockedby=null WHERE id=1;

But all databases should be checked for LOGLOCK entries.