Portal tenant-provisioning container not starting

Products

• CA API Developer Portal

Issue/Introduction

Portal tenant-provisioning container not starting and the container logs shows the following error 

Errors in the portal_tenant-provisioning container log.

[email protected]    | Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'schedulerFactoryBean' defined in class path resource [com/ca/apim/provision/tenant/scheduler/SupervisorConfig.class]: Invocation of init method failed; nested exception is org.quartz.JobPersistenceException: Couldn't retrieve trigger: No record found for selection of Trigger with key: 'DEFAULT.provisionerTrigger' and statement: SELECT * FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'schedulerFactoryBean' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ? [See nested exception: java.lang.IllegalStateException: No record found for selection of Trigger with key: 'DEFAULT.provisionerTrigger' and statement: SELECT * FROM QRTZ_SIMPLE_TRIGGERS WHERE SCHED_NAME = 'schedulerFactoryBean' AND TRIGGER_NAME = ? AND TRIGGER_GROUP = ?]

Environment

API Developer Portal 5.2.x

Cause

 QRTZ_TRIGGERS table shows that the TRIGGER_NAME 'provisionerTrigger' is in TRIGGER_STATE = 'BLOCKED'.

Resolution

1. Connect to external portal docker swarm MySQL database and run this command.

mysql> use tenant_provisioning;

2. Check if this table - QRTZ_SIMPLE_TRIGGERS table is NOT empty by running this query

mysql> select * from QRTZ_SIMPLE_TRIGGERS;
+----------------------+--------------------+---------------+--------------+-----------------+-----------------+
| SCHED_NAME           | TRIGGER_NAME       | TRIGGER_GROUP | REPEAT_COUNT | REPEAT_INTERVAL | TIMES_TRIGGERED |
+----------------------+--------------------+---------------+--------------+-----------------+-----------------+
| schedulerFactoryBean | provisionerTrigger | DEFAULT       |           -1 |           30000 |             238 |
+----------------------+--------------------+---------------+--------------+-----------------+-----------------+
1 row in set (0.00 sec)

- if the above query returns an empty set,  ,  add back this entry in the tenant_provisioning database as following example :

Insert into "public"."qrtz_simple_triggers" ("sched_name","trigger_name","trigger_group","repeat_count","repeat_interval","times_triggered") values ('schedulerFactoryBean','provisionerTrigger','DEFAULT',-1,30000,238);

3. Run this query to check if the TRIGGER_STATE is BLOCKED for 'TRIGGER_NAME='provisionerTrigger'.

mysql> select * from QRTZ_TRIGGERS\G;

4. If the TRIGGER_STATE is BLOCKED, run the below update query to change the TRIGGER_STATE to ACQUIRED.

mysql> update QRTZ_TRIGGERS SET TRIGGER_STATE='ACQUIRED' where TRIGGER_NAME='provisionerTrigger';

Restart the portal stack and the portal_tenant-provisioner service should automatically start.