Services not starting after refresh, stuck at Cache Messenger

Products

• Clarity PPM On Premise

Issue/Introduction

After refresh, unable to bring up the services (app, bg)

The services hang at starting Cache Messenger, log examples:

APP service:

SYS 2019-09-26 20:45:04,233 [localhost-startStop-1] niku.union (clarity:none:none:none) Starting Cache Messenger...

BG Service:

2019/09/26 20:44:51.267 | SYS 26-09 20:44:51,203 - Starting Cache Messenger...
2019/09/26 20:44:51.267 | SYS 2019-09-26 20:44:51,203 [WrapperSimpleAppMain] niku.union Starting Cache Messenger...
2019/09/26 20:44:51.784 | No tenantId available to get the tenant databaseId. Proceeding with the dbId=Niku

Additional error message in the logs:

No tenantId available to get the tenant databaseId. Proceeding with the dbId=Niku

Note: The environment is using JDBC Ping. The services get stuck on statement DELETE FROM CMN_JGROUPS_PING

Resolution

• With JDBC PING if table CMN_JGROUPS_PING has entries from previous environment that will mislead the scripts and cause a problem.
• Broadcom Support recommends to add a step to truncate this table after each refresh to prevent this type of issue. This is a documented requirement
• After refresh the table CMN_JGROUPS_PING has to be reset for the service to add fresh entries
• Each service upon starting should delete the existing entry and recreate it, and if this is not happening and the statement will be left hanging

Workaround:

1. Truncate the table CMN_JGROUPS_PING
2. Restart the services