Process Engine Gray status & JDBC Ping issue on 16.1.3.2+

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

Recently found having process engines grey and security.logs servers not displaying due to a JDBC Ping issue although the environment was upgraded to 16.1.3.2 to resolve DE69321

This issue also may cause process delays.

STEPS TO REPRODUCE:

1. Upgrade a cluster environment to 16.1.3.2
2. Leave it for a month without restarting

Expected Results: No orphan entries in CMN_JGROUPS_PING table and no JDBC Ping issues

Actual Results: Security logs servers do not display, process engines grey and lots of orphans in CMN_JGROUPS_PING table

Upon review the CMN_JGROUPS_PING table will have data going back to the date of last restart and the records are in much larger amount (expected about 2 per Clarity service)

Note: This issue has only been reported by customers that have Clarity deployed on AWS using JDBC Ping.

Resolution

This is DE79465, fixed in 16.2.2 and patches 16.1.3.3 and 16.2.1.1

With this fix a new column named "CHANNEL_HEART_BEAT" is introduced in table CMN_JGROUPS_PING, which would be regularly updated - every 1 minute - by the live nodes. When this column is not updated any orphan records will get identified and automatically cleaned up 

Workaround:

1. Truncate table CMN_JGROUPS_PING
2. Restart all services