Jobs stuck and "Error getting a DB connection" and "Connection unavailable" in logs on GCP

Products

• Clarity PPM SaaS
• Clarity PPM On Premise

Issue/Introduction

Some or all jobs are not running and we can see the following error in bg logs:

ERROR 2020-11-16 07:15:56,640 [Dispatch Time Slicing : bg@HOST(tenant=clarity)] niku.blobcrack (clarity:admin:42766287__F5161F63-2431-4F36-B884-16B05BF096A6:Time Slicing) Exception during blobcrack process
com.niku.union.persistence.PersistenceException: Error getting a DB connection
              at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:620)
              at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:311)
              at com.niku.xql2.pmd.PMDRecordSet.executePMDStatement(PMDRecordSet.java:275)
Caused by: java.sql.SQLException: Connection unavailable
              at com.niku.union.persistence.connection.ApacheContext.getConnection(ApacheContext.java:213)
              at com.niku.union.persistence.PersistenceController.createLocalContext(PersistenceController.java:461)
              at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:569)
              ... 28 more

 

Error in bg-system log:

2020/11/16 07:15:57.653 | Nov 16, 2020 7:15:57 AM org.apache.tomcat.jdbc.pool.ConnectionPool terminateTransaction
2020/11/16 07:15:57.654 | WARNING: Unable to terminate transaction, connection will be closed.
2020/11/16 07:15:57.654 | org.postgresql.util.PSQLException: This connection has been closed.
2020/11/16 07:15:57.654 |        at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:782)
2020/11/16 07:15:57.654 |        at org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:742)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.terminateTransaction(ConnectionPool.java:871)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.shouldClose(ConnectionPool.java:897)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.returnConnection(ConnectionPool.java:933)

Environment

Release : 15.8.1+

Component : 

Cause

When checking with SAAS DBA we find that:

This is caused by an OOM event on the PostgreSQL database (OS level) and the database that goes into recovery mode. When it recovers, BG no longer processes some or all jobs

Resolution

To resolve the problem, BG service has to be restarted

This is logged as DE58908, and currently fixed in 15.9.0