search cancel

Jobs stuck and "Error getting a DB connection" and "Connection unavailable" in logs on GCP

book

Article ID: 203413

calendar_today

Updated On:

Products

Clarity PPM SaaS Clarity PPM On Premise

Issue/Introduction

Some or all jobs are not running and we can see the following error in bg logs:

ERROR 2020-11-16 07:15:56,640 [Dispatch Time Slicing : [email protected] (tenant=clarity)] niku.blobcrack (clarity:admin:42766287__F5161F63-2431-4F36-B884-16B05BF096A6:Time Slicing) Exception during blobcrack process
com.niku.union.persistence.PersistenceException: Error getting a DB connection
             at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:620)
             at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:311)
             at com.niku.xql2.pmd.PMDRecordSet.executePMDStatement(PMDRecordSet.java:275)
Caused by: java.sql.SQLException: Connection unavailable
             at com.niku.union.persistence.connection.ApacheContext.getConnection(ApacheContext.java:213)
             at com.niku.union.persistence.PersistenceController.createLocalContext(PersistenceController.java:461)
             at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:569)
             ... 28 more

 

Error in bg-system log:

2020/11/16 07:15:57.653 | Nov 16, 2020 7:15:57 AM org.apache.tomcat.jdbc.pool.ConnectionPool terminateTransaction
2020/11/16 07:15:57.654 | WARNING: Unable to terminate transaction, connection will be closed.
2020/11/16 07:15:57.654 | org.postgresql.util.PSQLException: This connection has been closed.
2020/11/16 07:15:57.654 |        at org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:782)
2020/11/16 07:15:57.654 |        at org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:742)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.terminateTransaction(ConnectionPool.java:871)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.shouldClose(ConnectionPool.java:897)
2020/11/16 07:15:57.654 |        at org.apache.tomcat.jdbc.pool.ConnectionPool.returnConnection(ConnectionPool.java:933)

Environment

Release : 15.8.1+

Component : 

Cause

When checking with SAAS DBA we find that:

This is caused by an OOM event on the PostgreSQL database (OS level) and the database that goes into recovery mode. When it recovers, BG no longer processes some or all jobs

Resolution

To resolve the problem, BG service has to be restarted

This is logged as DE58908, and currently fixed in 15.9.0