"Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available" (database connections limit)

book

Article ID: 210726

calendar_today

Updated On:

Products

Clarity PPM On Premise Clarity PPM SaaS STARTER PACK-CLARITY PPM

Issue/Introduction

  • We are unable to log into Clarity.
  • We have leaked connections, jobs stuck or processes hanging

Error in app-ca/bg-ca log:

ERROR 2021-03-10 00:13:36,700 [https-jsse-nio2-8444-exec-27] niku.security (clarity:unknown:xxxxxx:none) UserSessionCache.get:PMD error

com.niku.union.persistence.PersistenceException: Error getting a DB connection

              at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:620)

              at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:311)

Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [https-jsse-nio2-8444-exec-27]Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:1000; busy:1000; idle:0; lastwait:30000].

 

Note: This article is also applicable if you experience a number of fast growing connections in the database and the DBA kills them before they hit the 1000 limit. 

Environment

Release : Any

Component : CA PPM INTEGRATIONS & INSTALLATIONS

Resolution

  • The issue is that you ran out of connections on your database. There was no more available connections over 1000, which caused the issue.
  • This usually is caused by a connection leak. It could be custom processes that do not close the connections or something else in the product. 
  • It could be a REST API integrations that may have flooded the app in a short burst

 

To remedy the problem, restart the services, this should free up the connections. 

If this happens again:

  1. Take a heap dump of the application (instructions here) and provide to Broadcom Support for analyze:

https://knowledge.broadcom.com/external/article?articleId=140774

  1. Take a look at the /niku/apache page of the application. After restart, monitor what type of connections are created: 

http://SERVERNAME:PORT/niku/apache

  1. Ask the DBA to provide data from the db server side regarding open connections / running queries in case they are unleaked connections but long running blocked/stuck queries
Powered by Wolken Software