"Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available"

Products

• Clarity PPM On Premise
• Clarity PPM SaaS
• STARTER PACK-CLARITY PPM

Issue/Introduction

We are unable to log into Clarity.

Error in app-ca log:

ERROR 2021-03-10 00:13:36,700 [https-jsse-nio2-8444-exec-27] niku.security (clarity:unknown:33808649__EB0E57CE-E9D5-416F-8764-B1119F7C6E35:none) UserSessionCache.get:PMD error

com.niku.union.persistence.PersistenceException: Error getting a DB connection

              at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:620)

              at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:311)

Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [https-jsse-nio2-8444-exec-27] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:1000; busy:1000; idle:0; lastwait:30000].

 

Environment

Release : Any

Component : CA PPM INTEGRATIONS & INSTALLATIONS

Resolution

• The issue is that you ran out of connections on your database. There was no more available connections over 1000, which caused the issue.
• This usually is caused by a connection leak. It could be custom processes that do not close the connections or something else in the product. 
• It could be a REST API integrations that may have flooded the app in a short burst

 

To remedy the problem, restart the services, this should free up the connections. 

If this happens again:

1. Take a heap dump of the application (instructions here) and provide to Broadcom Support for analyze:

https://knowledge.broadcom.com/external/article?articleId=140774

2. Take a look at the /niku/apache page of the application. After restart, monitor what type of connections are created: 

http://SERVERNAME:PORT/niku/apache

3. Ask the DBA to provide data from the db server side regarding open connections / running queries in case they are unleaked connections but long running blocked/stuck queries