BG stops working, processes (or process engine) are stuck until restart.
Error in logs:
Caused by: com.niku.union.persistence.PersistenceException: Error getting a DB connection
... 6 more
Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [Process Controller (tenant=clarity)] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:1000; busy:1000; idle:0; lastwait:30000].
Alternatively, the issue may be detected before the BG outage occurs if the DBA is receiving alert son database connection numbers and kills them in a timely manner
This is not an Out Of Memory nor a Clarity product issue. It's an indication that the database connections have run out. This usually only happens when a custom process or job is opening SQL connections but not closing them. It's also not a configuration of the database. 1000 connections should be plenty enough for anything done with the product. If you have a lot of custom processes that open connections, some of them may have a connection leak. This is why when you run your custom processes, you can eventually run out of connections.
Support may be able to review any heap dump evidence and determine the root cause
Closing GEL connections automatically upon completing the scripts will be included in 16.0.1 and 184.108.40.206 as DE63078
We recommend the following that should be addressed by the developers of the custom process: