- The issue is that you ran out of connections on your database. There was no more available connections over 1000, which caused the issue.
- This usually is caused by a connection leak. It could be custom processes that do not close the connections or something else in the product.
- It could be a REST API integrations that may have flooded the app in a short burst
To remedy the problem, restart the services, this should free up the connections.
Consider gathering the below information before restarting in order to be able to determine the root cause. Or plan to do this if you have already restarted if this happens again:
- Take a heap dump of the application (instructions here) and provide to Broadcom Support for analyze:
Manual heap dump for application hanging or slow performance
- Take a look at the /niku/apache page of the application. After restart, monitor what type of connections are created:
- Ask the DBA to provide data from the db server side regarding open connections / running queries in case they are unleaked connections but long running blocked/stuck queries