After several weeks the OneClick tomcat that is integrated with Performance Manager runs out of memory and fails.
While the java memory is high, we see high CPU on the OneClick server that leads to synch failures with Performance Manager.
From the Java core files that were created we could see that the Send Notifications queue was at extremely high of memory during the crash.
Release : 10.4
Component : Spectrum OneClick with Performance Manager
Misconfiguration of an ip address in the Performance Manager SSO Tool that was pointing to an interface that OneClick could not reach, causing the failure messages to grow.
From the java core file that was created on the OneClick server we saw an extremely large amount of SendNotification failures, when we enabled REST debug on the OneClick side we saw that OneClick was receiving send notification failures from PM due to an issue with SSO Tool and an unreachable interface that was configured incorrectly.
Once this was reconfigured correctly on both the WebService Host and on the override in Performance Manager SSO Tool, then the OneClick tomcat and Performance Manager services were fully restarted we could no longer see the error messages in the debug on OneClick and the memory stopped growing.