10.4.1 Spectrum OneClick tomcat Out of Memory

Products

• CA Spectrum
• CA eHealth
• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

When DX Spectrum and DX Performance Center (CAPM) are integrated, the Spectrum Tomcat process may run out of memory.  The hprof file will show GCSubscriber as a leak suspect, and there will be thousands of restful subscriptions for the connection to CAPM.

Environment

Release : 10.4.1

Component : Spectrum OneClick

Cause

CAPM has a timeout value of 100 seconds for the subscription request to be returned.  The subscription request comes into Spectrum, Spectrum generates the subscription id, runs the request, and sends the subscription id response after it completes processing the data.  The problem is that there was so much data to process, it took longer than 100 seconds.  The subscription would timeout on the PM side and not send a proper Delete for the subscription.  The next time a request was made (about 3 minutes later) a new subscription was created.  The old subscriptions still exist in tomcat memory.  This cycle eventually runs tomcat out of memory.

Resolution

This is resolved by Spectrum_10.04.01.D20c for 10.4.1.  The request response was moved to it's own thread, which returns the successful response immediately and still allows the original request to process the subscription data:

 

DE452329 Details:
   31812662 - Charter - 10.4.1 OC tomcat OOM
  (DE452329, 31812662)

Additional Information

Please note there are 3 fixes in Spectrum_10.04.01.D20c.  These fixes are tentatively scheduled to be included in the first bi monthly patch for 10.4.1 (Spectrum_10.04.01.BMP_10.4.101) and tentatively scheduled to be included in 10.4.2 once released.

1. OOM issue with CAPM REST subscriptions

https://knowledge.broadcom.com/external/article?articleId=187713 

2. Illegal Argument error causing event sync to stop

https://knowledge.broadcom.com/external/article?articleId=187708

3. Enhanced processing of inventory sync

https://knowledge.broadcom.com/external/article?articleId=187906