Java heap space crash loop when processing VNA notification backlog - Spectrum

Products

• Network Observability
• Spectrum
• Virtual Network Assurance

Issue/Introduction

Spectrum OneClick server experiences a recurring crash loop shortly after service startup. The process exhausts available memory while attempting to synchronize a large volume of historical notifications from Virtual Network Assurance (VNA).

• OneClick Tomcat logs (catalina.out or stdout.log) show java.lang.OutOfMemoryError: Java heap space.
• Logs show SDNFaultEventProcessor stuck on old timestamps (e.g., from a year prior).
• Tomcat crashes abruptly and, upon restart, begins processing from the same historical date again.

Environment

• DX NetOps Spectrum integrated with VNA
• Versions 24.3.x
• High volume of historical VNA notifications (e.g., >10GB backlog)

Cause

When Tomcat restarts, the integration requests a full synchronization of historical notifications. If the backlog is massive, the memory required to process the bursts (default 10,000 alerts per packet) exceeds the Tomcat heap before the sync pointer can be updated.

Resolution

1. Stop OneClick Tomcat.
2. Reset VNA Sync Pointer: On the VNA Gateway, navigate to /opt/CA/VNA/collector/[ENGINE_UUID]/repository/work/subscription/ and delete the file Alarms_LastSyncTime.
3. Purge VNA Database (Manual): Stop the VNA WildFly service and use MySQL to delete notifications older than a specific date:
   sql

   DELETE FROM notification_attributes WHERE notification_id IN (SELECT id FROM notifications WHERE lastUpdateTime < (UNIX_TIMESTAMP('2026-01-01') * 1000));
   DELETE FROM notifications WHERE lastUpdateTime < (UNIX_TIMESTAMP('2026-01-01') * 1000);
   COMMIT;

4. Adjust Memory: Ensure OneClick Tomcat -Xmx is set to at least 24576M (24GB) in catalina.sh or OneClickService.conf.
5. Clear Cache: Delete contents of $SPECROOT/tomcat/work/ and $SPECROOT/tomcat/temp/.
6. Restart Services.