Spectrum Server crashes when adding to Global Collection

Products

• Network Observability
• Spectrum

Issue/Introduction

Spectrum Server crashes when adding to Global Collection

[hostname] kernel: SpectroSERVER [19584]: segfault at xxx error 6 in libmts64․so

Environment

Spectrum Server All Releases

Cause

Because there are more than 500 conditions in the GC, while traversing the condition list the accumulated stack frames (each containing several local variables) eventually exceed the SpectroSERVER worker thread's stack size at around a tree depth of 500. This causes a native stack overflow and an unrecoverable crash.

Resolution

 

1. Split the conditions (Recommended): Keep the number of conditions low, or split the search conditions across multiple GCs. This is the highly recommended approach.

OR

2. Increase the stack size: Increase the stack frame size by adding the following parameter to the $SPECROOT/SS/.vnmrc file and restarting the SpectroSERVER:

stack_size_modifier=2

We have successfully reproduced the issue locally and verified that setting stack_size_modifier=2 prevents the SpectroSERVER from crashing.

However, increasing the stack size is not the recommended long-term solution; Splitting the conditions across multiple GCs is the best long term approach.

Additional Information

How to Split Spectrum Global Collection