SpectroSERVER Linux crash with core and GDB trace shows empty data

Products

• CA Spectrum

Issue/Introduction

One of our SpectroSERVERs have crashed and when we run the GDB trace we see a stack without good information in it.

 

(gdb) where
#0  0x00007f78b944fe26 in ?? ()
#1  0x00007f78bdcbe78e in ?? ()
#2  0x00007f76e9a88548 in ?? ()
#3  0x00007f76e9a883c0 in ?? ()
#4  0x00007f770edd4490 in ?? ()
#5  0x00007f78bdcc715f in ?? ()
#6  0x00007f781cb2f200 in ?? ()
#7  0x00007f76e9a88548 in ?? ()
#8  0x00007f76e9a8b598 in ?? ()
#9  0x0000000000000000 in ?? ()

 

How can we prevent this in the future?

Environment

Release : 10.x, 21.2.x

Component : SpectroSERVER Core

Cause

Not enough memory to write the stack at the time of crash when core file was created.

Resolution

In order to prevent this empty stack and create a good GDB trace in the future, I would highly recommend adding the following line to both the primary and secondary $SPECROOT/SS/.vnmrc file

stack_size_modifier=4

 

After this line has been added you will need to stop and restart both SpectroSERVERs for the setting to go into effect.  The Secondary SS should restart itself after a successful online backup, so you really only need to manually restart the primary SS.

Additional Information

What this setting does is allocate an extra 512mb of ram for the SpectroSERVER process which will do 2 things.

1. It provides extra memory to write a good stack incase of a future crash.

2. It provides an extra bit of memory to buffer the SS in case of an inefficient search that could also cause the SS to crash this way.

 

Only downside is that the SpectroSERVER process will grow at the very most by 512 megs of memory then previously, but that shouldn't be a concern in today´s environments.