How to generate stack dumps for the SpectroSERVER on Linux

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

How can stack dumps be generated for the SpectroSERVER (SS) process?

For instance, if there are high cpu or a hang type of performance issue with the SS, OneClick may show switched as it had to disconnect and reconnect to the secondary SS.  How can we generate stack dumps for support to determine the issue?

The Spectrum code will try to generate stack dumps using pstack on linux if it can detect that the internal processes are busy/hung for 5 minutes.  Or you may need to generate the stack dumps yourself.  This doc outlines requirements and steps needed on a linux SS.

Environment

DX NetOps Spectrum all currently supported releases

Resolution

On linux, the pstack utility is required.  This utility is installed as part of the linux debugging package known as gdb.  To obtain pstack, just install the gdb package:

yum install gdb

Once installed, pstack will be located under:

/usr/bin/pstack  

Spectrum code will utilize pstack to try and generate stack dumps.  If you run the linux process output "top" command and see the SpectroSERVER process consistently consuming high cpu (ie greater than 90%), you can manually run pstack to generate stack dumps as per the following:

1. Obtain the process id of the SpectroSERVER (<PID of SS>).  Either use the linux "top" command or "ps -ef | grep -i SpectroSERVER"
   
   
2. As root, or sudo, run pstack and output the stack to a file:
   
   ./usr/bin/pstack <PID of SS> >> SS_pstack.OUT
   
   for example, if the process ID of the SpectroSERVER is 3442, then:
   
   ./usr/bin/pstack 3442 >> SS_pstack.OUT


3. Run this same command 5x.  Provide the SS_pstack.OUT to Broadcom support.
   
   
4. Run "top -H" and grab a screenshot.  Provide this to Broadcom support as well.

This will show the threads that the SS is using, and will show which thread number is using the most cpu.  You can then search the SS_pstack.OUT file for the thread # to see what the offending thread is. 

Additional Information

To have Spectrum automatically monitor the SpectroSERVER cpu and generate stack dumps automatically, you can enable the Self Health monitoring on the VNM model.  You still need to have pstack installed for this to work:

Refer to the following section of the TechDocs for further details:

TechDocs : DX NetOps Spectrum 23.3 : Self-Health Monitoring