How to generate stack dumps for the SpectroSERVER on Linux
Article ID: 139708
Updated On:
Products
Issue/Introduction
How can stack dumps be generated for the SpectroSERVER process?
For instance, if there are high cpu or a hang type of performance issue with the SpectroSERVER, OneClick may show switched as it had to disconnect and reconnect to the secondary SS. How can we generate stack dumps for support to determine the issue?
The Spectrum code will try to generate stack dumps using pstack on linux if it can detect that the internal processes are busy/hung for 5 minutes. Or you may need to generate the stack dumps yourself. This doc outlines requirements and steps needed on a linux SpectroSERVER.
Environment
Release: 20.2.x
Component: SPCCSS - SpectroSERVER Core
Resolution
On linux, pstack is needed. pstack is installed as part of the linux debugging package known as gdb. To obtain pstack, just install the gdb package:
yum install gdb
Once installed, you will have /usr/bin/pstack. Spectrum code will utilize pstack to try and generate stack dumps. If you run the linux process output "top" command and see the SpectroSERVER process consistently consuming high cpu (ie greater than 90%) you can manually run pstack to generate stack dumps.
1. Obtain the process id of the SpectroSERVER. Either use the linux "top" command or "ps -ef | grep SpectroSERVER"
2. As root, or sudo, run pstack and output the stack to a file:
./usr/bin/pstack pidofSS >> SS_pstack.OUT
for example:
./usr/bin/pstack 3442 >> SS_pstack.OUT
Run this same command 5x. Provide the SS_pstack.OUT to Broadcom support.
3. Run "top -H" and grab a screenshot. Provide this to Broadcom support as well.
This will show the threads that the SS is using, and will show which thread number is using the most cpu. You can then search the SS_pstack.OUT file for the thread # to see what the offending thread is.
Additional Information
To have Spectrum automatically monitor the SpectroSERVER cpu and generate stack dumps automatically, you can enable the Self Health monitoring on the VNM model. You still need to have pstack installed for this to work:
Feedback
