Information to collect if the SpectroSERVER is non-responsive and no errors are logged
search cancel

Information to collect if the SpectroSERVER is non-responsive and no errors are logged

book

Article ID: 100039

calendar_today

Updated On:

Products

CA Spectrum DX NetOps

Issue/Introduction

We have noticed a couple of abnormal terminations of the SpectroSERVER servers that were normally quite stable in the past. The operations team requires documented instructions of what data to capture in the event of another non-responsive SpectroSERVER occurrence is seen in the future.

Therefore, what data should be captured when the SpectroSERVER is non-responsive and no errors are logged?

Environment

Release: 10.4.x
Component: SPCCSS

Resolution

Anytime a SpectroSERVER appears to be unresponsive, you should do the following while the SpectroSERVER is in the unresponsive state.

  • Enable the General Errors and Communication debugs on the VNM model if possible. You may not be able to do this since OneClick may not update the views to allow for the debugs to be enabled. If the issue happens frequently, you may need to enable the debugs before the problem is seen.



To Enable the "General SpectroSERVER Error" and "General SpectroSERVER Communication" debugs:



  1. Locate the VNM model in OneClick.
  2. In the Component Detail pane - Information tab, scroll down to the Dynamic Debug subview.
  3. Enable the “General SpectroSERVER Errors" and "General SpectroSERVER Communication" debugs.
  4. The output will be written to the $SPECROOT/SS/VNM.OUT.


 

  • Gather the SpectroSERVER configuration files and logs, by running the getSpectrumInfo script:



  1. From the command line on the server (as the Spectrum Owner)
  2. cd $SPECROOT/bin/support
  3. Copy the getSpectrumInfo.sh file directly to the $SPECROOT directory
  4. cd $SPECROOT
  5. Run: ./getSpectrumInfo.sh
  6. The script will create a file called logs-<hostname>-<date>-<time>.tar.gz.
  7. Open a case with CA Support and upload a copy of the logs-<hostname>-<date>-<time>.tar.gz for Support to review.


​

  • When the problem returns, while the SpectroSERVER seems unresponsive, try running a CLI command to see if the SpectroSERVER is truly non-responsive. If the SS is unresponsive, you may find the connection fails, or times out.

    To open a CLI Command Line:



  1. From the command line on the server (as the Spectrum Owner).
  2. cd $SPECROOT/vnmsh
  3. Run the following command: ./connect
  4. Run the following command: ./show models
    1. If the SpectroSERVER responds, execute a "Ctrl C" to break out of the command, and notate the results in the CA Support case you opened. Upload a copy of the $SPECROOT/SS/VNM.OUT file.
    2. If the command is unsuccessful continue with gathering the data requested below.
  5. Run ./disconnect to disconnect the CLI session.


​

  • While the SpectroSERVER is unresponsive, collect threading information, and moot tracing, so CA Support can get a snapshot of the activity the SpectroSERVER process is currently attempting to perform.



To get the moot trace:



  1. From the command line on the server (as the Spectrum Owner)
  2. Run kill -USR1 [SS_PID]
  3. Wait a couple of minutes and run the kill -USR1 command again to end the trace.
  4. Repeat steps 3 and 4, five times.
  5. Upload a copy of the $SPECROOT/SS/.moot.trace file and any $SPECROOT/SS/support/Incident.Data files, along with a copy of the $SPECROOT/SS/VNM.OUT file, to the support case you opened.


 

  • (Optional) Once this data is collected, you can go ahead and try stopping the SpectroSERVER and restarting Spectrum to see if this returns the SpectroSERVER to normal activity for a time. It is very likely the SpectroSERVER will not stop cleanly, and require you to terminate/kill the SpectroSERVER process. Terminating the SpectroSERVER process will result in corrupting the SSDB database. Therefore, you will need a good backup of the SSDB database for you to run a restore from. If a known, good, backup of the SSDB does not exists, contact CA Support for further help.


 

  • (Optional) It's not uncommon for the SpectroSERVER to be busy processing traps, or running reconfigurations/rediscoveries, which could be impacting the SpectroSERVER performance. It is a good idea, to collect the SSPerformance data, so the CA Support Engineer can review this data, and see if the unresponsive SS is related to performance issues.


 

Running the perfCollector9 utility, is a good idea to gather the SSPerformance data for review.



  1. As the Spectrum Owner, open a command line on the SpectroSERVER.
  2. cd to $SPECROOT
  3. Run: ./perfCollector9
  4. The utility will make calls to Archive Manager on active SpectroSERVER and gather the SSPerformance events, and a few other key events, known to impact Spectrum performance. It will create a $SPECROOT/Performance directory and tar up the data in the results.tar.gz file. Once it completes, please upload a copy of the $SPECROOT/Performance/results.tar.gz to the support case you opened, for CA Spectrum Support to review.
Powered by Wolken Software