SpectroSERVER process using 100% CPU fails over to secondary

Products

• Network Observability
• Spectrum

Issue/Introduction

You see the SpectroSERVER process using 100% CPU and it eventually fails over to secondary.  You try to restart the SpectroSERVER but maybe it hangs for a long time and needs to be killed .

Cause

VNM threads were increased beyond the capacity of the server, the server ran out of resources and hung.  The VNM threads should only be altered at the request and with the supervision of engineering.  Please do not attempt to alter the threads by yourself or you risk performance issues and eventually a hung SpectroSERVER.

Resolution

If the VNM threads have been altered incorrectly and CPU is at 100% for the SpectroSERVER process, please undo the changes to the VNM threads and restart the SpectroSERVER. 

Additional Information

The SpectroSERVER process is single threaded so for better performance of the process, a faster CPU is more important than the number of CPUs the server has.  If you have a very heavily loaded SpectroSERVER and you wish it to perform as well as possible, please consider the type of CPU you are using on your SpectroSERVERs.  Modern CPUs are faster and more efficient so you should upgrade the CPUs as often as possible to have the most efficient SpectroSERVER.  Again this is important in very heavily loaded environments or environments that have grown in size and we see performance issues.

To check your CPU and get clear, summarized info run 

 lscpu

What to look for:

Clock Speed:
Strong IPC (Instructions Per Cycle). This is a primary factor in single-threaded performance, often more important than raw clock speed alone.
Good Boost Clocks: Some CPUs can boost individual cores to significantly higher frequencies for short bursts or when only a few cores are active, which directly benefits single-threaded applications.
Large Cache: This can reduce memory access latency.

Are you using a virtual Machine?  VM Contention is the #1 Risk to and the most significant factor for "demanding single-threaded processes" in a VM as it may be sharing physical CPU time with potentially many other VMs on the same host.  A demanding single-threaded application is highly sensitive to CPU scheduling latency and jitter. If the hypervisor frequently pauses your vCPU to run another VM's workload, your application's effective "speed" will suffer, even if the underlying physical core is fast. 

For truly "demanding" single-threaded enterprise applications (e.g., high-frequency trading calculations, real-time analytics, critical legacy applications), the unpredictability of a shared VM environment can be a major problem.