Spectrum VNM.OUT file notes Triggering moot trace over and over again

Products

• Network Observability
• Spectrum

Issue/Introduction

When reviewing the $SPECROOT/SS/VNM.OUT file the following message noting "Triggering moot trace" shows repeatedly.

Sep 19 11:50:06 WARNING at ItcQdItem.cc(374): Request 5726168 accepted, but not complete after 60000ms, cs delta=412746. Triggering moot trace.

Sep 19 11:51:23 WARNING at ItcQdItem.cc(374): Request 5727111 accepted, but not complete after 60000ms, cs delta=452137. Triggering moot trace.

Sep 19 11:59:11 WARNING at ItcQdItem.cc(374): Request 5733299 accepted, but not complete after 60000ms, cs delta=450371. Triggering moot trace.

Sep 19 12:00:12 WARNING at ItcQdItem.cc(374): Request 5734097 accepted, but not complete after 60000ms, cs delta=326282. Triggering moot trace.

Sep 19 12:01:27 WARNING at ItcQdItem.cc(374): Request 5734893 accepted, but not complete after 60000ms, cs delta=414744. Triggering moot trace.

Sep 19 12:14:27 WARNING at ItcQdItem.cc(374): Request 5943477 accepted, but not complete after 60000ms, cs delta=1299617. Triggering moot trace.

 

If you review the file size of the $SPECROOT/SS/.moot.trace file it may be many GB in size.

When reviewing the stacks in the $SPECROOT/SS/.moot.trace file and searching for the "Request" number from the VNM.OUT the following is seen:

(Note - in this example we use 5943477 but you can look for any/all request ids that note Triggering moot trace):

Thread@000002696029A8A0,pri=90,WAITING,DYNAMIC,name=Native Thread Request #5943477

0x7ffa990447f8  libsskrnl.dll!CsModel::get_calc_attr_val
0x7ffa9904a4ae  libsskrnl.dll!CsModel::read_ulong
0x7ffa99062f4f  libsskrnl.dll!CsModelIf::read_ulong
0x7ffa985a858f  libIHapi.dll!CsIHBase::read_ulong
0x7ffa986f002f  libmdlsvint.dll!CsIHGetPorts::is_physical_port
0x7ffa986ef2a0  libmdlsvint.dll!CsIHGetPorts::get_physical_ports

Cause

Spectrum was externally polling devices for information more often than was needed causing delays when the SS was processing the information.

The reason the debug is showing now is from a change in output to the VNM.OUT in version 24.x  Prior this output was only enabled via debug options. It is now exposed by default.

The ramifications of the extra polling are higher CPU load on the SS

 

Resolution

This has been fixed in NetOps 25.4.2 and above.

As a workaround for versions before 25.4.2 (25.4.1 and below) you can add the following entries to the $SPECROOT/SS/.vnmrc file:

sync_all_ext_interface_attrs=false
int_copyattr_enable=true

Stop and restart the SpectroSERVER using the $SPECROOT/bin/stopSS.pl and $SPECROOT/bin/startSS.pl scripts.

Additional Information

If the $SPECROOT/SS/.moot.trace file is GB in size, you can zero out the file by navigating to the SS directory and running the following command:

cat /dev/null > .moot.trace