Troubleshooting "WARNING at ItcQdItem.cc(374)" in DX NetOps Spectrum

Products

• Network Observability
• Spectrum

Issue/Introduction

Users may encounter frequent warning messages within the VNM.OUT log file. These entries typically occur every minute and indicate that internal requests are not completing within the expected timeframe.

 

Example Log Entry:

WARNING at ItcQdItem.cc(374): Request 687559678 accepted, but not complete after 60000ms, cs delta = 11513149. Triggering moot trace.

Environment

Spectrum 24.x

Cause

The error is triggered when a request remains incomplete after 60,000ms (60 seconds).
In this specific case, investigations revealed the delay was caused by high system load related to Global Collection (GC) updates and intensive search criteria. Specifically, a search returning over 210,000 entities was identified as a primary contributor to performance bottlenecks.

Symptoms and Observed Log Files:

• VNM.OUT: Contains the primary warning message and associated debug traces.

• moot.trace: Automatically triggered by the warning; this file can grow excessively (e.g., 13GB or larger) as the system attempts to trace the stalled request.

• SS_pstack.OUT: Used for troubleshooting thread activity to identify which process is consuming CPU resources.

• Performance Alarms: Users may see "CONTACT LOST TO SECONDARY SPECTROSERVER" alarms or delays in alarm synchronization to Performance Management (PM).

Resolution

1. Optimization of Global Collections

• Search Refinement: Review complex searches involving the "Port" class (Attribute ID 0x11ee8).

• Scheduling: Change "Real Time" Global Collections to "Scheduled" intervals, preferably every 3 hours, to reduce continuous processing overhead.

   

2. SpectroSERVER Configuration (.vnmrc)

Update the .vnmrc file with the following parameters to reduce polling stress and SNMPv3 lock issues:

• sync_all_ext_interface_attrs=false

• int_copyattr_enable=true

• failed_cache_for_unmanaged_trap=true

• tds_local_v3_profiles_store=true

   

3. Thread Management

If threads are peaking, adjust the following limits in the VNM Model (SpectroSERVER Control):

• Max Available Poll Threads: Increase to 1500.

• IH Timer Threads: Increase to 960.

• Total Work Threads: Increase to 6000.

   

4. Maintenance

• Trace File Management: If no actual performance dumps (.dmp files) exist in $SPECROOT/SS/ or $SPECROOT/SS/support/, the moot.trace file can be safely deleted or moved to a different location to save disk space.

Additional Information

These warnings are considered diagnostic noise in some scenarios and are formally addressed in Release 25.4.1 or higher, where these messages will no longer appear.

Spectrum Global Collection for port models are taking a long time to populate.

Spectrum VNM.OUT file notes Triggering moot trace over and over again