Spectrum SNMPv3 False Management Agent Lost (MAL) Alarms - DX NetOps Spectrum

Products

• Spectrum
• Network Observability

Issue/Introduction

When managing SNMPv3 devices in Spectrum, you may experience a false Management Agent Lost alarm. Using the "Reconfiguration -> Reset SNMPv3 Authentication" does not appear to work or doesn't resolve the problem.  When you run a sniffer trace you see that either a different engineid is being sent to Spectrum or Spectrum is "looping" in the communication (the SS sends a blank request for the v3 info, the agent sends the report packet back.  Instead of Spectrum then using that data to send the next request, Spectrum sends a blank request again for the v3 info.  This cycle repeats).

If you stop and restart the SpectroSERVER the issue is cleared.  This is because the entire SNMPv3 cache is cleared and rebuilt when the SS is restarted.

Symptoms

• "MANAGEMENT AGENT LOST" alarm appears in the OneClick console.
• Devices are reachable via CLI and respond correctly to SNMPv3 queries.
• Reconfiguring/Resetting SNMPv3 Authentication temporarily clears the alarm, but it reappears after a short time.
• SpectroSERVER logs may indicate issues with SNMPv3 profile processing or cache synchronization.

Environment

Release: DX NetOps Spectrum 
Component: SPCCSS

Cause

The issue is caused by inefficient SNMPv3 profile matching and SNMPv3 remote cache synchronization defects in older versions of Spectrum 25.4.x. These defects result in the SpectroSERVER failing to process responses correctly, triggering false MAL alarms.

 

The problem is due to running action code 0x10330 debugging the issue:

./update action=0x10330

There are two tables in the Spectrum cache that store the IP/engineid.  When using action code 0x10330 to clear the v3 cache, it only removes the engine id from one table.

 

 

Resolution

Phase 1: Upgrade

1. Upgrade to DX NetOps Spectrum version 25.4.9 or higher (25.4.11 is recommended). These versions contain the permanent code fixes for SNMPv3 profile processing.

Phase 2: Post-Upgrade Configuration Cleanup

If you previously applied workaround parameters to the $SPECROOT/SS/.vnmrc file, they must be removed after the upgrade to ensure optimal product performance.

1. Create a backup of your current $SPECROOT/SS/.vnmrc file.
2. Remove the following lines if present:
   • failed_cache_for_unmanaged_trap=true
   • tds_local_v3_profiles_store=true
   • v3_profile_hash_factor=16384
   • delete_invalid_empty_profiles=false
   • delete_invalid_profiles=false
   • delete_unknown_report_type_profiles=false
3. Restart the SpectroSERVER process to apply the changes.

Phase 3: Manual Cache Flush (If alarms persist)

If alarms persist after the upgrade and cleanup, manually flush the SNMPv3 cache using the CLI:

1. Log into the SpectroSERVER host.
2. If on Windows, run bash -login.
3. Navigate to the $SPECROOT/vnmsh directory.
4. Start a CLI session: ./connect
5. Run the update command (replace mh= with your model handle): ./update action=0x10333 mh=0x####

 

A new action code of 0x10333 has been introduced that will properly remove all engineid entries from the cache.

If there is an SNMPv3 issue you need to troubleshoot, you should first use the "Reset SNMPv3 Authentication". If that does not work, then you can use the update action code of 0x10333 command in CLI:

./update action=0x10333 mh=<mh_of_VNM>

This will clear the SNMPv3 cache.  At any time you can view the cache using action code 0x10331:

./update action=0x10331 mh=<mh_of_VNM>

If you are unable to upgrade Spectrum, then to resolve this issue you need to stop and restart the SpectroSERVER which flushes and rebuilds the SNMPv3 cache.

Additional Information

Please make sure you are on an updated version of Spectrum and have installed the necessary patches as well for the SNMPv3 authentication reset failure:

https://knowledge.broadcom.com/external/article?articleId=7010