Syslog event from the Rsyslog server are sometime days behind in Spectrum

Products

• Spectrum
• Network Observability

Issue/Introduction

Syslog event from the Rsyslog server are sometime days behind in Spectrum

Cause

It was identified there were a few devices sending thousands of syslog messages to the Rsyslog server over a short period of time. This caused a delay in forwarding these syslog messages from the Rsyslog server to Spectrum.

Resolution

Identify the offending devices that are sending large numbers of syslog messages to the Rsyslog server and resolve it at the device.

You can grep the syslog logs to id the problematic devices for top 10 "chatty" devices e.g. 

grep  "Jan  20" <processedsev3syslog.log> > /tmp/sev3log
$ wc -l /tmp/sev3log
awk '{print $4}' /tmp/sev3log |sort |uniq -c | sort -nr | head -n 10  

We should then check the discarded log for the worst offenders in the above list and scan the active discardedmessages.log e.g.

$ grep  <discardedmessages> | grep "Jan  20"  | wc -l
$ grep <device name> <discardedmessages.log> | grep "Jan  20"  | wc -l

 

 

Additional Information

Are there any events/alarms on RsyslogServer model related to trap/event discards?
A packet capture on SS machine to confirm Rsyslog server is sending traps will help.  
If traps itself are not coming to SS, then we have to investigate from rsyslog server/package side.
Consider to modify TrapStormLength to 10 and traps_per_sec_storm_threshold to 50 on VNM model and RsyslogServer systemEDGE host model and check weather it brings any change?
When there are no events on systemEdge model, are they seeing any increase in values for Traps Ignored or Traps discarded in storm variables on VNM model?
This will allow us to know whether it is SS issue or environment issue.