Massive Issues with event processing since update to version 10.4.2.2

Products

• CA Spectrum

Issue/Introduction

Problems seen with the event processing in Spectrum occurring after upgrade with alarm views not correctly displaying alarms, some being false, and others not appearing. 
 
Some examples are:

• Link condition is suppressed with oper_status=up.
  
  
• Event 0x10d66 occurs many thousand times, but the interfaces are always up.
  
  
• Event 0x10d67 (started responding to polls) is missing on these ports.  
  
  
• Even if event 0x10d10 (status good) occurs, the interface condition often remains suppressed
  
  
• Similar behavior for application models: Many "stopped responding to polls (0x10d09)" followed by a "started responding to polls (0x10d0b)" events, although the SNMP agent is contactable.
  
  
• Also see an alarm “device has been rebooted” without any evidence of a reboot with the uptime of the device being greater than the alarm timeframe.
  
  
• Since the Spectrum update we also see “Link Aggregation Condition is critical” alarms, without any port down.

Environment

Spectrum Release : 10.4.2.2.
Component : Spectrum Core / SpectroSERVER
RedHat Linux

Cause

From the output of a "netstat -suna" we could see many UDP packet errors being received, this often means that the UDP traffic is arriving too fast for the network buffer to handle it and the UDP packets are being dropped.

Resolution

1. Increased the DCM timeout value of all devices  to  6 sec
   
   
2. Updated the sysctl socket buffers configuration to 8MB using below commands .
   
   sysctl -w net.core.rmem_default=8388608
sysctl -w net.core.wmem_default=8388608
sysctl -w net.core.rmem_max=8388608
sysctl -w net.core.wmem_max=8388608
sysctl -w net.core.netdev_max_backlog=2000

       
These changes can be made permanent by adding the above configuration entries in /etc/sysctl.conf.  Spectrum version upgrade scripts will take care of these configurations in future spectrum releases
       
   

Additional Information

Further information can be reviewed here.

 

Linux Kernel Optimizations