On all the SSIM appliances the Event Services queue is showing red and 100% full
There might be more than one cause:
1. Poor network connectivity to the Directory SSIM
2. Upstream SSIM is to busy to accept more events due to:
3. Summarizers are enabled on the Archiver
4. The Event Service is overwhelmed due to incorrectly parsed events
4. Degraded RAID array due to failed drive causing slow writing to Disk for archiving
Check the Network Interface for Speed and Duplex as well as for an excessive number of dropped packets with the ifconfig command.
To check for dropped packets on the Directory SSIM:
If there is an excessive amount of dropped packets or errors, make sure the Speed and Duplex are right. Also check the network cable and on the health/configuration of the switch/router the SSIM is plugged into.
To check the speed and duplex on the Directory SSIM:
The Duplex must always be Full. If the Duplex displays Half, then make sure the Switch the SSIM is plugged into has the same Speed and Duplex settings the SSIM's Network Interface is set for.
Solution for Cause #2
The ICE Service is having problems:
Follow the Instructions in the KB article Troubleshooting Correlation service issues and why correlation engine sometimes stops creating new incidents under Technical Information for how to clear the ICE queues.
Correlator is overwhelmed due to a poorly configured Correlation Rule or Incident Forwarding Rule:
Large number of Open Incidents causing slowness on the Correlator:
If the total number of Open Incidents is close to 25000 or more it is likely the amount of system memory being used by them is robbing other services of memory. this will impact the other services ability to function and impact performance.
Solution for Cause #3
Summarizers are a legacy functionality from earlier versions of SSIM when there were no State Collectors and no Trending Query function. Most point products which would apply to using Summarizers for state type information all have State Collectors now.
In SSIM 4.7 Symantec introduced Trending Queries which allow queries to span time ranges and provide Trending information for the data you specified in the criteria.
It is recommended to disable all Summarizers due to the fact that they have the potential to have a negative impact on performance and due to the introduction of Trending Queries and the use of State collectors.
Solution for Cause #4
In the eventservice.log file when their is a significant number of lines reporting data in a core field as invalid. The most common field reported is the IP Address field.
A couple of examples:
2012-03-28 10:42:49,228 1897211183 [Normalizer] WARN com.symantec.sim.eventservice.util.Networks - Invalid IP address: ResolveIP(BackRef(source_host_name))
2012-04-04 20:15:24,829 636118481 [Normalizer] WARN com.symantec.sim.eventservice.util.Networks - Invalid IP address: ::ffff:10.200.7.39
Solution for Cause #5
If the RAID array on SSIM or configured External Storage Device is not optimal, then it is degraded and there may be a disk that needs replaced/reseated and built back into the array. When a RAID array is degraded it cannot perform as fast and depending on the typical event rate in the SSIM environment may cause queues to fill up.
Find the reason for the degraded RAID array and resolve it.