How to troubleshoot SSIM Event Collectors

book

Article ID: 177383

calendar_today

Updated On:

Products

Security Information Manager

Issue/Introduction



Symptoms
A collector or collectors that used to work have suddenly stopped sending events.


 

Resolution

Are any Statistical Events for the collector showing up in the SSIM UI?

    1. Log in to the SSIM client
    2. Click Events, right click My Queries, choose Query Wizard
    3. Choose Event Query - Next
    4. Event Details - next
    5. View - Last 15 minutes
    6. In the Query Filter section add: Product =  <collector>
    7. Execute the query and see if there are any events other than "informational".


Note that your point product must be attempting to send actual events for this to work. If a firewall doesn't have any traffic to it, there may be nothing in the logs. Similarly, an antivirus server that does not have any clients may not issue any events to the log and therefore falsely appear to be down

For help building queries see this document.

If Statistical events are received for the collector, but no actual events, then the collector is communicating with the SSIM, but the collector is not communicating with the point product. In this case look carefully at the collector configurations for anything that may be causing the issue. See below for 'collector types' for troubleshooting collector configurations. Also look at the collector log files for more detail.

If no statistical events are received then the collector has not communicated with the appliance. The appliance itself may not have connectivity or may not be receiving events. Check the Statistics page on the appliance to see if any events at all are flowing. If no events are flowing to the appliance from any collector, it is probably not a collector issue, it is an issue with the appliance.
 

  • Basic configuration checklist:
  1. Make sure the collector is properly installed. On box collector, off-box collector.
  2. Make sure a collector configuration is created and distributed to the correct machine.
  3. Make sure a sensor configuration is created and enabled.

See this document for more information about creating collector and sensor configuration.

 

  • Check Networking Connectivity
    • On an offbox collector that does not communicate with SSIM at all may not have correct networking/connectivity. If this is the case, usually you will find the Agent is not bootstrapped.
    • Ping the SSIM from the collector.
      You should be able to Ping the SSIM appliance from the collector. If you cannot ping a SSIM that is online, there is a networking problem.
    • Hosts file entry.
      It is often necessary to create hosts file entries for the SSIM on collector machines, or edit vice versa on the appliance. Take a look at this document for more information.
     
  • Are any collectors reporting correctly to the appliance?
    If some collectors are reporting to an appliance, then you know the appliance itself is working properly. Check to see if there are any differences in the networking route between collectors that are working and those that are not.
    If an appliance receives nothing from any collector, the issue is probably with the appliance itself or the networking up to the appliance. Ping the appliance from the collector machine. The appliance should reply to Ping, if it does not, then the problem is most likely networking.
  • Are any other sensors for the collector working?
    If you have multiple sensors for the same collector it is useful to know if any of the sensors are working. Enhance the query you created above to filter for only events coming from specific sensors. If some sensors are working but others are not look at the specific sensor configurations for the sensors that are working compared to those that are not. For Windows Events Collector (and other logfile sensors) these issues will often be caused by differing permissions on the collector machines.
  • Are there any other collectors on the same agent as the one that is failing? Are they working?
    In some configurations one agent may work with several collectors. If some collectors are reporting to an agent but one isn't it is very likely that the collector configuration is incorrect.
  • Check Collector Filter and Aggregator settings
    1. Collector Filter
      There are Filter settings within a collector that may prevent the collector from working as expected.
      • In the SSIM GUI click on System - Product Configurations
      • Open the Collector Configuration for the collector in question
      • Click on the Filter tab
        This is an EXCLUDE filter, any items that match the settings on this filter will NOT create events in SSIM. The default is empty, make note of anything you see listed so you can recreate it if necessary, then delete any filter settings to test if events are created properly.
    2. Click the Aggregator tab
      Many events may be aggregated into one event and cause the collector to appear to not be working properly to create events. Incorrect aggregator settings may cause you to see only one incident but find that many events are aggregated under that incident. Please see this document for more information about Aggregator.


 
  • Check the Correlation Forwarding Rule filter
    1. In the SSIM GUI click System - Server Configurations
    2. Open the Domain and click on the SSIM in the Directory Tree
    3. Click Event Forwarding Rules
    4. Open the active Correlation forwarding rule. See this document for more information on how to find the active correlation forwarding rule.
    5. Verify that the settings in "Inclusion Filter" section are correct. If any criteria are set here, only the events that match the criteria will be forwarded and create incidents.
       
  • Collector Debug mode
    All collectors have a Debug mode. Debug mode will cause the collector to make extremely verbose logs including the full text of every raw event that passes the collector. Most failures that would stop a collector from receiving events do not require DEBUG mode to isolate. All ERRORS are logged in any mode. Debug mode is especially useful for situations where events do not appear to be mapped correctly or SSIM does not appear to display or store the event properly as it will show exactly how the event arrived at the collector allowing us to determine if the collector itself is causing the problem, or if the problem existed at the time the event arrived at the collector.
    ESDiag is a great way to put a collector into Debug mode. See below for more information on ESDiag
    If ESdiag is not available a collector can be put into Debug mode manually by following the steps in this document.
  • ESDiag
    The Event Stream diagnostic Utility can be very useful for debugging collectors. It can be used to place a collector into Debug mode as well as gather logs and configuration files easily. For more information see this document. ESDIAG is especially handy because it grabs the sensor configuration files for evaluation by support.
  • If the agent is only processing events from one collector you can check to see if the agent is processing events using the agent management script. For information about the agent management script please see this document.
    1. Start the agent management script
    2. Enter '1' to check the agent status, make a note of the event counts by looking into "queue status" section
    3. Wait a moment
    4. Repeat and check if the event counts have changed


If you find that the agent is receiving events then you know that all the pieces up to the agent are working correctly so you can focus troubleshooting on the connection between the Agent/Collector and the appliance. If the agent does not receive any events trace backwards to locate the point of failure. techniques for doing this depends on the type of collector you are working with. See below for more detail in tracking these problems down.
 

  • Restart the agent service using agent management script
    1. Start the agent management script
    2. For agent 4.7.0,option 9 and 10 are for starting and stopping the agent respectively. For agent 4.7.1, option 10 and 11 are for starting and stopping the agent respectively.

Try this if you find that the event service is receiving events but does not forward them. Do not do this on a SSIM appliance.
 

  • Check the SSIM Databases
    If none of your collectors are receiving events your archives may be full.
    1. Log in to the SSIM client as an administrator
    2. Click Statistics
    3. On the System tab, check the Database Space indicators. If any portion of your database is indicating that it is full, you must purge the database or acquire additional storage.
  • Verify the agent version
    If the agent is unstable and drops out of memory periodically requiring restart you may find it is the older agent and in need of update.
    1. Open the sesa-agent log.
    2. Find the most recent "Symantec Event Agent is starting..." entry
    3. Below the Agent is starting you will find the agent version. You should see version 4.7.1.21
    If you find an earlier version, you must update the agent version by following the steps in this document.


Check for errors in the sesa-agent.log, and <collector name> log, search the Knowledge base for any error you find.

    • In Windows:
      1. Program Files\Symantec\Event Agent\logs\sesa-agent.log
      2. Program Files\Symantec\Event Agent\logs\<collectorname>.log
    • In Linux:
      1. /opt/Symantec/sesa/Agent/logs/sesa-agent.log
      2. /opt/Symantec/sesa/Agent/logs/<collectorname>.log

      Go to the bottom of the log and search backwards for the message indicating the collector is starting. Look for any errors after the most recent entry in the logs indicating "Collector started" or "Symantec Event Agent is starting". (Errors prior to the "start" events may refer to a previous configuration and may not apply to the current problem. If you find any errors search the Symantec Knowledge base for the text of the error that you see.
      For example: An error stating: "No Valid Sensors in Working group" indicates a particular problem that has its own solution.

      Once you have worked through all the issues with the Agent, you will find that working with each collector is a little different. You will have different things to look for on the "Windows Event Collector" compared to, for example, Symantec Endpoint protection. Read through any errors you find in the log and attempt to identify a cause.

      Additional collector troubleshooting depends on the type of collector involved. Specific collector may have their own troubleshooting guide or specific issues. It is important to search for any articles pertaining to the particular collector that is causing problems.


There are 3 main types of collectors:
 

  1. Syslog Collectors
      • These collectors read events from a point product on a particular port. Syslog collectors are often installed on the appliance itself, and are usually integrated with the Syslog Director.
      • If you are working on a collector that works with the Syslog Director please see this document for troubleshooting steps specific to the Syslog Director.
      • Examples of Syslog collectors are: SAV, Cisco Pix, Cisco IOS, Snort, Checkpoint, Juniper NSM, Juniper Netscreen, Kiwi, Unix, Linux
         
      • Verify the point product is sending its logs to the collector .
        • For on-box collectors or collectors on Linux use TCPDUMP to verify that the point product is sending its logs to that device
      1. Log in to the SSIM client
      2. Click System - Product Configurations
      3. Open the Collector that you are concerned about
      4. Click on the Syslog Sensor tab and verify the Port
      5. SSH to the Linux box with the collector installed
      6. Log in as db2admin then type su - and provide the root password.
      • Type tcpdump port <port number>
        You should see messages indicating the traffic. If you do not see anything indicating traffic then the point product is not properly configured to send its logs to the collector or there is a firewall blocking the traffic. Note that you can only use tcpdump to verify external traffic coming in to the SSIM, tcpdump does not capture internal port traffic.
      • If TCPDUMP does not run on your Linux box it is either in a different path or is not installed. Please see your Linux administrator for more assistance with this problem. If tcpdump does not run on the appliance ether you did not "su -" before you typed TCPDUMP, or the appliance is not properly configured.
      • On Windows you will need a 3rd party program to perform tcpdump functions. WinDump is one example of such a utility.
      • Use NETSTAT to verify that the collector is listening on its port
      1. SSH to the Linux box with the collector installed
      2. Log in as db2admin then type su - and provide the root password.
      3. Type netstat -an |grep <port #>
        You should see the collector listening on its port. If you do not see the collector listening on its port then the collector configuration is not correct.
    1. Logfile Collectors
      • These collectors read a log file that is generated by a point product. Examples include Windows Event Collector, Nessus
      1. Check the permissions of the Service that is running the agent. See this document for more information.
      2. Check that the path to the log file is accurate
    2. Database Collectors
      • These collectors read a database. ISS Site Protector, Symantec Endpoint Protection, Symantec Critical System Protection
      1. Check the location and version of the database drivers match the sensor settings
      2. Verify the connection string used to connect to the database