Endpoint Detection and Response stops sending events to Splunk

Products

• Endpoint Detection and Response

Issue/Introduction

Symantec Endpoint Detection and Response (EDR) stopped sending events to Splunk log host.

Sometimes the forwarding of events to Splunk freezes and EDR stops sending logs.

Environment

• SEP Clients are enrolled with SEDR appliance
• Symantec EDR Add-On for Splunk.
• Symantec EDR App for Splunk.
• Symantec EDR Splunk Integration reports a healthy connection.

Cause

The SEDR token appears to expire frequently.  Even though the token expiration problem may eventually appear to resolve itself the Splunk connector will still try to run the same query and never forward any recent events that have occurred.

Resolution

Workaround:

As a workaround disable the Splunk integration and re-enable it. After this EDR will send events again.

Solution:

Method 1 - Update the time range filter to a more recent timestamp

1. In the EDR web user interface go to Settings > Data Sharing > Splunk Integration
2. Click [Show Filters] button and find 'Event Forwarded From' label
3. Click [Edit] and [Ok], then update 'Event Forwarded From' time to a more recent date. For example, choose yesterday's date if possible or choose the date closest to today's current date that the system allows.

Additional Information

The output of the CLI command df -h may show a high amount of disk usage for the partition that stores events.

For example, the output of df -h might show the elasticsearch partition at 85% of capacity. When the elasticsearch partition reaches 80%, the emergency purge checks and purges data every 15 minutes until the disk capacity reaches 75%.  In this scenario, EDR may be purging events so quickly that the data set EDR is trying to forward no longer exists.

 

Events are still not being forwarded to Splunk:

It is possible that the issue being seen is not related to the issues and solutions presented in this case.  If the logs instead show that the upload of data failed or that the server was busy when EDR is attempting to upload events to Splunk then we recommend referring to article 175221, https://knowledge.broadcom.com/external/article/175221