Events are not being forwarded to Splunk

Products

• Endpoint Detection and Response
• Advanced Threat Protection Platform

Issue/Introduction

• Events are not being forwarded to Splunk from Symantec Endpoint Detection and Response (SEDR) or Advanced Threat Protection (ATP).
• There is a delay in event forwarding to Splunk from SEDR or ATP.

From the atp-splunk_connector.log you see the following error:

Upload failed with httpcode: [503], status code: [9], Reason: [Server is busy]

Cause

This can be caused by overloading the Splunk indexer.

Resolution

This can be resolved on the Splunk side by switching from a single Splunk data collection node to a distributed deployment configuration.

This condition may be solved by lowering the bitrate to a value that the Splunk server can handle:

Additional Information

It is possible the issue is different, in which case we recommend referring to Article Id: 215022 if necessary.  This article titled EDR stops sending events to Splunk https://knowledge.broadcom.com/external/article/215022 refers to an issue where the time range trying to be collected may be too large and EDR may be purging the data before it can be forwarded to Splunk.

NOTE: It is possible that the 503 errors cause SEDR to reach the 2 Million event limit on the internal database queries. If this occurs, it is possible that SEDR will begin transmitting duplicate event data. You will need to reboot the appliance to fix this condition.