Endpoint Protection fails to purge the Data Recorder database to the configured size.

Products

• Endpoint Protection
• Advanced Threat Protection Platform
• Endpoint Detection and Response
• Endpoint Protection with Endpoint Detection and Response
• Endpoint Security Complete

Issue/Introduction

When using Symantec Endpoint Detection and Response's (SEDR) Data Recorder feature, the Endpoint Protection (SEP) client fails to honor the configured Data Recorder database size.  Files in the following directory will consume more drive space than has been configured in the policy:

C:\ProgramData\Symantec\Symantec Endpoint Protection\CurrentVersion\Data\EDR\localdatastore

 

The following additional symptoms may also occur:

- Errors related to disk size, disk capacity, or disk full may eventually appear in the Application Log, System Log of Windows Event Viewer. 

Environment

Endpoint Protection 14.X

Advanced Threat Protection 3.x or Symantec Endpoint Detection and Response 4.x.  

Integrated Cyber Defense Management (ICDm)

Cause

This issue can have multiple causes including:

1. The endpoint is generating so many events that eventually the endpoint is unable to delete the localdatastore subfolder until the next day.
2. The scheduled purge job on the client is scheduled in the future due to a change in Windows time.

Resolution

Identify if the issue is related to the endpoint generating too many events, or scheduling:

1. Review the subfolders in C:\ProgramData\Symantec\Symantec Endpoint Protection<current version>\Data\EDR\localdatastore
2. If there is a folder that ends in 998 (i.e. 20230901998) then see "Tuning the EDR policy" below
3. If there are no folders that end in there a large number of subfolders, none ending in 998, then restart the SEP client

Tuning the EDR policy:

1. Perform a full dump of the endpoint:
   1. EDR On-Prem:
      1. Log in to the EDR GUI
      2. Navigate to Search -> Database -> Entities
      3. In the "Entity Search" type the name of the endpoint and press
      4. In the results click on the name of the endpoint
      5. In the tab that opens, click "Full Dump", then click "Ok"
   2. ICDm:
      1. Log in to the ICDm console
      2. Navigate to Devices -> Managed Devices
      3. In the "Filter by" box type Device Name: <MY_DEVICE_NAME> where <MY_DEVICE_NAME> is the name of the endpoint
      4. In the results page on click on the name of the endpoint
      5. In the window that opens click More Actions -> Full Dump
      6. Select "Collect all the EAR events" and click "Yes"
2. Once completed, review the full dump:
   1. EDR On-Prem:
      1. Log in to the EDR GUI
      2. Navigate to Search -> Endpoint
      3. Click on the full dump for the endpoint
   2. ICDm:
      1. Log in to the ICDm console
      2. Navigate to Investigate -> Endpoint
      3. Click on the full dump for the endpoint
3. Using the event search, identify events that occur frequently
   1. EDR On-Prem - https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/endpoint-detection-and-response/4-8/about-the-ways-to-search-for-indicators-of-comprom-v115770112-d38e14827/how-to-write-successful-endpoint-search-expression-v116088065-d38e14973.html
   2. ICDm - https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/endpoint-security/sescloud/Alerts-and-Events/investigation-page-overview-v134374740-d38e87486/query-and-filter-operators-by-data-type-v134689952-d38e88796.html
4. Create Endpoint Activity Recorder (EAR) rules for events that occur frequently and have been identified as events that are of little to no interest.
   1. EDR On-Prem - https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/endpoint-detection-and-response/4-8/about-policies-v115121914-d38e34170/Creating-a-Recorder-policy.html
   2. ICDm - https://techdocs.broadcom.com/us/en/symantec-security-software/endpoint-security-and-management/endpoint-security/sescloud/Endpoint-Detection-and-Response/default-detection-and-response-policy-details-v134153921-d38e87208/EDR-policy-do-not-submit.html

Additional Information

Within EDR appliance, the number of events to send in each batch can be adjusted on the SEPM Policy on Settings> Global. Expect that an average client sends about 2 events per minute. Less than that (fewer than 10 events per 5 minutes) can back up the clients. More than that (greater than 15 events per 5 minutes) increases the load on your server during peak performance. Ensure that your system isn't already fully loaded if you increase the batch size significantly.