Build up of .err files observed in SEPM\Data\Inbox\Log directories (traffic, security, system, etc)

Products

• Endpoint Protection

Issue/Introduction

Numerous .tmp.dat.err files are found in SEPM\Data\Inbox\Log\ directories (traffic, security, system, etc) and are building up and eating away disk space and causing poor SEPM performance / slowness.

Over time the Symantec Endpoint Protection clients (SEP) forward events to the SEPM manager for processing and insertion into the SEPM database.  The SEPM manager receives these events from each client, and then creates a .tmp.dat file (comprising of events from many different systems), in preparation for processing and insertion into the SEPM database.  

Generally once the SEPM manager processes the .tmp.dat files successfully it will drop them from the list and move on to the next one. (freeing up disk space in the process)
This process is generally quick, which helps the SEPM manager keep up with all other management tasks. 

Environment

Symantec Endpoint Protection Manager (SEPM) 
Symantec Endpoint Protection Client (SEP)

Cause

Occasionally the SEPM manager receives "bad" events from SEP clients, which causes the SEPM manager to switch from its default processing mode: Batch mode using bcp.exe (which is faster), to batch mode without bcp.exe (much slower) to import each event one at a time in the .tmp.dat file, when this happens its common to see the following results: 

- Numerous .ERR files left over after successful processing in SEPM\Data\Inbox\Log\ directories (traffic, security, system, etc)
- Increased disk space utilization
- Slower than normal SEPM performance

NOTE: Per current design when the SEPM manager recognizes "bad" events are present in a .tmp.dat file, it will still insert all "good" events into the SEPM database, but it will use batch mode w/out bcp.exe, as a result its common to see slower processing and will cause the .err files to still show afterwards, thus causing an increase in disk space utilization.  

Resolution

To quickly address this issue do the following: 
#1. Stop the SEPM manager services

• Symantec Endpoint Protection Manager
• Symantec Endpoint Protection Manager API Service
• Symantec Endpoint Protection Manager Webserver

#2. Delete the .ERR files still present.
#3. Restart the SEPM manager services, and then contact Broadcom Technical Support.  

To help Support in determining why the SEP clients are sending up "bad" events to the SEPM manager which as a results is generating .ERR files within your environment kindly do the following: 
a. Kindly collect a SymDiag from the SEPM manager (where the .ERR files are being generated) per kb: https://knowledge.broadcom.com/external/article/157777

b. If the SymDiag is taking too long, or fails in the middle then do the following: 
#1. Navigate to the SEPM\Tools folder and run the collectLog.cmd tool to gather the data. 
#2. Kindly collect a dozen or more of the .ERR files (from SEPM\Data\Inbox\Log\ directories (traffic, security, system, etc)), zip them up.
#3. Open a case with Broadcom Technical Support, and upload the .ERR files, and CollectLog output to the case for further review.