Incidents queueing up on DLP detection servers with High CPU usage and Detection server randomly going into unknown state

Products

• Data Loss Prevention Core Package
• Data Loss Prevention
• Data Loss Prevention Plus Suite
• Data Loss Prevention Network Prevent for Email
• Data Loss Prevention Network Monitor and Prevent for Web
• Data Loss Prevention Network Monitor and Prevent for Email and Web
• Data Loss Prevention Network Monitor and Prevent for Email
• Data Loss Prevention Network Monitor
• Data Loss Prevention Network Discover
• Data Loss Prevention Enterprise Suite
• Data Loss Prevention Enforce
• Data Loss Prevention Endpoint Prevent
• Data Loss Prevention Endpoint Discover
• Data Loss Prevention Discover Suite

Issue/Introduction

Incidents have begun to queue up on a DLP detection server(s) according to the "Incident Queue" column on the System -> Servers and Detectors -> Overview page in the DLP Enforce Console.

Constant high CPU is observed on the detection server.

The detection server will also go into "unknown" state on the Enforce console randomly.

The SymantecDLPEnforceConnector logs on the detection server contains the following log entries with name_of_incident_file being replaced with the name of the actual corrupt incident file in the folder.

com.symantec.dlp.storage.spi.impl.persistence.local.LocalDataFile getMetaDataFromKeys
INFO: Unable to get metadata of local file E:\DataLossPrevention\DetectionServer\Account-storage\EnforceSlot-uuid\INCIDENTS\name_of_incident_file= Attempt #1: java.io.EOFException

AND

com.symantec.dlp.storageandnotification.StorageInstrumentationImpl logSevere
SEVERE: Exception while getting dataReader for file name_of_incident_file= in folder Account-storage/EnforceSlot-uuid/INCIDENTS.
com.symantec.dlp.storageandnotification.exceptions.RecoverableStorageSecurityException: Cannot decrypt file, no file encryption key found.
    at com.symantec.dlp.storage.spi.impl.security.EncryptedDataFile.getFileEncryptingKeyContainer(EncryptedDataFile.java:290)
    at com.symantec.dlp.storage.spi.impl.security.EncryptedDataFile.buildFileEncryptingKeyContainerForDecrypt(EncryptedDataFile.java:206)
    at com.symantec.dlp.storage.spi.impl.security.EncryptedDataFile.getInputReader(EncryptedDataFile.java:135)

OR

Class: com.symantec.dlp.storageinfrastructure.databuffer.DataBufferReaderFromStorageInputReader
Method: handleIOException
Level: WARNING
Message:  Reading data from storage failed: 
com.symantec.dlp.storageandnotification.exceptions.RecoverableStorageSecurityException: Something went wrong while decrypting and reading data.File: name_of_incident_file , Folder: INCIDENTS at com.symantec.dlp.storage.spi.impl.security.EncryptedInputReader.read(EncryptedInputReader.java:134) at  com.symantec.dlp.communications.commlayer.impl.DataConnectionImpl.getDataToEnqueueFromReplicatorAndRegisterThemAgainIfInterested(DataConnectionImpl.java:1424) at com.symantec.dlp.communications.commlayer.impl.DataConnectionImpl.enqueueOneOutboundDataFrame(DataConnectionImpl.java:1376) at com.symantec.dlp.communications.commlayer.impl.DataConnectionImpl.enqueueOutboundDataForReplicators(DataConnectionImpl.java:1322) at 

Caused by: java.io.IOException: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. at com.symantec.dlp.storage.spi.impl.security.EncryptedInputReader.read(EncryptedInputReader.java:109) ... 14 more

Caused by: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 17 more

Environment

DLP 16.1, 25.1

Cause

This issue is caused by an incident file becoming corrupt in the ...DataLossPrevention\DetectionServer\Account-storage\EnforceSlot-uuid\INCIDENTS folder on the detection server.

Resolution

Delete the corrupt incident files mentioned in the SymantecDLPEnforceConnector log from the following folder on the Detection Server. Delete both the file ending in "=" and the file ending in "=.mtd" with the name specified in the log file.

This is the default path, refer to the SymantecDLPEnforceConnector log for the actual path on your detection server:

C:\ProgramData\Symantec\DataLossPrevention\DetectionServer\Account-storage\EnforceSlot-uuid\INCIDENTS