Incident Queue Grows and does not clear on Network Prevent For Email Detection Server
Article ID: 202575
Updated On:
Products
Issue/Introduction
On a Network Prevent for Email Detection server, it is observed that the incident queue grows continuously without clearing. It is observed that incidents on Enforce are processing, and other detection servers for Network Prevent for Email may function normally and not exhibit a building Incident Queue.
Environment
Release : DLP 15.x
Component : Network Prevent for Email Network Prevent for Web
Cause
IncidentWriter makes a loopback connection via BoxMonitor to facilitate transfer of Incident Data to Enforce. When investigating IncidentWriter log, the following appears:
Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.transport.ChannelManager
Method: checkOperationTimeouts
Level: INFO
Message: timing out operation: com.vontu.communication.transport.ConnectWaitOperation:1603983762862:boxmonitor:controller-server:<IPADDRESS>:8100:null
Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.transport.ChannelManager
Method: processOperationResult
Level: INFO
Message: Operation com.vontu.communication.transport.ConnectWaitOperation:1603983762862:boxmonitor:controller-server:<IPADDRESS>:null failed with exception: com.vontu.communication.transport.exception.OperationTimeoutException
Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.dataflow.ShippingTask
Method: run
Level: WARNING
Message: ShippingTask(boxmonitor:controller-server, Incident, l1600605403668.idc, 16320): Requested connection to address <boxmonitor:controller-server> could not be established!
Subsequently, investigation of BoxMonitor reveals the following:
Oct 29, 2020 11:02:01 AM com.vontu.communication.transport.SSLAcceptOperation invokeAcceptOp
INFO: SSLAccept from: incidentwriter:controller-server completed successfully. This connection will NOT use SSL
Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager checkOperationTimeouts
INFO: timing out operation: com.vontu.communication.transport.AcceptWrapperOperation:1603983721502:null:com.vontu.communication.transport.SessionIdentifier@3953b86d
Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager processOperationResult
INFO: Operation com.vontu.communication.transport.AcceptWrapperOperation:1603983721502:null:com.vontu.communication.transport.SessionIdentifier@3953b86d failed with exception: com.vontu.communication.transport.exception.OperationTimeoutException
Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager handleOperationFailure
INFO: removing session from cache: com.vontu.communication.transport.SessionIdentifier@3953b86d for id: null
Based on above output, the "AcceptOperation/AcceptWrapperOperation" object is timing out as it's passing 1 minute, 60,000ms, as noted by the timestamps in bold.
Resolution
The timeout being surpassed is governed by the "acceptTimeout" property in communication.properties. The default timeout is 60000, and this value is in milliseconds (ms).
To resolve, raise the acceptTimeout in communication.properties, located in the config folder in the DLP installation directory under
■ Windows:
<drive>:\Program Files\Symantec\DataLossPrevention\DetectionServer\<version>\Protect\Config
■ Linux:
/opt/Symantec/DataLossPrevention/DetectionServer/<version>\Protect\Config
Broadcom recommends doubling the timeout to 120000 and then re-testing the behavior, and continuing to double the timeout value until success is achieved.
Example (from)
# Timeout (milliseconds) for accept operations.
# Used by the BoxMonitor only.
acceptTimeout = 60000
Example (to)
# Timeout (milliseconds) for accept operations.
# Used by the BoxMonitor only.
acceptTimeout = 120000
Once Communication.properties has been updated, restart the Detection Server Service to update BoxMonitor with the new acceptTimeout value.
Additional Information
The same resolution typically applies when you see the following warning in a detection server's BoxMonitor logs:
com.vontu.communication.dataflow.ShippingTask run WARNING: ShippingTask(controller-server, Structured Data Push, Statistics_C , 40345): Requested connection to address <controller-server> could not be established!
Another sign that acceptTimeout needs to be raised on a detection server is when you see the following in MonitorController logs:
com.vontu.monitor.communication.configset.dataflow.ShipmentErrorHandler recover
INFO: Sent the local state to the remote config set <fqdn> <DetectionServer>:filereader:EMDIProfile.
Feedback
