search cancel

Incident Queue Grows and does not clear on Network Prevent For Email Detection Server

book

Article ID: 202575

calendar_today

Updated On:

Products

Data Loss Prevention Network Monitor and Prevent for Email and Web

Issue/Introduction

On a Network Prevent for Email Detection server, it is observed that the incident queue grows continuously without clearing. It is observed that incidents on Enforce are processing, and other detection servers for Network Prevent for Email may function normally and not exhibit a building Incident Queue.

Environment

Release : DLP 15.x

Component : Network Prevent for Email Network Prevent for Web

Cause

IncidentWriter makes a loopback connection via BoxMonitor to facilitate transfer of Incident Data to Enforce. When investigating IncidentWriter log, the following appears:


Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.transport.ChannelManager
Method: checkOperationTimeouts
Level: INFO
Message:  timing out operation: com.vontu.communication.transport.ConnectWaitOperation:1603983762862:boxmonitor:controller-server:10.110.216.34:8100:null

Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.transport.ChannelManager
Method: processOperationResult
Level: INFO
Message:  Operation com.vontu.communication.transport.ConnectWaitOperation:1603983762862:boxmonitor:controller-server:10.110.216.34:8100:null failed with exception: com.vontu.communication.transport.exception.OperationTimeoutException

Date: 10/29/2020 11:06:42 AM
Class: com.vontu.communication.dataflow.ShippingTask
Method: run
Level: WARNING
Message:  ShippingTask(boxmonitor:controller-server, Incident, l1600605403668.idc, 16320): Requested connection to address <boxmonitor:controller-server> could not be established!

Subsequently, investigation of BoxMonitor reveals the following:

Oct 29, 2020 11:02:01 AM com.vontu.communication.transport.SSLAcceptOperation invokeAcceptOp
INFO: SSLAccept from: incidentwriter:controller-server completed successfully. This connection will NOT use SSL

Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager checkOperationTimeouts
INFO: timing out operation: com.vontu.communication.transport.AcceptWrapperOperation:1603983721502:null:[email protected]

Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager processOperationResult
INFO: Operation com.vontu.communication.transport.AcceptWrapperOperation:1603983721502:null:[email protected] failed with exception: com.vontu.communication.transport.exception.OperationTimeoutException

Oct 29, 2020 11:03:01 AM com.vontu.communication.transport.ChannelManager handleOperationFailure
INFO: removing session from cache: [email protected] for id: null

Based on above output, the "AcceptOperation/AcceptWrapperOperation" object is timing out as it's passing 1 minute, 60,000ms, as noted by the timestamps in bold.

Resolution

The timeout being surpassed is governed by the "acceptTimeout" property in communication.properties.  The default timeout is 60000, and this value is in milliseconds (ms).

To resolve, raise the acceptTimeout in communication.properties, located in the config folder in the DLP installation directory under

■ Windows:
<drive>:\Program Files\Symantec\DataLossPrevention\EnforceServer\<version>\Protect\Config
■ Linux:
/opt/Symantec/DataLossPrevention/EnforceServer/<version>\Protect\Config

Broadcom recommends doubling the timeout to 120000 and then re-testing the behavior, and continuing doubling the timeout value until success is achieved.

Example (from)

# Timeout (milliseconds) for accept operations.
# Used by the BoxMonitor only.
acceptTimeout = 60000


Example (to)

# Timeout (milliseconds) for accept operations.
# Used by the BoxMonitor only.
acceptTimeout = 120000

Once Communication.properties has been updated, restart the Detection Server Service to update BoxMonitor with the new acceptTimeout value.

Additional Information

The same resolution typically applies when you see the following warning in a detection server's BoxMonitor logs:

com.vontu.communication.dataflow.ShippingTask run WARNING: ShippingTask(controller-server, Structured Data Push, Statistics_C , 40345): Requested connection to address <controller-server> could not be established!

Another sign that acceptTimeout needs to be raised on a detection server is when you see the following in MonitorController logs:

com.vontu.monitor.communication.configset.dataflow.ShipmentErrorHandler recover
INFO: Sent the local state to the remote config set <fqdn> <DetectionServer>:filereader:EMDIProfile.

Attachments