Error 1011 Restarted RequestProcessor on Email detection Servers

Products

• Data Loss Prevention Network Monitor and Prevent for Email

Issue/Introduction

You see multiple alerts in the Enforce console for error 1011 on network prevent for email servers where the RequestProcessor has been restarted.

 

Request Processor logs appear to be showing normal operation whilst running, however...

 

boxmonitor_operational logs show frequent entries such as:

[SEVERE] (BOXMONITOR.14) Process [RequestProcessor] has not responded for 16 minute(s) 0 ms

 

Example in FileReader logs:

INFO: Request Processor Listener #12 Waiting for connection from Request Processor
INFO: Request Processor Listener #5 Waiting for connection from Request Processor
INFO: Request Processor Listener #46 Waiting for connection from Request Processor
INFO: Request Processor Listener #20 Waiting for connection from Request Processor

Environment

Release : 15.x

Cause

In this instance the cause was a memory leak on the downstream device (Ironport) that was causing ever greater delays to accepting messages and resulting in the the upstream MTA to time-out and terminate the connection.  DLP showed upstream disconnections although the root of the problem was downstream.

Symptomatic of this was the RequestProcessor log in DLP showing very long average receive times for messages (circa 40 seconds) although detection times were of the order of fractions of a second.

Resolution

Fixing the memory leak on the downstream device resolved the issue.