All Outbound Emails completely stopped, Email Prevent servers stuck in starting.

Products

• Data Loss Prevention Network Prevent for Email

Issue/Introduction

All Outbound Emails completely stopped, Email prevents servers were stuck in starting, after reboot or restart of services they go to unknown.

BoxMonitor log reports:

SEVERE: (BOXMONITOR.14) Process [RequestProcessor] has not responded for 30 minute(s) 0 ms

Request processor log reports:

*********** PM com.vontu.mta.rp.EventLoop handlePeerDisconnect
WARNING: RPT(2d): Disconnect from sending peer com.vontu.mta.rp.connection.UpstreamConnection@**********

********** PM com.vontu.mta.rp.connection.ConnectionsStateManagerAction cleanUpAfterAbort
WARNING: RPT(31): Message aborted during transfer. Clearing message chain state.

SmtpPrevent_operational logs report:

WARNING] (SMTP_MESSAGE.5301) Sender rejected during redirect (tid=29 cid=Downstream

Environment

Release : 15.7

Component : Web Prevent

Cause

Monitor control java heap space was starved.

The rule of thumb is 4 GB per 12 detection servers, and there are other factors at play so give it a bit more than you would think necessary. 

In this case, there were 31 detection servers, and they only had 8GB allocated

Resolution

Edit SymantecDLPDetectionServerController.conf Located at:

C:\Program Files\Symantec\DataLossPrevention\EnforceServer\Services "Windows"

/opt/Symantec/DataLossPrevention/EnforceServer/Services "Linux"

and adjust wrapper.java.maxmemory to the appropriate value for the environment.

In this case, we increased to 16GB

Restart the SymantecDLPDetectionServerControllerService

Make sure the machine has enough physical memory on the box to make the correction.