Filereader fails to start

book

Article ID: 160754

calendar_today

Updated On:

Products

Data Loss Prevention Network Monitor

Issue/Introduction

FileReader on Monitor server fails to start

Error in FileReader logs right after the waiting for policies messages:

INFO: Waiting for policies...

Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager processOperationResult

INFO: Operation com.vontu.communication.transport.ReadOperation:1289481294494:boxmonitor:controller-server:127.0.0.1:8100:null failed with exception: com.vontu.communication.transport.exception.TransportException: remote endpoint closed connection

Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager handleOperationFailure

 

A restart of FileReader will follow and will continue until Filereader hits maximum number of retries.

Restart of  Monitor server does not resolve issue.

 

Resolution

Policies are not stored locally on the Detection Server.  This is for security reasons.  They are provided by Enforce from the Oracle DB through the MonitorController. When a Monitor server is waiting for policies this can be indicative of a communication problem between Enforce and Oracle.  It can also be indicative of a communication problem between Enforce and the Detection Server. The problem may no longer exist but has exhausted the threads used for communication.  The symptoms occur at the time of a FileReader restart, not at the time of the communication issue because that is when the FileReader is requesting the policies.

The Short Term Solution is to restart the Monitor Controller process on Enforce, then restart VontuMonitor process on the Detection Server. The restart of Monitor Controller on Enforce will clear up all threads used for Oracle communication.

After the restart you will see in FileReader0.log a statement similar to:

INFO: Loaded 7 policies. 7 policies were loaded

Where 7 is the number of policies in customer environment

Long Term Solution is to identify the communication issues. You can enable fine logging on Monitor Controller and when this issue occurs again you can search in MonitorController0.log on Enforce for either of the two statements below to find a DB error when Filereader is not starting

“FINE: Failed to send instruction to remote config set” or “Unable to read clob from database”

 

From the error you can see that the Enforce is unable to communicate with the database. Without that communication it is unable to push the policy down to Monitor server.  The FileReader stays in starting until it times out and restarts the entire process again.

The main culprit of communication issues is the network between Enforce and Oracle. The issue could exist sporadically and will be difficult to identify. In customer environments this can be seen as much as every day to only once every couple weeks or months. The frequency of the issue will depend on how frequent the communication problems exist.  The customer will need to identify the issue.