Filereader fails to start

Products

• Data Loss Prevention

Issue/Introduction

• FileReader on detection server fails to start
• Error in FileReader logs right after the waiting for policies messages:

INFO: Waiting for policies...
Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager processOperationResult

INFO:Operationcom.vontu.communication.transport.ReadOperation:1289481294494:boxmonitor:controller-server:127.0.0.1:8100:null failed with exception: com.vontu.communication.transport.exception.TransportException: remote endpoint closed connection
Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager handleOperationFailure

• A restart of FileReader will follow and will continue until Filereader hits the maximum number of retries.
• A restart of  the Symantec DLP Detection Server service does not resolve issue.

Resolution

Policies are not stored locally on the Detection Server.  This is for security reasons.  They are provided by Enforce from the Oracle DB through the Detection Server Controller service.

• When a detection server is waiting for policies this can be indicative of a communication problem between Enforce and Oracle. 
• It can also be indicative of a communication problem between Enforce and the Detection Server.
• The problem may no longer exist but has exhausted the threads used for communication. 
• The symptoms occur at the time of a FileReader restart, not at the time of the communication issue because that is when the FileReader is requesting the policies.

 

• The Short Term Solution is to restart the Symantec DLP Detection Server Controller service on Enforce, then restart Symantec DLP Detection Server service on the Detection Server. The restart of the Detection Server Controller service on Enforce will clear up all threads used for Oracle communication.
  • After the restart you will see in FileReader0.log a statement similar to:

INFO: Loaded 7 policies. 7 policies were loaded  (Where 7 is the number of policies in customer environment)

• Long Term Solution is to identify the communication issue(s). You can enable FINE logging on Monitor Controller logs, and when this issue occurs again you can search in MonitorController0.log on Enforce for either of the two statements below to find a DB error when Filereader is not starting

“FINE: Failed to send instruction to remote config set” or “Unable to read clob from database”

 

• From the error you can see that Enforce is unable to communicate with the database.
• Without that communication it is unable to push the policy down to Monitor server. 
• The FileReader stays in starting until it times out and restarts the entire process again.
• The main culprit of communication issues is the network between Enforce and Oracle.
• The issue could exist sporadically and will be difficult to identify.
• In customer environments this can be seen as much as every day to only once every couple weeks or months.
• The frequency of the issue will depend on how frequent the communication problems exist.  
• The customer will need to identify the issue.