FileReader on Monitor server fails to start
Error in FileReader logs right after the waiting for policies messages:
INFO: Waiting for policies...
Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager processOperationResult
INFO: Operation com.vontu.communication.transport.ReadOperation:1289481294494:boxmonitor:controller-server:127.0.0.1:8100:null failed with exception: com.vontu.communication.transport.exception.TransportException: remote endpoint closed connection
Nov 11, 2010 8:14:54 AM com.vontu.communication.transport.ChannelManager handleOperationFailure
A restart of FileReader will follow and will continue until Filereader hits maximum number of retries.
Restart of Monitor server does not resolve issue.
Policies are not stored locally on the Detection Server. This is for security reasons. They are provided by Enforce from the Oracle DB through the MonitorController. When a Monitor server is waiting for policies this can be indicative of a communication problem between Enforce and Oracle. It can also be indicative of a communication problem between Enforce and the Detection Server. The problem may no longer exist but has exhausted the threads used for communication. The symptoms occur at the time of a FileReader restart, not at the time of the communication issue because that is when the FileReader is requesting the policies.
The Short Term Solution is to restart the Monitor Controller process on Enforce, then restart VontuMonitor process on the Detection Server. The restart of Monitor Controller on Enforce will clear up all threads used for Oracle communication.
After the restart you will see in FileReader0.log a statement similar to:
INFO: Loaded 7 policies. 7 policies were loaded
Where 7 is the number of policies in customer environment
Long Term Solution is to identify the communication issues. You can enable fine logging on Monitor Controller and when this issue occurs again you can search in MonitorController0.log on Enforce for either of the two statements below to find a DB error when Filereader is not starting
“FINE: Failed to send instruction to remote config set” or “Unable to read clob from database”
From the error you can see that the Enforce is unable to communicate with the database. Without that communication it is unable to push the policy down to Monitor server. The FileReader stays in starting until it times out and restarts the entire process again.
The main culprit of communication issues is the network between Enforce and Oracle. The issue could exist sporadically and will be difficult to identify. In customer environments this can be seen as much as every day to only once every couple weeks or months. The frequency of the issue will depend on how frequent the communication problems exist. The customer will need to identify the issue.