Enforce Server down: Detection Server Controller Service and Incident Persister Service will not run

Products

• Data Loss Prevention
• Data Loss Prevention Enforce

Issue/Introduction

Enforce Server Web UI down this morning.  Investigation showed Detection Server Controller Service and Incident Persister Service not running.  Have reviewed applicable Windows and DLP logs, replaced service exes and conf files and rebooted a few times.  They will not stay up and the Enforce Server cannot be accessed.  The system has otherwise been very stable and we are working on upgrading it to 16.0.

Cause

Incident Persister log shows:

(SEVERE) Thread: 16 [com.vontu.incidenthandler.IncidentPersister.start] Unexpected error starting Incident Persister.
java.lang.RuntimeException: Database is not accepting connections, or bad Jdbc.properties file.
    at com.vontu.util.jdbc.JDBCTestConnection.checkDatabaseConnection(JDBCTestConnection.java:137)
    at com.vontu.incidenthandler.IncidentPersister.start(IncidentPersister.java:114)
    at com.vontu.incidenthandler.IncidentPersister.main(IncidentPersister.java:217)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:325)
    at java.lang.Thread.run(Thread.java:748)

MonitorController logs show:

com.vontu.monitor.controller.Service start
SEVERE: Error starting monitor controller
java.lang.RuntimeException: Database is not accepting connections, or bad Jdbc.properties file.
    at com.vontu.util.jdbc.JDBCTestConnection.checkDatabaseConnection(JDBCTestConnection.java:137)
    at com.vontu.monitor.controller.Service.initializeService(Service.java:122)
    at com.vontu.monitor.controller.Service.start(Service.java:73)
    at com.vontu.monitor.controller.Service.main(Service.java:113)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:325)
    at java.lang.Thread.run(Thread.java:748)

Resolution

Stop Enforce services, then recycle the OracleServiceProtect and OracleOraDB19Home1TNSListener services.  If this is a 3-tier installation, those services can be found on the Oracle server.

Restart the Enforce services, and check to make sure they stay running.

Login to Enforce console and verify all detection servers are in a running state.