DLP Endpoint Agents appear as Not Reporting in the Enforce Console even though they are online and connecting to the Endpoint Detection Server

Products

• Data Loss Prevention Endpoint Prevent

Issue/Introduction

The DLP Endpoint Agents are in Not Reporting status in the Enforce Console even though the Endpoint Agent services are running and the agents are connecting to the Endpoint Server.

The following event is also present on the Enforce Console:

Event Code: 4050

Event Summary: Agent data batch persist error

Event Detail: Unexpected error occured while agent data being persisted : Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Database connection failed. Please look at the detection server controller logs for more information.

The DetectionServerController logs contain the following:

SEVERE: Agent data batch persist error. Unexpected error occured while agent data being persisted : Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Database connection failed. Please look at the detection server controller logs for more information.

SEVERE: AgentStatus unexpected error while processing the marshallable received from monitor with monitorId 1 and its corresponding Id is 89cc2da4-9cf3-45dd-bddc-bb62c44167b4, the monitor connection will move onto the next batch.
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLException: Database connection failed

...

Caused by: com.vontu.util.jdbc.DatabaseConnectionPoolExhaustedException: A connection could not be obtained from the pool
 at com.vontu.util.jdbc.DefaultDataSource.verifyConnectionIsNotNull(DefaultDataSource.java:137)

Cause

The DetectionServerController service, which is the process on Enforce that communicates with the detection servers, cannot obtain a database connection to the protect database on oracle because it can't obtain one from the database connection pool, presumably because they are all exhausted.

Resolution

1. Stop the SymantecDLPDetectionServerService on the Endpoint server(s)

2. Stop the Enforce Services in the following order:

1. SymantecDLPDetectionServerControllerService
2. SymantecDLPIncidentPersisterService
3. SymantecDLPManagerService
4. SymantecDLPNotifierService

3. Start the Enforce Services n the following order:

1. SymantecDLPNotifierService
2. SymantecDLPManagerService
3. SymantecDLPIncidentPersisterService
4. SymantecDLPDetectionServerControllerService

4. Start the SymantecDLPDetectionServerService on the Endpoint server(s)

Restarting services allows the database connection pool to be reset and the Symantec DLP Detection Server Controller Service to obtain a new database connection from that pool. After receiving a good database connection the Detection Server Controller can then connect to the detection servers and receive the updated status from the Endpoint Agents.