Abrupt loss of agent connectivity. ORA-21525 observed in Monitor Controller logs.

Products

• Data Loss Prevention Endpoint Prevent

Issue/Introduction

Many Agents have stopped reporting at once.

The following is observed in the MonitorController#.log(s) 

SEVERE: AgentStatus unexpected error while processing the marshallable received from monitor with monitorId XXXX and its corresponding Id is {guid}, the monitor connection will move onto the next batch.
org.springframework.jdbc.UncategorizedSQLException: CallableStatementCallback; uncategorized SQLException; SQL state [72000]; error code [21525]; ORA-21525: attribute number or (collection element at index)  violated its constraints
; nested exception is java.sql.SQLException: ORA-21525: attribute number or (collection element at index)  violated its constraints

Environment

Release : 15.8

Component :

Cause

Agent data including IP address, connection status, group membership, etc, is persisted to the DLP database when an agent connects to the endpoint server. 
At times, the values provided can exceed data length limitations, especially with relation to AD User Group Membership and agents being assigned IPV6 addresses. 

This can cause agent data persistence errors. It will not prevent other data from persisting, such as Endpoint servers reporting agents as 'not reporting' in the event the agent has been connecting to a different server for a while. 
Normally these 'not reporting' events would be ignored because newer data had come in, but in this case, the newer data is being discarded due to errors, so the 'not reporting' events appear to be the newest data and thus persist to the database, causing agents to enter a 'not reporting' status even though they are currently online and active. 

 

Resolution

To identify which constraint was violated,

1. Enable JDBC logging 

2. Wait for issue to recur. 

3. Locate the ORA-21525 error in the JDBC logs. 

4. Identify content that exceeds the values set forth by the Database schema. (These can be identified with the 'desc' SQL command in Oracle, or with tools such as SQL Developer)

Examples include:
IP Addresses that exceed 32 characters(IPv6 is not supported on DLP endpoint agents as of DLP 15.8)
AD Group membership that exceeds 2048 characters. (Identified in the AGENTGROUPATTRIBUTERESULT table)

 

Additional Information

Agents may appear offline in the console if these errors are present.  However, they may be online and will still receive policies and configuration updates, and are only shown as offline because new events(not to be confused with incidents) are not persisting.