Error: "Significant row contention was detected in the TABLE "PROTECT.MGR_QRTZ_LOCKS" "

Products

• Data Loss Prevention
• Data Loss Prevention Enforce

Issue/Introduction

Unable to log into the Enforce console.

The DBA found that this was blocking with an average elapsed time is 4138 seconds:

 SQL statement with SQL_ID 5bp2kvd0bbzyx.
         SELECT * FROM MGR_QRTZ_LOCKS WHERE SCHED_NAME = 'managerScheduler'
         AND LOCK_NAME = :1 FOR UPDATE

One of the DBA's findings is:

"Significant row contention was detected in the TABLE
      "PROTECT.MGR_QRTZ_LOCKS" with object ID 93408. Trace the cause of row
      contention in the application logic using the given blocked SQL.
      Related Object
         Database object with ID 93408."

Also, the localhost log shows many "java.lang.OutOfMemoryError: Java heap space" errors:

Level: SEVERE
Source: org.apache.catalina.core.ContainerBase.[Catalina
Message: Servlet.service() for servlet [springmvc] in context with path [/ProtectManager] threw exception [Request processing failed; nested exception is java.security.PrivilegedActionException: org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space] with root cause
Cause:
java.lang.OutOfMemoryError: Java heap space

Level: SEVERE
Source: com.hazelcast.spi.impl.eventservice.impl.EventServiceImpl
Message: [127.0.0.1]:5701 [enforceMessaging] [3.6.5] hz._hzInstance_1_enforceMessaging.event-5 caught an exception while processing task:EventProcessor{envelope=null}
Cause:
java.lang.OutOfMemoryError: Java heap space

Level: SEVERE
Source: com.vontu.enforce.domainlayer.common.database.DatabaseConnectionStatusTask
Message: Unexpected error checking database state
Cause:
java.lang.OutOfMemoryError: Java heap space

Level: SEVERE
Source: org.apache.coyote.http11.Http11NioProtocol
Message: Failed to complete processing of a request
Cause:
java.lang.OutOfMemoryError: Java heap space

Level: SEVERE
Source: org.apache.catalina.core.ContainerBase
Message: Exception processing background thread
Cause:
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap spacejava.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

Level: SEVERE
Source: com.hazelcast.nio.tcp.nonblocking.iobalancer.IOBalancer
Message: [127.0.0.1]:5701 [enforceMessaging] [3.6.5] IOBalancer failed
Cause:
java.lang.OutOfMemoryError: Java heap space

 

Environment

Release: 15.x

Resolution

The contention, block, on the "ROTECT.MGR_QRTZ_LOCKS" is being caused by the system being overwhelmed.
The logging is set far too high.
This can cause the MGR_QRTZ_LOCKS issue.
And it can cause the Java heap size errors that are in the logs.

In this case, we reset these values in the ManagerLogging.properties file.

The com.vontu.jdbc.threshold_ms value was also set far too aggressively.
Reset the value to the default of "500".

Reset the com.vontu.jdbc.level back to the default of "INFO".

Reset the "com.vontu.level" value to the deault of "INFO".