JVM Settings: Configure defaultReadTimeout to Prevent Integration Timeouts and Stuck Processes

Products

• Clarity PPM SaaS
• Clarity FedRAMP
• Clarity PPM On Premise

Issue/Introduction

The parameter -Dsun.net.client.defaultReadTimeout=900000 is used in Clarity to implement a 15-minute timeout for JVM services. It is specifically used to address issues where integration processes become stuck or experience SocketTimeoutException: Read timed out during long-running operations.

Symptoms may include:

• Stuck Processes: Background processes frequently hang without completion (Integration process stuck)
• Socket Timeouts: External integrations (like Saviynt or XOG) encounter java.net.SocketTimeoutException (Web Services issues)

Environment

Clarity 16.4.1, 16.4.2 and above

Cause

• Unoptimized Queries: Long-running background or integration processes contain underlying database queries that are not optimized, causing them to execute past acceptable thresholds.
• By default, the Clarity BG JVM services may lack an explicit read timeout limit for long-running operations. Without the -Dsun.net.client.defaultReadTimeout parameter, background process threads can become stuck indefinitely or hang without completing
• Timeout Mismatch: The timeout configurations on the network load balancer are misaligned with the JVM settings, causing the load balancer to drop the connection and throw a java.net.SocketTimeoutException: Read timed out during heavy integration payloads (e.g. SOAP or XOG Calls)

Resolution

On Premise Customer: Please engage Broadcom Support to investigate the root cause by providing thread dumps of the BG services. It is recommended to capture multiple thread dumps at regular intervals, as this will provide better insight into the state of the application over time and help identify any blocking, deadlocks, or performance-related issues during the investigation.

• Stop the BG Services
• Apply the JVM Parameter to Background (BG) Services
  • Value: 900000 milliseconds (15 minutes).
  • Implementation: Add the following parameter to the JVM startup arguments for the BG service:
    • -Dsun.net.client.defaultReadTimeout=900000
• Start the BG Service
• Verify Network Load Balancer Alignment: If external integrations continue to encounter java.net.SocketTimeoutException after applying the JVM argument, verify if the network load balancer timeouts align properly with this 15-minute JVM setting.

SaaS Customers: This setting is enabled in Broadcom SaaS enviornment. Please engage Broadcom Support to investigate the root cause and implement remediation as needed. 

Additional Information

While increasing the JVM timeout may temporarily resolve the immediate service hang issues, optimizing the underlying queries is the preferred long-term solution. Addressing the root cause will help prevent the application from reaching these timeout thresholds and improve overall system performance and stability.