DX APM SaaS - Weblogic Agent Crashes

Products

• DX APM SaaS

Issue/Introduction

Weblogic Agent Crashes and the log 'hs_err_pid#' shows below stuck thread messages: 

0x0000565371bda000 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2572248, stack(0x00007f5de9a63000,0x00007f5de9b64000)]
  0x00007f6670845800 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2572185, stack(0x00007f5de824b000,0x00007f5de834c000)]
  0x00007f66e0502000 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2572109, stack(0x00007f5de814a000,0x00007f5de824b000)]
  0x00007f6634406800 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2571985, stack(0x00007f5de834c000,0x00007f5de844d000)]
  0x00007f667cb9a000 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2571854, stack(0x00007f5de854e000,0x00007f5de864f000)]
  0x00007f6713c75800 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2571782, stack(0x00007f5de8851000,0x00007f5de8952000)]
  0x00007f6634405000 JavaThread "PO Exception Async Executor" daemon [_thread_blocked, id=2571650, stack(0x00007f5de8750000,0x00007f5de8851000)]

Environment

Application OS: Linux,

DX APM WebLogic Agent version: 23.12.1.14 (Build 990014), 

WebLogic Version: 14

Cause

Excessive number of "PO Exception Async Executor"  threads, caused by connectivity issue between Weblogic Wily Agent and EM so the Agent keeps reconnecting with EM.

Somehow the old and failed connections were not cleaned up properly.

Resolution

Engineering created a fix to clear the older and failed connections 

And merged the fix with the release v 24.9.1 

ETA of the new release is September 2024.