After Linux server patching the ESP agent 12.0 failed to start and re-install
search cancel

After Linux server patching the ESP agent 12.0 failed to start and re-install

book

Article ID: 411068

calendar_today

Updated On:

Products

Workload Automation Agent

Issue/Introduction

The WA Agent 12.0 that running on dozens of Linux servers pre-patching couldn't be restarted and re-installed. This occurred on servers that had the 12.0 agent up and running successfully for long periods of time before the patching. 

Environment

Workload Automation Agent 12.0

Cause

Compatibility issue with an older JRE build being utilized by Agent 12.0, particularly when interacting with the updated RHEL environment. This incompatibility can lead to unexpected memory access errors within the JVM's JIT compiler.

____________

Unhandled exception
Type=Segmentation error vmState=0x00070000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F89EF8C5E00 Handler2=00007F89EF1B79F0 InaccessibleAddress=0000000099669B00
RDI=00007F89D96384A3 RSI=00007F89D96384A8 RAX=0000000000000098 RBX=00000000FFE34138
RCX=0000000099669900 RDX=00007F89D96384A8 R8=00000000FFE61018 R9=0000000000000000
R10=0000000000000006 R11=0000000000000001 R12=0000000000000000 R13=00007F89EF9011E2
R14=0000000000016700 R15=00007F89EF9011E2
RIP=00007F89ED4C68BA GS=0000 FS=0000 RSP=00000000000E3C90
EFlags=0000000000010213 CS=0033 RBP=0000000000016700 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000099669B00
xmm0 0000000000000006 (f: 6.000000, d: 2.964394e-323)
xmm1 000000003e3ab283 (f: 1044034176.000000, d: 5.158214e-315)
xmm2 00007f89f4846fe0 (f: 4102320128.000000, d: 6.928307e-310)
xmm3 bfe00002e0b0759c (f: 3769660928.000000, d: -5.000014e-01)
xmm4 bfd01eae7f513a67 (f: 2136029824.000000, d: -2.518727e-01)
xmm5 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm7 0000000000000001 (f: 1.000000, d: 4.940656e-324)
xmm8 0000000000000001 (f: 1.000000, d: 4.940656e-324)
xmm9 5b07101919140106 (f: 420741376.000000, d: 3.197288e+130)
xmm10 0d001b1c195a0506 (f: 425329920.000000, d: 4.607003e-246)
xmm11 000000004a570a3c (f: 1247218176.000000, d: 6.162077e-315)
xmm12 000000004709a025 (f: 1191813120.000000, d: 5.888339e-315)
xmm13 00000000482c0833 (f: 1210845184.000000, d: 5.982370e-315)
xmm14 0000000048e50de0 (f: 1222970880.000000, d: 6.042279e-315)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/tmp/install.dir.15111/Linux/resource/jre/lib/amd64/compressedrefs/libj9jit29.so
Module_base_address=00007F89ED1B0000
Target=2_90_20200416_608 (Linux 4.18.0-553.69.1.el8_10.x86_64)
CPU=amd64 (4 logical CPUs) (0xe512c000 RAM)
----------- Stack Backtrace -----------
(0x00007F89EF1E6D32 [libj9prt29.so+0x49d32])
(0x00007F89EF1B8753 [libj9prt29.so+0x1b753])
(0x00007F89EF1E6DAE [libj9prt29.so+0x49dae])
(0x00007F89EF1E6EA4 [libj9prt29.so+0x49ea4])
(0x00007F89EF1B8753 [libj9prt29.so+0x1b753])
(0x00007F89EF1E6C0B [libj9prt29.so+0x49c0b])
(0x00007F89EF8C5944 [libj9vm29.so+0x92944])
(0x00007F89EF1B8753 [libj9prt29.so+0x1b753])
(0x00007F89EF8C5BB5 [libj9vm29.so+0x92bb5])
(0x00007F89EF8F2995 [libj9vm29.so+0xbf995])
---------------------------------------

 

 

Resolution

Workarounds:

  1. During Agent Installation: To successfully install Agent 12.0 in the affected environment, disable the JIT compiler during the installer execution. For example;
    # JAVA_TOOL_OPTIONS="-Xint" /opt/esp/setup.bin -f /opt/esp/installer.properties
  2. Post-Installation: For an already installed agent experiencing SEGFAULTs, add the -Xnoaot JVM option parameter to the agentparm.txt file and restart the agent.
    • Example: oscomponent.jvm.x.options=-Xmx512m;-Xnoaot
    • Action: After modifying agentparm.txt, ensure the agent service is restarted for the changes to take effect.

Recommendations:
To resolve this compatibility issue and ensure system stability:

  1. JRE Upgrade: Replace the current JRE used by the Agent with an updated version (JRE 8 Update recommended).
  2. Agent Upgrade: Upgrade to a later agent version (e.g., 12.1.1 or 24.0), as these versions are bundled with an updated and compatible JRE.

For detailed instructions on replacing the agent's JRE or upgrading, please refer to docs: replace Agent's JRE and Installing and Upgrading.

Additional Information

Powered by Wolken Software