smconsole generates core dump on exit

Products

• SITEMINDER

Issue/Introduction

Policy server smconsole generates core dump on exit after upgraded to 12.80.801.3003.
malloc_consolidate(): invalid chunk size
<Install_dir>/siteminder/bin/smconsole: line 16: 1068719: Abort(coredump)

hs_err_log shows below:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007efc2e1e84bc, pid=58952, tid=58974
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.15.0.6-1) (17.0.15+6) (build 17.0.15+6-LTS)
# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.15.0.6-1) (17.0.15+6-LTS, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x804bc]  malloc_consolidate+0x12c
#

Environment

SiteMinder: 12.8SP8CR1 build 3003

Policy server OS: Red Hat Enterprise Linux Server release 8.10

Cause

1. Eliminated java version as a factor, after having tried openjdk version "1.8.0_452" and openjdk version "11.0.19" LTS, no improvement.

2. Tried to increase max locked memory (ulimit -l 65534), did not resolve.

3. The problem is irrelevant to SiteMinder version, as a clean installation does not show the problem.

Resolution

3rd party software Dynatrace was loaded when smconsole was launched.

liboneagentproc.so is one of libs that Dynatrace uses.

After removal of /lib64/liboneagentproc.so from /etc/ld.so.preload, smconsole is no longer getting core dump.