"Unable to open socket file" for Heap dump on policy server

Products

• SITEMINDER

Issue/Introduction

Engineering team asks for heap dump on a long GC time issue, but the heap dump fails sometimes with error,

744265: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Environment

smps version : Version: 12.8; Update: 08.00; Build: 2892; CR: 00;
OS: Red Hat Enterprise Linux release 8.9 (Ootpa) x86_64

java version "1.8.0_401"

Cause

The smpolicysrv is not really a java application, it's actually a c++ application -- from the smps.log, you can see a lot of .CPP objects.

But it also has a JNI layer -- it will init a jvm thread when needed. i.e. it does not always have the JVM (the Java code handles Federation code and Custom Active expressions/ schemes, etc.)

If the JVM is not init, and you try to do heap dump on it, the smpolicysrv will crash and smexec will restart the smpolicysrv.

The trace log has the details when and how the JVM is initialized.

Resolution

Steps for heap dump,

1.  Get the pid of smpolicysrv
   

   ps aux|grep smpolicysrv|grep -v grep|awk '{print $2}'

2.  Run following command to list the jvm and to ensure the smpolicysrv has already init the jvm
   

   jps -v|grep <pid of smpolicysrv>

   If the jvm on <pid of smpolicysrv> is not found, stop here, we cannot make heap dump or thread dump, otherwise smpolicysrv will crash.

3.  Make the heap dump
   

   jmap -dump:live,format=b,file=heapsm.hprof <pid of smpolicysrv>

4. Make the thread dump
   

   jstack <pid of smpolicysrv> > thread.tdump