Additional Access Gateway Tomcat/Java crash debugging tips

Products

• SITEMINDER

Issue/Introduction

The Access Gateway Tomcat/Java process can crash but rarely, no evidence is found in the standard logs so additional troubleshooting steps are needed.

Environment

Applies to Access Gateway

RHEL

Cause

Normally, if the process dies under most of the known conditions, either it will write to /var/log messages (any SIGTERM) or hs_errid.log if there was an SIGABRT/SIGSEGV/. In order to troubleshoot this issue, additional debugging at system level was enabled.

Resolution

For when the issue already occurred:

 

Run these commands to gather the complete system info. They may have some pointers for the Access Gateway process if it was shutdown with no visible reason.

1. dmesg > dmesg.txt 
2. journalctl> journalctl.txt  
3. journalctl --boot > journalctl_boot.txt

 

To monitor for a future issue occurrence, there are 2 options: 

- Option 1

1. check if the coredumps is enabled

 

ulimit -S -c unlimited

If you want to enable core dump permanently, add following line in /etc/security/limits.conf to update system limit

* soft core unlimited

You can make core dump file path changes permanant, by adding following line in /etc/sysctl.conf

kernel.core_pattern="/coredumps/core-%e-%s-%u-%g-%p-%t"

 

          2.Start SPS , get the java process id and attach it to gdb.

  commands:  ps -ef | grep java 

capture the process id 

gdb <java path> <processid>

once gbd is attached , on the prompt issue command <continue>

 

refer to screenshots

gdb /opt/CA/jdk1.8.0_251/bin/java 31179

 

 

 

 if the process terminates/aborts/segmentation it will stop at this gdb prompt. Please capture the screen shot and also issue the command  bt

 

 

- Option 2)  

top -H -p $(pgrep java); pstack  $(pgrep java)