Collecting and Analyzing Crash Data from a Solaris Environment.

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

When a SiteMinder component crashes and generates a core file, what is the best approach to collect the necessary data for the Support team to analyze?

 

Environment

Release:
Component: SMPLC

Resolution

While the core file itself is useful for analysis, the associated symbols within linked libraries are also necessary. There is a script, "pkg_app.sh", that will automatically gather the relevant linked libraries and executables of the crashed process, create a tarfile with these and the actual core file, and also create files for later use with the 'dbx' debugger. The steps to use this are as follows:

(This example assumes a Policy Server crash, just substitute the actual crashing process name)

Step 1) Create the situation where the Policy Server process (smpolicysrv) crashes, and generates a core file.

Step 2) Run "pkgapp -c <corefile name> -p <full path to process binary including the name of the executable>". This creates a tarfile that includes the core file, all relevant linked libraries, relevant executables of the crashed process, and generated files for use with the dbx debugger.

Step 3) Upload the resulting tarfile to Support for analysis, along with any additional requested log files (ideally all generated logs surrounding the crash).

(These last steps may be performed by Support after uploading.)

Step 4) run 'opencore' (an auto-generated script within the tar file). This should only be run on Solaris (the Solaris version doesn't need to match, but should ideally be the same or later OS version). Running this should place you in the 'dbx' debugger.

Step 5) Once in 'dbx', the following commands will generally provide the necessary debugging information for a crash analysis:

> 'where -l' (lowercase ell; shows backtrace of current (crashed) thread, plus libraries associated with the crash)
> 'lwps' (lists all threads)
> 'proc -map' (shows address ranges of different libraries, may be used to determine which library actually failed)