Smarts: How to generate a core file in Linux

Products

VMware

Issue/Introduction

Symptoms:

The Smarts server is crashing, and I need to create a Memory dump on Linux server for troubleshooting by EMC Customer Service.

This article describes two methods to take a memory dump in LINUX. One requires the application to crash before the memory dump will happen, and one allows you to force a memory dump while it remains running.

NOTE: In some cases, you may not be able to generate a core file in Smarts IP/Smarts SAM environment due to the fact that gcore is not installed on Linux server. If this symptom exists, you may see this log file entry:

Cannot run gcore -o sm_gcore.out <pid> command in Smarts IP/Smarts SAM environment because gcore not installed on the Linux server.

Environment

VMware Smart Assurance - SMARTS

Resolution

To facilicate a LINUX core dump, follow these steps:

Stop the Smarts server:

(NOTE: These example commands are only applicable to a SAM server.)

Running from the command line:
<BASEDIR>/smarts/bin/sm_server -n InCharge-SAM -c ics -b localhost:427 --ignore-restore-errors --output --daemon

To confirm it's stopped, execute:
<BASEDIR>/smarts/brcontrol

When it's installed as a service:
<BASEDIR>/smarts/bin/sm_service stop InCharge-SA1. Enable core file dumping

To confirm the service is stoppped, execute:
<BASEDIR>/smarts/sm_service show

We now set the memory usage to unlimited to make sure our Smarts server is not simply being killed by LINUX for consuming more memory than the set limit. In the same putty window, run this LINUX command:

<BASEDIR>/smarts/ulimit -c unlimited

Reload the SAM server in the same Putty sesion, or command line window.

Running from the command line:
<BASEDIR>/smarts/bin/sm_server -n InCharge-SAM -c icf --bootstrap=bootstrap-am-pm.conf -b localhost:427 --output --daemon

To confirm the domain has started, execute this command:
<BASEDIR>/smarts/bin/brcontrol

When it's installed as a service:
<BASEDIR>/smarts/bin/sm_service start InCharge-SAM

To confirm the domain has started, excecute this command:
<BASEDIR>/smarts/bin/sm_service show

Take the steps necesary to reproduce the hang. Or, if need be, wait for the issue to reproduce.
Collect core file from bin directory. We also suggest collect the rps file. Upload the files to the EMC support case.

The rps file will be called by the same name as the domain name, and be stored in the <BASEDIR>/smarts/local/repos/icf folder.
The memory dump fies should be left in the same folder that you started the Smarts domain in. However, if you started it as a service here are some proposed folder locations you may find them in:

<base dir>/SAM/smarts/bin
<base dir>/SAM/smarts/local/logs
/home/usr

To cause a memory dump to happen at will, follow one of these two suggestions.

Running GDB:

(gdb) attach <pid>
(gdb) generate-core-file <optional-filename>
(gdb) detach

NOTE: If you cannot generate a core file because gdb is not installed, you can use "gdb" which is shipped with Smarts products (located at <Base dir>/smarts/bin/gdb) to generate a core file. This can be done as follows:

Edit the following commands into a file called gcore.gdb:

set pagination off
set width 65536
gcore
detach
quit

Run the following command:
smarts/bin/gdb -x gcore.gdb -p <pid>

Using the kill signal command:

kill -s SIGABRT <pid>

NOTE: Look for the memory dump files in the folder where the application was started from, or from these locations:

<base dir>/SAM/smarts/bin
<base dir>/SAM/smarts/local/logs
/home/usr

Additional Information

What is the .gdb file in <basedir>smarts/bin is used for and can I remove it?

Smarts needs gdb to dump stacktrace in real time . When smarts hits an exception (either fatal or error) or receives a signal, the exception or signal handler would invoke the included gdb to create stacktrace into log file.
If gdb is removed, stacktrace information will not be logged in the log file if the system ever crashes. The included gdb has been tested to work with smarts to generate stacktrace. And if the native gdb is not on the search path of smarts runtime, smarts may not be able to find gdb if the included one is removed.

How to test for the current ulimit and core dump setting:

1: Search your logs:

U can search your current SAM, or IP log file for this text - "Current limit on CORE file"
In a linux environment , you would use the "VI" editor and search for the above text, as per the below example.

vi <basedir>smarts/local/logs/<Name of SAM log file_en_US_UTF-8.log>

Here an example of what you'd see:

Current limit on CORE file size: 15.999 EiB (18446744073709551613)
Current address space limit: 15.999 EiB (18446744073709551613)
Current file descriptor limit: 256
Current top of heap: 4300218368

2: Using the linux command line environment:

Check the OSs setting for name of the core files by

[root@itops-tco-229 user1]# cat /proc/sys/kernel/core_pattern

If you only see the name "core" here, it indicates that this will be the name of the core file name and the default location will be from where process is executed.

Check the ulimit settings

[root@itops-tco-229 user1]# ulimit -c unlimited
[root@itops-tco-229 juser1]# ulimit -a

An example result from thes commands is below:

    core file size (blocks, -c) unlimited
    data seg size (kbytes, -d) unlimited
    scheduling priority (-e) 0
    file size (blocks, -f) unlimited
    pending signals (-i) 139264
    max locked memory (kbytes, -l) 32
    max memory size (kbytes, -m) unlimited
    open files (-n) 1024
    pipe size (512 bytes, -p) 8
    POSIX message queues (bytes, -q) 819200
    real-time priority (-r) 0
    stack size (kbytes, -s) 10240
    cpu time (seconds, -t) unlimited
    max user processes (-u) 139264
    virtual memory (kbytes, -v) unlimited
    file locks (-x) unlimited