SpectroSERVER core file generation configuration on RHEL 8.x

Products

• Network Observability
• Spectrum

Issue/Introduction

When a process crashes the operating system will attempt to create a core file which contains information about the thread(s) that may have caused the crash.

Sometimes the operating system is not configured properly to generate the core file on RedHat Linux 8.

core file generation on RedHat linux RHEL 8 for the SpectroSERVER

SpectroSERVER crashed, tried to create a core file but failed to do so.

Environment

All supported Network Observability DX NetOps Spectrum releases with SpectroSERVER on Linux RH 8.x

Cause

The gdb package is not installed

ulimit is not configured properly

corefile location does not have permissions or space

Resolution

The core file creation is performed as the same user that owns and runs processd. This is normally the root user on a Linux SpectroSERVER.

The core file is generated in the $SPECROOT/SS directory. Ensure there is sufficient disk space for core file generation. They can be as large as the memory in use by the SpectroSERVER process when it crashes. If server has 20 GB available ensure at least that much space remains free for core file generation.

To ensure the core file is created complete the following configuration.

1. 1. Ensure the $SPECROOT/SS directory has sufficient free space.
      • The core files are usually a few GB in size.
      • Maintaining at least 10 GB free space should be sufficient in most instances.
   2. Ensure the gdb debugging package is installed:
      • Check for install? Run "rpm -qa gdb"
      • Not installed? Install using "yum install gdb"
   3. Set ulimit to unlimited for corefilesize.
      • Check current value?
        • Run "ulimit -c" as the root user.
        • Alternative: Run "ulimit -a" as root user and examine "core file size" value configured.
      • Set to unlimited? Run "ulimit -c unlimited" as root user. Run the check to confirm it was set properly.
   4. Configure the /etc/security/limits.conf file.
      • Open the file for editing. Add the following lines to the file and save the changes.
        
        *               soft    core            unlimited
        
        
   5. Configure the /etc/sysctl.conf file.
      • Open the file for editing. Add the following lines to the file and save the changes.
        
        kernel.core_pattern=/var/crash/core_%e_%p_%h_%t
        kernel.core_uses_pid=1
        fs.suid_dumpable=1
   6. Configure the /etc/systemd/system.conf file.
      • Open the file for editing. Add the following lines to the file and save the changes.
        
        DumpCore=yes
        DefaultLimitCORE=infinity
        
        
   7. Reboot the server.
   8. Test core file generation:
      1. Trigger the core file generation with these two commands:
         
         echo "/tmp/core" > /proc/sys/kernel/core_pattern
         nohup sleep 600 &
         
         
      2. Run this to find the PID of the sleep process "ps -ef | grep -i sleep"
      3. Run this to shut it down and generate the core file "kill -11 <PID_of_sleepProcess>".
      4. Check the tmp directory for file creation with "ls -al /tmp/". Should see something like this.
         
         total 180
         drwxrwxrwt.  7 root    root      4096 Jun 24 20:04 .
         dr-xr-xr-x. 17 root    root       224 Apr 28  2023 ..
         -rw-------.  1 root    root    389120 Jun 24 20:04 core.1002921

Additional Information

Instructions may be the same for RH 7.x and RH 9.x releases. Consult your system administration team for additional information or environmental requirements and limitations for core files.