SpectroSERVER core files truncated during crashes on RHEL 8 - DX NetOps Spectrum

Products

• Network Observability
• Spectrum

Issue/Introduction

After upgrading to DX NetOps Spectrum 25.4.x, the SpectroSERVER may crash every few days. When attempting to analyze the resulting core dump with GDB, the file is reported as truncated, and VNM.OUT may be empty.

Symptoms:

• SpectroSERVER terminates unexpectedly.
• VNM.OUT contains no error messages or stack traces.
• GDB reports: Warning: core file is truncated: expected core file size >= ####, found: ####.
• GDB reports: Cannot access memory at address ####.

Environment

• DX NetOps Spectrum 25.4.x (including 25.4.9)
• Red Hat Enterprise Linux (RHEL) 8.x (including 8.10)
• systemd-coredump enabled

Cause

RHEL 8 uses systemd-coredump to manage crash logs. By default, ExternalSizeMax and ProcessSizeMax in /etc/systemd/coredump.conf are limited (often to 2 GB). Since the SpectroSERVER process often occupies 5 GB or more of resident memory, the OS terminates the dump process prematurely, resulting in a useless core file.

Resolution

1. Log in to the RHEL 8 server as a user with sudo privileges.
2. Open the coredump configuration file: sudo vi /etc/systemd/coredump.conf
3. Locate the [Coredump] section and ensure the following parameters are set to infinity:
   • ProcessSizeMax=infinity
   • ExternalSizeMax=infinity
4. Save the file and exit.
5. Apply the changes by reloading the systemd daemon: sudo systemctl daemon-reload
6. Restart the SpectroSERVER to apply .vnmrc changes.