ems probe creating very large log files

Products

• DX Unified Infrastructure Management (Nimsoft / UIM)

Issue/Introduction

The ems probe log file may grow indefinitely, reaching several gigabytes in size. Despite configuring a SizeBasedTriggeringPolicy in the log4j2.xml file, the log fails to roll over at the expected size, or it rolls at seemingly random intervals with inconsistent file sizes.

Environment

• DX UIM - Any Version (see note regarding 23.4.7 and earlier below)
• ems probe - any version

Cause

the ems probe utilizes two independent logging frameworks—Legacy Nimbus and Log4j2—that simultaneously target the same physical file (ems.log). This dual-ownership architecture creates resource contention and data corruption because both frameworks maintain open write-handles, leading to race conditions during file rotation or truncation.

Size Discrepancy: Log4j2 tracks file size via an internal byte counter rather than monitoring the actual disk size. It is "blind" to any data written by the Nimbus framework.

Delayed Rollover: Because Log4j2 only triggers a rollover once its own internal counter reaches the limit, the physical ems.log can grow significantly larger than configured (e.g., a 10 MB limit might result in a 50 MB+ file) due to uncounted Nimbus log entries.  Meanwhile, the Nimbus "logsize" key is unable to roll over the ems.log when it is held open by Log4j2.

 

Resolution

1. Open the Raw Configure interface for the ems probe
2. inside the "setup" section, ensure that youhave the follwing key/value:
   
   key: log4j
   value: true
   
   
   
   
3. Navigate to the startup->opt section
4. Edit the "java_opts" key and ensure that it contains the following value:
   -server -Dlog4j.configurationFile=file:./log4j2.xml -XX:ErrorFile=./hs_err_pid.log
   
   
   
   
   2. edit the file (UIM_HOME)\Nimsoft\probes\service\ems\log4j2.xml using a text editor such as Notepad++ and find the value for "Root level" as seen in the below screenshot.  Set the value to "all".  This will ensure the majority of logging is done by Log4j2, and the log will be more likely to roll over at the appropriate size.
   
   If you use the values "info", "warn", "error", or "fatal",  the log may grow to a very large size based on Nimbus log entries before being rolled over; if you use "off" this may cause the log file to grow indefinitely.  
   
   

 

Alternate Workaround

 

If the above workaround is not reliable, it is possible to separate the logging frameworks to point at separate files.

Navigate to (UIM_HOME)\Nimsoft\probes\service\ems\.
Open log4j2.xml in a text editor.
Locate the <RollingFile> appender section.
Change the fileName and filePattern attributes to a new filename, such as ems_log4j.log.

Example Configuration:

<Appenders>
    <RollingFile name="ProbeLogFile" fileName="./ems_log4j.log" filePattern="./ems_log4j-%i.log">

 

With this change, the Nimbus logger will write to the standard ems.log based on the loglevel key in ems.cfg and will roll that log over to _ems.log based on the logsize key, while the ems_log4j.log will be written to by Log4j2 and will respect the SizeBasedTriggeringPolicy settings.

Standard probe initialization and "heartbeat" logs remain in ems.log (managed by ems.cfg), while the more detailed application-level logs are captured in ems_log4j.log, so for troubleshooting it may be necessary to consult both logs.

 

Additional Information

We are currently investigating a more robust, longer-term solution.  This is tracked as Feature ID: F165182: Improve Log Rollover in EMS probe.

Link to DX UIM Feature Tracker