GemFire: "Error: java.lang.StringIndexOutOfBoundsException" during startup due to conflicting log file names

Products

• VMware Tanzu Gemfire
• VMware Tanzu Data Intelligence
• VMware Tanzu Data Suite
• VMware Tanzu Data Suite
• VMware Tanzu Greenplum / Gemfire

Issue/Introduction

A VMware Tanzu GemFire member may fail to start automatically following a reboot or service restart. The process terminates unexpectedly, and log analysis reveals a "StringIndexOutOfBoundsException" Java exception during the log rollover initialization. This behavior is triggered when external log rotation tools create files in the active log directory using a naming convention that conflicts with the internal GemFire log management handler. 

Symptoms:

• GemFire member fails to start, terminating with exit status 1.
• The startup logs report the following error: java.lang.StringIndexOutOfBoundsException.
• The stack trace includes the following method: org.apache.geode.internal.io.MainWithChildrenRollingFileHandler.calcNextChildId).
• Historical log files created by external processes exist in the active log directory with patterns similar to *.log-YYYY-MM-DD[-HH].

Cause

External log rotation (e.g., Linux logrotate) produces files with a timestamp suffix (such as *.log-YYYY-MM-DD-HH) that matches the internal log naming patterns used by GemFire. When the MainWithChildrenRollingFileHandler attempts to process these existing historical log files during the startup sequence, it encounters a parsing error, resulting in a StringIndexOutOfBoundsException and a failed member startup.

Resolution

1. Identify the log rotation configuration currently in use by your external job (e.g., Control-M or Linux logrotate).
2. Modify the dateformat or suffix in your log rotation script to use a naming convention that does not overlap with GemFire’s internal patterns. A recommended pattern is _archived_%Y%m%d_%H instead of -YYYY-MM-DD-HH.
3. Ensure the active GemFire log directory is cleared of any existing files that match the conflicting naming pattern.
4. Configure GemFire's internal log management properties to manage log lifecycle and prevent directory size issues automatically:
   • Add log-file-size-limit=### (e.g., 100MB) to your gemfire.properties or server configuration.
   • Add log-disk-space-limit=#### (e.g., 1000MB) to your gemfire.properties or server configuration.
5. Restart the GemFire member and verify that startup completes successfully without exception.