Intermittent High Filesystem Utilization and Archive File Retention Issues

Products

• CA Application Performance Management (APM / Wily / Introscope)

Issue/Introduction

We are regularly seeing high utilization (above 80%) on the enterprise manager path /usr/sap/ccms, which currently has a size of 32GB. This triggers filesystem alerts, but the utilization drops below the threshold and the alerts close before they can be investigated.

Additionally, large data files are being generated inside the /data/ folder, with some crossing 1–1.5 GB. We have also noticed that .data.gz files under the /data/archive/ path are residing longer than the retention period defined in the properties file and are not clearing automatically.

Environment

APM 10.8 SP1

Introscope

Resolution

Follow these steps:

1. Tune Data Retention: Open your IntroscopeEnterpriseManager.properties and consider reducing the transaction event retention period. For example, set introscope.enterprisemanager.transactionevents.storage.max.data.age=7 (days) to help clear space in the data folders.
2. Address Archive Clearing: Ensure that the SmartStor cleanup properties are correctly set. If .data.gz files are persisting beyond their defined time, we may need to verify if the maintenance threads are encountering any errors in the IntroscopeEnterpriseManager.log.
3. Implement Log Rotation: To prevent em.log from contributing to disk usage, you can configure log rotation. This can be done by unsetting lax.stdout.redirect=console in Introscope_Enterprise_Manager.lax or using a standard Unix cron job to truncate the log file regularly.
4. Apply Query Limits: Add the following properties to IntroscopeEnterpriseManager.properties to prevent heavy historical queries from impacting system performance:
   • introscope.enterprisemanager.query.datapointlimit=100000
   • introscope.enterprisemanager.query.returneddatapointlimit=100000