DevTest 10.9.1 Upgrade: Rapid RDS Storage Consumption Due to VSE Event Accumulation

Products

• Service Virtualization

Issue/Introduction

After upgrading to DevTest 10.9.1, the RDS database storage fills up rapidly, even after increasing the allocated storage. This issue is typically observed when the system is under load.

Cause

The rapid storage increase is caused by the accumulation of VSE event history in the VIRTUAL_SERVICE_EVENT database table. The default persistence and cleanup settings may not be sufficient for environments with high transaction volumes post-upgrade.

Resolution

To address this issue, perform an immediate manual cleanup and adjust your long-term configuration settings.

Immediate Action: Reclaim Database Space

1. Purge Aged Rows: Have your Database Administrator (DBA) manually purge aged rows from the VIRTUAL_SERVICE_EVENT table in batches. Do not rely solely on the default automatic hourly job to clear large volumes of data.
2. Reclaim/Vacuum: After the purge, perform a storage reclaim or vacuum operation specific to your RDS engine. This ensures the space is released back to the database file system rather than just marked as free space internally.

Long-Term Configuration

Modify your local.properties file to manage event persistence according to your environment's requirements:

Option 1: Disable Event Persistence (If not required) If you do not need VSE event history persisted in the database:

devtest.virtual_service_environment.events.persistence.enabled=false

Option 2: Tighten Retention and Cleanup (If history is required) If you need to keep event history, adjust the retention and cleanup frequency to be more aggressive.

• Maximum Age: Set how long events are kept (default is 7200 seconds).
• Purge Period: Set the frequency of the cleanup job (default is 600 seconds).

devtest.virtual_service_environment.virtual_service_event.expiration.maximum_age_in_seconds=7200

devtest.virtual_service_environment.virtual_service_event.expiration.purge_period_in_seconds=600

Option 3: Review Transaction Threshold Check your typical transaction-per-second (TPS) rate against the default threshold (16 TPS). If your environment consistently exceeds this, you may need to adjust the threshold:

devtest.virtual_service_environment.virtual_service_event.transaction_count_per_second_threshold=16