High disk usage alarms are constantly seen in NSX for Malware Prevention SVM.

Products

• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

High SVM disk usage alarms are seen in NSX. There is very little or no space left in "/filestore" partition in the SVM.

Environment

SSP 5.1.0 deployed.

Malware Prevention Service (MPS) enabled.

SVMs deployed on selected compute clusters.

Cause

This issue occurs in environments experiencing high traffic volumes, particularly when:

• Workload VMs generate or transfer large files.
• File inspection requests occur at a high rate.

In such scenarios:

• Files are temporarily stored in the SVM’s /filestore partition for analysis.
• The rate of file ingestion exceeds the rate of analysis and cleanup.
• A periodic cron job is configured to remove files older than 4 hours.
• The cleanup job runs every 4 hours and deletes files older than 4 hours.

When traffic is sustained at a high rate, the partition fills faster than the scheduled cleanup frequency, resulting in disk accumulation and alarms.

Resolution

NOTE: This issue is fixed in SSP5.1.1

1. Manually cleanup old files accumulated in "/filestore" partition.

2. Modify cleanup cron job to run more frequently. Currently, the job runs every 4 hours and deletes files older than 4 hours.

 

Steps to manually cleanup old files:

1. Log into SVM via SSH,

OR

Access SVM via console.

2. Go to /filestore partition and delete all the files.

 

Steps to modify the cleanup cron job:

1. Login/SSH as root to the SVM facing high disk usage alarm.

 

2. Edit cleanup configuration:

vi /opt/vmware/nsx-sh/scripts/config.py

 

3. Find:

FILE_AGE_THRESHOLD_IN_HR = 4

 

4. Change it to a lower value (e.g. 1 or 2 hours):

FILE_AGE_THRESHOLD_IN_HR = 1

 

5. Save and exit.

 

6. Edit cron schedule:

vi /etc/cron.d/file_cleanup

 

7. Modify frequency:

Look for number "4" in the regex pattern, and change it to a lower value (e.g. 1 or 2) e.g regex pattern looks like "0 */4 * * * nsx-sh python3 /opt/vmware/nsx-sh/scripts/stale_file_monitor.py"

 

8. Save and exit.