SSP :Resolving MPS Excessive Queuing Time for Analysis Submission Alarms

Products

• VMware vDefend Firewall with Advanced Threat Prevention
• VMware vDefend Firewall

Issue/Introduction

MPS alarm: Excessive queuing time for analysis submission

Alarm Triggered:: malware_analysis.excessive_queuing_time

Summary: Analysis submissions in queue for more than 2 hours.  When the alarm is active, new malware analysis submissions are automatically rejected 

Environment

    
Security Services Platform (SSP) 5.2.0 and later and MPS activated with On-Premise Malware Analysis

Cause

Potential causes:

This may be caused by lack of resources or component failures.

Resolution

Consider decreasing the amount of file analysis and checking for alarms related to failures of malware analysis components
Decrease the amount of file analysis
To reduce the volume of new analysis data being stored, refer to the Malware Prevention Services (MPS) documentation for instructions on configuring which files to capture:

• Use the File Exclusion Filter: You can define specific directory paths (e.g., C:\Windows\Temp\) or file extensions (e.g., .log, .vmdk) to be explicitly excluded from analysis (see documentation at Filter Malware Exclusions). This minimizes unnecessary scanning of trusted or noisy applications.
• Refine Policy Configuration: Reduce the capture of files by narrowing the environments/VMs from which files are collected (see documentation Add Rules for Distributed Malware Prevention), or by reducing the specific file types that are configured for capture in your MPS profiles (see documentation Add a Malware Prevention Profile).
• Reduce Submission Rate Limits: Malware analysis can be configured to accept fewer submissions at a more granular level by tweaking the rate-limiting parameters in the malware-analysis-api deployment.

For convenience, you can run a bash script on the SSPI appliance to safely read and update these limits.

       1.  Download the script update_mps_submission_rate_limits.sh onto SSPI:

       2. Make the script executable:

      chmod +x update_mps_submission_rate_limits.sh

3. Run the script without arguments to view the current configuration (append `--ssp <instance_name>` if needed):

     ./update_mps_submission_rate_limits.sh

Output example:

  $ ./update_mps_submission_rate_limits.sh
=== Current Submission Rate Limits ===
    submission_rate_limiting:
      enabled: true
      rate: "42r/m"
      burst: 1260
      zone_size: "64k"


=== Usage ===
  To apply new limits, run the script with <NEW_RATE> and <NEW_BURST>:

  ./update_mps_submission_rate_limits.sh <NEW_RATE> <NEW_BURST> [--ssp <instance_name>]

  Example (Rate: 20r/m, Burst: 600): ./update_mps_submission_rate_limits.sh 20r/m 600

 Malware analysis submissions go through NGINX, which enforces the constant `rate` of submission strictly, but uses the `burst` queue to absorb instantaneous spikes.

4. Run the script to lower the limit (append `--ssp <instance_name>` if needed, for example, set <NEW_RATE> to `20r/m` and <NEW_BURST> to `600`):

./update_mps_submission_rate_limits.sh <NEW_RATE> <NEW_BURST>