vDefend SSP Alarm: Delay Detected In Messaging
search cancel

vDefend SSP Alarm: Delay Detected In Messaging

book

Article ID: 384121

calendar_today

Updated On:

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

  • You are running SSP 5.0 and later.
  • Rawflow lag alarm is observed: "Delay Detected In Messaging Rawflow."
    • Indicating that all flows processing is experiencing lags
  • Overflow lag alarm is observe: "Delay Detected In Messaging Overflow."
    • Indicating that external flows processing is experiencing lags
  • Pairable-flow lag alarm is observed: "Delay Detected In Messaging Pairableflow."
    • Indicating that internal flows processing is experiencing lags

Environment

SSP 5.0

Cause

The primary cause of this issue is that the volume of traffic flows exceeds the system's processing capacity.

When the flow volume is excessively high and there is significant uniqueness in the flows, the system's processing rate may degrade.

The 'Messaging Delay' alarms are triggered when the System Storage Platform (SSP) messaging module is unable to process incoming flows at the required rate.

Resolution

Recommended Workaround: Scale out Analytics and Data Storage services

Traffic flows are stored across both the Analytics and Data Storage services. The Analytics service requires a minimum of five nodes to scale out, whereas the Data Storage service requires a minimum of eight nodes.

To determine the recommended number of worker nodes for the current traffic flow volume, utilize the SSP Sizing Tool. For detailed instructions on using this tool, refer to the relevant KB article: https://knowledge.broadcom.com/external/article/373793/security-intelligence-sizing-tool.html

Prerequisites:

  • All existing nodes in your Kubernetes cluster must be in a healthy and ready state before you can scale out the Security Service Platform.
  • Before proceeding with the scale-out procedures, ensure that your infrastructure administrator has already allocated the minimum number of nodes required for scaling out the SSP services.

Procedure

  1. From your browser, log in with Enterprise Admin privileges to SSP at https://<ssp-fqdn>.
  2. Navigate to System - Infrastructure - Platform & Services.
  3. In the bottom-left corner of the Platform & Services section of the UI page, click Scale Out button.

    Note: The Scale Out action is only supported if you deployed the SSP using the Advanced form factor. The action is not supported for Evaluation form factor deployment.

    If all of the services are scaled out already, the Scale Out button is disabled on pop up dialog. In this case, it indicates that your cluster nodes have reached the maximum number of nodes allocated. Initially, the advanced form factor is deployed with four nodes. You must first request for your infrastructure administrator to add four more nodes to your current cluster before you can continue with the next steps. To scale out all of the services, you must have a total of eight worker nodes in your cluster.

  4. Select the All checkbox.
  5. In the Advanced Options section, ensure that all of the services available for the scale-out action are selected.

    Unless specifically advised by the Broadcom support team, ensure that all of the core services are selected so that the system can decide which of the core services must be scaled out. Scaling out one core service arbitrarily can lead to more resources being used without any improvement to the system performance. Before proceeding with single-category service scale out procedure, consult the Broadcom support team or confirm that you know clearly what can happen if you scale out a single-category service.

  6. Click Scale Out.

    The UI displays the progress of the scale out operation.

For reference please review the "Scale Out" section(WIP) the following guide: https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/vdefend/security-services-platform/5-0.html

 

Other options: 

Note: Please try the primary workaround to scale out first before trying to the following options.

If the recommended number of worker nodes exceeds the maximum supported limit, or if scaling out to the recommended size is not currently feasible, consider implementing the following options:

 

option 1: Configure Data Collection in SSP

If you can identify the ESXi hosts and vSphere clusters with mostly East-West (EW) traffic, for example over 90% of traffic is EW and 10% is North-South (NS), you can enable data collection for those EW traffic first and gradually enable for NS. North-south traffic tends to have more unique IPs, which is more likely to adversely affect the data compaction.

This will help alleviate the high storage growth, while other tuning options are explored below.

Procedure:

By default, SSP collects network traffic data on all standalone hosts and clusters of hosts. If necessary, you can optionally stop data collection from a standalone host or cluster of hosts.

  1. From your browser, log in with Enterprise Administrator privileges to an SSP at https://<ssp-fqdn>.
  2. In the SSP UI, select System tab Settings section, select Data Collection.
  3. To manage traffic data collection for one or more hosts, perform one of the following steps.
    The system updates the Collection Status value for each affected host to Deactivated or Activated, depending on the data collection mode you had set.
    1. To stop traffic data collection, select the host or hosts in the Standalone Host section, click Deactivate, and click Confirm when prompted if you are sure.
    2. To start traffic data collection, select the host or hosts, click Activate, and click Confirm when prompted if you are sure.
  4. To manage traffic data collection for one or more clusters of hosts, perform one of the following steps.
    1. To stop data collection for one or more clusters, select the cluster or clusters in the Cluster section, click Deactivate, and click Confirm when prompted if you are sure.
    2. To start traffic data collection, select the cluster or clusters, click Activate, and click Confirm when prompted if you are sure.

For reference please review the "Configure SSP Settings" section(WIP) in the following guide: https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/vdefend/security-services-platform/5-0.html 

 

option 2: Filter out broadcast and/or multicast flows.

Note: This option can be used where broadcast and/or multicast flows are not required for security policy or similar guidance. If broadcast and/or multicast flows are important to you, do not enable this option.

You can disable broadcast and/or multicast flows from getting stored in SSP to reduce disk usage.

This will only affect new flows which are not yet processed by SSP. Existing broadcast/multicast flows will still be visible, until the retention period (30 days) is reached.

 

For detailed information on how to achieve this , please contact Broadcom Support for further assistance.

 

Powered by Wolken Software