High CPU Utilisation on Service Engines due to WAF XML traffic

Products

• VMware Avi Load Balancer

Issue/Introduction

• High CPU Utilisation on Service Engines due to WAF XML traffic
• Effectively inspect and troubleshoot XML-based payloads and add exceptions to the WAF-Policy using scripts.

Resolution

High CPU utilisation (100%) was observed on Avi Service Engine (SE) after migrating traffic  from L4 to L7 (WAF enabled).

The issue was traced to encrypted, signed, and Base64-encoded XML elements being analyzed by the WAF.

Avi cannot detect real attacks in such encrypted payloads, leading to false positives and excessive CPU usage.

 

Recommended Action Plan:

1. Exclude Encrypted XML Elements from WAF Inspection:

   • Add WAF exceptions for specific signatures affecting performance.
   • Navigate to: WAF Policy > Signatures > Edit CRS Group > CRS Rule > Add Exception.

2. Disable Unnecessary WAF Rules:

   • Disable the entire CRS group: "CRS_933_Application_Attack_PHP".
   • Disable unused database-related signatures, especially for Java apps:
     • 942200: Check for SQL Injection: MySQL (1/2)
     • 942300: Check for SQL Injection: MySQL (2/2)

We have scripts which can find the XML payload and add the necessary exception in the WAF Profile.

Step1: Generate the VS Pcap from Avi ALB UI - https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/avi-load-balancer/avi-load-balancer/30-2/monitoring-and-operability-guide/application-troubleshooting/packet-capture/capturing-virtual-service-traffic-using-cli-ui.html

Step2: Download the VS Pcap from Avi ALB UI and move it to system when script and tshark tool installed. Now untar the VS Pcap file using below command
#tar xvf vs_virtualservice-xxxx.tar

After untar you will see 2 files 1) .pcapng and 2) key.txt (SSL/TLS enabled VS SSLKEY file to decrypt the traffic in wireshark)

Step3: Install tshark tool using below command (Ref: https://tshark.dev/setup/install/)
#Ubuntu: sudo apt install -y tshark
#CentOS: sudo yum install -y wireshark

Step4: Check tshark version using below command on your machine
#tshark -v

Step5: Now store VS Name as NAME as variable & Run tshark tool from cli to parse and export the http body in .json format

#export NAME=<VS-NAME>
#tshark -o tls.keylog_file:${NAME}_key.txt -r ${NAME}.pcapng -T json -e http.request.method -e http.request.version -e http.request.uri -e http.request.line -e http.file_data > ${NAME}.json

Ex. VS PCAP File Name: vs_virtualservice-xxxcba.1234xxx.tar
#export NAME=vs_virtualservice-xxxcba.1234xxx

Step6: Create virtual python environment & Install the module to run setup_waf_exclusions.py script

#python3 -m venv waf_xml
#source waf_xml/bin/activate
#pip3 install avisdk
#pip3 install lxml

Step7: Run the script with VS Pcap HTTP Data (.json) file as input (which is generated with step5)

./waf_exclusions_from_http.py ${NAME}.json -s

===============
SAMPLE OUTPUT
===============
Recommended XML Exclusions:
 
XML://*[local-name() = 'DigestValue']
XML://*[local-name() = 'SignatureValue']
 
Done

Step8: Run the provided script on the Controller Leader Node's /tmp/ directory to add WAF policy exclusions (XML Payload) via command line

./setup_waf_exclusions.py -w <WAF-Policy-Name> -e "XML://*[local-name() = 'DigestValue']" -e "XML://*[local-name() = 'SignatureValue']"

Additional Information

 

• Minimum System Requirements: Ubuntu 24.04 LTS VM or CentOS 9 VM to run the WAF scripts.
• WAF Script: Please refer to the attached WAF script for further instructions.
• Install Dependencies: Install tshark and copy the waf_exclusion_from_http.py script onto the VM.
• Script Placement: Copy the setup_waf_exclusion.py script to the Controller Leader Node's /opt/avi/scripts directory.
• Make Executable: Grant execution permissions by running chmod +x setup_waf_exclusion.py.