Intermittent DNS Latency and Packet Drops observed after enabling Turbo (SCRX) Mode
Article ID: 439734
Updated On:
Products
Issue/Introduction
In environments where Turbo Mode (SCRX) is enabled along with a DFW L7 DNS rule, traffic that matches the L7 DNS rule is redirected to the SCRX engine for Deep Packet Inspection (DPI). This issue can result in intermittent DNS resolution delays, DNS failures, and/or application disconnects. The issue occurs even when no IDS/IPS rules are configured.
Symptoms Include:
- High rates of dropped packets for DNS traffic (UDP/TCP 53) reaching Domain Controllers.
- Intermittent Kubernetes/OpenShift authentication failures for containers authenticating via Active Directory.
- Latency or timeouts in environments heavily reliant on Layer 7 (L7) FQDN filtering rules in the Distributed Firewall (DFW).
- nslookup failures or intermittent "client connection lost" errors during peak traffic periods.
- The following errors are seen in vmkernel.log within the ESXi logs:
<date> In(182) vmkernel: cpu0:########)SCRX SVM: injection to iochain failed, portId:########, mayModify:1, VMK_IS_DISABLED:0 (Out of slots)
<date> In(182) vmkernel: cpu0:########)SCRX SVM: injection to iochain failed, portId:########, mayModify:1, VMK_IS_DISABLED:0 (Out of slots)
Environment
VMware NSX / vDefend Firewall 4.2.2 and later
Cause
This issue is caused by a priority inversion condition during DNS flow cleanup, which introduces significant latency in DNS processing, in some cases reaching up to 5 seconds. This latency can lead to DNS resolution failures across the network.
The issue is primarily observed in environments with mixed DNS traffic patterns, including valid DNS transactions, NS requests containing empty domain names, transactions returning zero response records, and queries that do not receive any response.
Resolution
A patch is available for those who hit this issue. If you suspect you are encountering this issue, please open a Support Request (SR) with Broadcom Support.
PR 3684776
Feedback
