Understanding High "Dropped Connections %" on Avi Service Engines
search cancel

Understanding High "Dropped Connections %" on Avi Service Engines

book

Article ID: 403943

calendar_today

Updated On:

Products

VMware Avi Load Balancer

Issue/Introduction

  • A high value for the Dropped Connections metric in the Avi UI may be observed when viewing Service Engine analytics.
  • This metric can often be misleading and does not necessarily indicate that the Service Engine is actively dropping packets due to an error or resource exhaustion.
  • Screenshot highlighting the metric in question:

Cause

  • The "Dropped Connections" percentage is derived from the Service Engine metric 'se_stats.pct_connections_dropped'.
  • This counter is incremented every time a TCP connection is terminated by a Reset packet (RST), regardless of the source of the reset.
  • Many legitimate network functions use RST packets to close connections. A common example is health monitors, which may terminate a check with an RST.
  • As a result, environments with numerous health monitors can show a persistently high "Dropped Connections" value as part of normal operations.
  • Therefore, this metric is a count of connections ending in a reset, not a direct measure of faulty packet drops by the Service Engine itself.

Resolution

To determine if the high metric is a cause for concern, you must perform a packet capture on the Service Engine. This allows you to differentiate between actual packet drops and the observed connection resets that increment the UI counter.

Verification Steps:

  1. Capture Packets: Initiate a packet capture on the affected Service Engine for 2-3 minutes. Ensure you capture 'All Traffic' to get a complete picture. After the capture is complete, download and extract the resulting files.
  2. Check for Actual Packet Drops:
    1. Locate and open the '<se-uuid>_dropped_core_0.pcapng' file.
    2. This file contains only packets that were dropped by the Service Engine. If this file is empty or does not contain any relevant TCP data connection packets, it confirms the Service Engine is not the source of dropped traffic.
  3. Investigate Connection Resets (RSTs):
    1. Next, open the main traffic file, '<se-uuid>_se_core_0.pcapng', in a tool like Wireshark.
    2. This file contains the normal traffic flow. Apply the display filter tcp.flags.reset==1 to isolate all the RST packets that increment the "Dropped Connections" counter.
  4. Assess the Source:
    1. Examine the source and destination IPs of the filtered RST packets from the 'se_core_0.pcapng' file.
    2. If the resets originate from expected connections or sources, such as health monitors or applications that normally terminate sessions this way, then the high "Dropped Connections" count is benign and can be safely ignored.
    3. If the resets are from unexpected sources, it indicates an issue with the client or backend application that warrants further investigation. 

Addressing the root cause of unexpected RST packets will not only resolve potential application issues but also allow the "Dropped Connections" metric to be used as a more accurate indicator of network health.

Please reach out to Broadcom Support if you need assistance with performing the service engine packet capture.

Additional Information

Powered by Wolken Software