Packet drops on Cisco ASAv virtual machines caused by high CPU ready time

Products

• VMware vSphere ESXi

Issue/Introduction

• Cisco ASAv (Adaptive Security Virtual Appliance) virtual machines experience packet drops at the virtual network interface level. The drops are observed on vmxnet3 interfaces and occur even when guest operating system CPU utilization does not indicate saturation.
• Performance analysis shows that the virtual machine is frequently waiting for physical CPU resources due to vSphere CPU scheduling delays. As a result, incoming network traffic cannot be processed in a timely manner, leading to receive (RX) buffer overflows and packet loss.

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x

Cause

The issue is caused by high CPU Ready time on the ASAv virtual machine. High CPU Ready occurs when the ESXi scheduler cannot immediately allocate physical CPU cores to all configured vCPUs, commonly due to CPU contention, overcommitment, or scheduling co-stop delays.

This condition is typically observed when:

• The ASAv virtual machine is oversized with more vCPUs than required

• The ESXi host is heavily overcommitted

• CPU limits are configured on the virtual machine

• Insufficient CPU reservations are in place for a latency-sensitive workload

Resolution

vSphere Infrastructure Configuration: 

1. Right-size vCPUs
   Reduce the vCPU count on oversized ASAv virtual machines. Lower vCPU counts reduce scheduler contention and improve execution frequency.

2. Configure CPU Reservations
   Apply a CPU reservation greater than the default 1000 MHz, based on the ASAv’s typical or peak CPU usage, to guarantee CPU availability.

3. Remove CPU Limits
   Ensure that no CPU limits are configured on the ASAv virtual machine, as limits can artificially increase CPU Ready time.

4. Manage CPU Overcommitment
   Reduce the physical CPU to virtual CPU overcommit ratio on the ESXi host and avoid placing multiple high-throughput network appliances on the same host.

ASAv-Specific Recommendations:

1. Align vCPU Count with License Tier
   Ensure the configured vCPU count matches the licensed throughput tier and avoid assigning more vCPUs than supported by the license.

2. Scale Out Instead of Scaling Up
   Deploy multiple smaller ASAv instances (2 - 4 vCPUs) instead of fewer large instances to reduce scheduling delays and improve packet processing performance.

Additional Information

• High CPU Ready time prevents timely servicing of vmxnet3 receive queues, resulting in packet drops before traffic reaches the guest operating system.

• CPU Ready time can be monitored using vCenter performance charts, esxtop, or vRealize Operations.

• Network and security appliances should be treated as real-time workloads and sized accordingly.