High Network Latency and CPU Ready Time on Virtual Appliances with CPU Limits

Products

• VMware NSX

Issue/Introduction

Virtual appliances (such as Juniper SRX firewalls or similar) may experience significant network latency, ping drops, or overall performance degradation. In the vSphere Client performance charts and esxtop command outputs, these virtual machines (VMs) show high CPU Ready (%RDY) times, even when the host’s overall CPU utilization appears low.

Environment

VMware ESXi 8.x

VMware NSX 4.2.3

Cause

This issue often occurs when a CPU limit is configured on the virtual machine.

• Even if the ESXi host has abundant CPU resources available, a configured CPU limit forces the VM to wait for scheduler cycles to prevent it from exceeding the arbitrary limit set by the administrator.
• This scheduling delay is reflected in the high CPU Ready (%RDY) metric, causing the VM to fail to process incoming network packets in a timely manner.

Resolution

To resolve this issue and improve performance for latency-sensitive virtual appliances:

1. Remove CPU Limits:

   • Navigate to the VM in vCenter.
   • Go to Edit Settings > VM Options > Resource Allocation.
   • Set the CPU limit to Unlimited.

2. Use CPU Reservations (Recommended):

   • For network and security appliances, it is highly recommended to prioritize CPU Reservations over CPU limits to guarantee physical resource availability and prevent artificial throttling.

3. Calculate Limits (If Required):

   • If business requirements mandate a CPU limit, calculate the maximum possible limit by multiplying the number of vCPUs by the base clock speed of the physical host processors.
   • Formula: Number of vCPUs × Physical Host Base Clock Speed (MHz) = Maximum Limit
   • Example: A 5 vCPU VM on a 2.4 GHz (2400 MHz) processor = 12,000 MHz maximum limit.