Troubleshooting VM performance

Products

VMware vSphere ESXi

Issue/Introduction

This article goes through how to identify and troubleshoot VM performance:

The guest operating system boots slowly
Applications running in virtual machines perform poorly
Applications running in virtual machines take a long time to launch
Applications running in virtual machines frequently become unresponsive

Environment

VMware vSphere ESXI

Cause

Due that the VM did not had enough resources this was not able to to able to handle all the traffic which was causing out of buffers

Resolution

To determine whether the poor performance is due to CPU, memory or networking constraints:

Open ESXTOP on the host were this VM reside:

- ESXTOP overview for Performance Troubleshooting
  - For CPU click c

Field	Value	Description
%USED	N/A	CPU Core cycles used by a VM. High values are an indicator for VMs causing performance problems on ESXi Hosts.
%SWPWT	≥ 5	Counter showing how long a VM has to wait for swapped pages read from disk. A reason for this could be memory overcommitment. Pay attention if %SWPWT is >5!
%MLMTD	≥ 1	Counter showing percentage of time a ready to run vCPU was not scheduled because of a CPU limit setting. Remove the limit for better performance.
%CSTP	≥ 3	This value is interesting if you are using vSMP virtual machines. It shows the percentage of time a ready to run VM has spent in co-deschedule state. If value is >3 decrease the number of vCPUs from the VM concerned.
%RDY	≥ 10	Percentage of time a VM was waiting to be scheduled.If you note values between five and ten percent take care. Possible reasons: too many vCPUs, too many vSMP VMs or a CPU limit setting (check %MLMTD)

Note: Keep a close eye on the RDY as this is related to the Scheduler, due the bigger the value the Scheduler is the longer it will take to respond or process something.

- - For Memory click m

Field	Value	Description
MCTLSZ	≥ 1	Amount of guest physical memory (MB) the ESXi Host is reclaiming by balloon driver. A reason for this is memory overcommitment.
ZIP/s	≥ 1	Values larger 0 indicate that the host is actively compressing memory.
UNZIP/s	≥ 1	Values larger 0 indicate that the host is accessing compressed memory. Reason for this behaviour is memory overcommitment.
CACHEUSD	≥ 1	Memory (in MB) compressed by ESXi Host
SWCUR:	≥ 1	Memory (in MB) that has been swapped by VMKernel. Possible cause: memory overcommitment.
SWR/s, SWW/s:	≥ 1	Rate at which the ESXi Host is writing to or reading from swapped memory. Possible cause: memory overcommitment.

- - For Networking click n

Field	Value	Description
%DRPTX	≥ 1	Dropped Packages transmitted/Dropped Packages received. Values larger 0 are a sign for high network utilization
%DRPRX	≥ 1
Used-by/Team-PNIC	N/A	provide information what physical NIC a VM is actually using

Get out of buffers information:
- Run the following command and grep for the VM name:
  - net-stats -l | grep -i VMNAME

example output:

- <PortNumber> 5 9 <Switch Name> <MAC Address> VMNAME.eth0

After getting the VM info we are going to run the following command to to check if we have any out of buffers
- vsish -e get /net/portsets/<Switch Name>/ports/<PortNumber>/vmxnet3/rxSummary | grep "running out of buffers"

If we see the VM has a high number of out buffer that mean the VM is lacking resources --> Reference KB:

Large packet loss in the guest OS using VMXNET3 in ESXi

Note: Memory equals buffers and CPU equals speed which we move data, if we are unable to process the amount of buffers being sent by the application those will be going out of buffer or dropping which is going to be causing slowness on the performance.

If that is the Case Increase the amount of CPU and memory.

- Note: If this is is a third party application, reach out to the vendor so they can preform the changes require to make it work better.