High co-stop (%CSTP) values seen during virtual machine snapshot activities

Products

VMware vSphere ESXi

Issue/Introduction

One or more snapshots are present on the virtual machine
High CPU utilization is seen within the virtual machine Guest Operating System
Under esxtop, sustained, high values are seen for the VM %CSTP (co-stop) value

Resolution

Virtual machine performance can be adversely affected during snapshot operations for a number of reasons, both due to their nature and due to enviromental issues. For more information about snapshots and their behavior, see Understanding virtual machine snapshots in VMware ESX (342618).

Snapshots introduce complexity to storage I/O. Due to the nature of snapshots, every read operation must traverse every snapshot disk and then the base disk in order to verify the appropriate disk block to return. Because these extended read operations are required, snapshots are the most performance-intensive disk format for virtual disks (as opposed to thin-provisioned, thick-provisioned, or eager-zeroed thick-provisioned virtual disks).

As storage I/O for snapshots grows, co-stop (%CSTP) values for a VM with multiple vCPUs can increase as the vCPUs wait on I/O completion.

To reduce the high %CSTP values and increase virtual machine performance, consolidate any snapshots into the main virtual disk. After consolidation, the %CSTP value is reduced or eliminated and VM performance is improved. If performance is not improved enough, continue researching other potential VM performance issues.

For more information, see:

Additional Information

As the size and number of snapshots on a virtual machine increase, so does the number of storage command operations within vmkernel. For each storage command issued by the virtual machine guest OS, multiple storage command operations may be necessary to traverse the entire snapshot chain to read the most appropriate block of data.

On production VMs with snapshots and high storage I/O (storage commands), or VMs with multiple snapshots in a chain, the size and complexity of snapshots can increase rapidly. This, in turn, can lead to an increase in the storage I/O required to complete each guest OS I/O action.

Performance impacts due to snapshot usage must be taken into account along with VM usage. Desktops, development, and low-use production servers may never see a noticeable performance hit during snapshot usage that follows best practices. High-usage production servers may need to be tuned to account for performance impacts of snapshots.

Tuning for snapshot usage may include policy-based options such as specific time windows for snapshot usage or size and age of snapshots. Tuning may also include an evaluation of the storage hardware that snapshots are created on. As snapshots reside in the same folder as the base disk by default, storing a VM's primary OS disk on a low-tier storage means snapshots reside there as well. If virtual machine data disks normally expected to be on high-tier storage are working off of low-tier storage snapshots, severe performance impacts may occur.

Troubleshooting a converted virtual machine that experiences poor performance
Understanding VM snapshots in ESXi / ESX
Best practices for using snapshots in the vSphere environment