Impact on VM performance when Compute and storage vMotion is performed with Virtual Disk IOPS - Limit on VM.

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• It is observed that xvMotion process is taking longer time to complete and it may impact the virtual machine performance during the time of migration.
• Compute and storage vMotion (also called as xvMotion) can do two different types of storage copy:   
  • a. Local storage copy: Source host has access to both source and destination datastore. Source host drives the copy without the involvement of the compute network and destination host.
  • b. Remote storage copy: Source doesn't have access to the destination datastore. So source host relies on the compute network to connect to destination ESX host which copies the data to destination datastore.
• This issue is only seen for remote storage copy during compute and storage vMotion.

Sample log snippets from Virtual Machine's vmware.log:

2023-03-16T04:44:19.017Z| vmx| | W003: SVMotion: scsi0:0: Disk transfer rate slow: 0 kB/s over the last 10.00 seconds, copied total 0 MB at 0 kB/s.

2023-03-16T04:44:29.022Z| vmx| | W003: SVMotion: scsi0:0: Disk transfer rate slow: 0 kB/s over the last 10.01 seconds, copied total 0 MB at 0 kB/s.

2023-03-16T04:44:39.025Z| vmx| | W003: SVMotion: scsi0:0: Disk transfer rate slow: 0 kB/s over the last 10.00 seconds, copied total 0 MB at 0 kB/s.

2023-03-16T04:44:49.027Z| vmx| | W003: SVMotion: scsi0:0: Disk transfer rate slow: 6552 kB/s over the last 10.00 seconds, copied total 64 MB at 1310 kB/s.

2023-03-16T04:44:59.037Z| vmx| | W003: SVMotion: scsi0:0: Disk transfer rate slow: 0 kB/s over the last 10.01 seconds, copied total 64 MB at 1091 kB/s.

Environment

VMware vSphere ESXi
VMware vCenter Server

Cause

The remote storage copy is programmed to run at GB/sec speeds. Having an IOPS limit which is lower than this rate will limit the rate at which the remote storage copy runs leading to slower than usual rate of copying the disk.
This would also mean that remote storage copy stream will dominate and take the majority of this IOPS limit, hardly leaving anything for the VM's workload I/O. So the VM performance is affected during remote storage copy. As a result, both the disk copy and the VM performance gets affected.

Resolution

This is a known issue and a fix is being worked on in the upcoming release. Please subscribe to the KB article to get alerted when the issue is fixed.

Workaround:

• Disable the IOPS limit on the disk when performing Compute and Storage vMotion.

Additional Information

• Disk IOPS limit on VM would not impact storage vMotion.
• Disk IOPS limit on VM would not impact compute and storage vMotion when storage copy is locally driven by the source ESX host without the involvement of destination ESX and compute network. In case of the local storage copy, it gets it owns scheduler queue without any limits.