Diagnosing and Resolving vMotion Performance Issues in 1 Gbps Network Environments

Products

• VMware vSphere ESXi

Issue/Introduction

• Users may experience severe network performance degradation and connectivity loss events during vMotion tasks in environments using 1 Gbps network interfaces. This article provides guidance on diagnosing throughput issues and offers resolution steps for improving vMotion performance.
  
  
• The following messages may be observed in /var/log/vmkernel.log when a 1 Gbps network is in use:
  Migrate: 616: ... totalVmknicLinkspeed=1000bps
VMotion: 7359: ... Using 1 threads for streaming
  
  
• The following messages may be observed in /var/log/vmkernel.log when migration of large virtual machines timeout:
  WARNING: VMotionUtil: 832: ########## D: failed to read stream keepalive: Connection closed by remote host, possibly due to timeout
Migrate: 101: ########## D: MigrateState: Failed
WARNING: Migrate: 257: ########## D: Failed: Connection closed by remote host, possibly due to timeout (0xbad003f)
WARNING: Migrate: 7074: ########## D: Migration considered a failure by the VMX. It is most likely a timeout, but check the VMX log for the true 

Environment

ESXi hosts with 1 Gbps network interfaces

Cause

vMotion operations can saturate 1 Gbps network links, especially when multiple concurrent vMotions occur. This saturation can lead to network-wide performance issues and connectivity problems for other services.

Resolution

1. Diagnose throughput issues:
   1. SSH into the ESXi host and use the following command to monitor vMotion network traffic:

      • esxtop

   2. Press 'n' to switch to the network view.
   3. Look for the vmknic associated with vMotion (usually vmk1) and observe the MbTx/s (Megabits transmitted per second) column during a vMotion operation.
   4. Calculate the average throughput over the duration of the vMotion:
      • Average Throughput (Mbps) = (VM Memory Size in MB * 8) / vMotion Duration in Seconds
        
        
2. If the average throughput approaches or exceeds 800 Mbps on a 1 Gbps link, consider the following steps:
   1. Upgrade network interfaces to 10 Gbps or higher.
   2. Configure a dedicated vMotion network stack:

      • esxcli network ip interface tag remove -i vmk1 -t VMotion

      • esxcli network ip interface set -i vmk1 -s vmotion

      • esxcli network ip interface tag add -i vmk1 -t VMotion

   3. Separate vMotion traffic onto its own VLAN.
   4. Implement vMotion scheduling and throttling:
      • In the vSphere Client, go to Host > Configure > System > Advanced System Settings
      • Set Migrate.NetTimeout to a higher value (e.g., 300 seconds)
      • Set Migrate.NetRetries to a lower value (e.g., 10)
   5. Optimize vSphere Distributed Switch (VDS) configuration:
      • Enable Network I/O Control (NIOC)
      • Configure load balancing policy to "Route based on physical NIC load"
      • Implement traffic shaping policies for vMotion traffic
        
        
3. Verify the changes:
   1. Perform a test vMotion and monitor the network throughput using the esxtop command.
   2. Confirm that other network services are not impacted during vMotion operations.

Additional Information

- Networking Best Practices for vSphere vMotion
- Troubleshooting networking issues when using VMware products
- vSphere Network I/O Control