Troubleshooting Slow Storage vMotion

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

• During Storage vMotion, performance may be substantially reduced when migrating virtual machines between datastores. This behavior can occur even under optimal conditions (such as off-peak usage periods), resulting in migration durations that are considerably longer than anticipated.
• This article aims to assist in reviewing the logs and determining if there are any underlying Storage performance issue.
• Primary symptoms can be any of the below:
  • Extended migration times for Storage vMotion operations
  • Poor throughput during data transfer
  • Higher than normal storage latency
  • Increased host CPU utilization during migrations

Environment

• VMware ESXi 7.x
• VMware ESXi 8.x

Cause

Several factors can contribute to degraded Storage vMotion performance:

• Disabled or misconfigured VAAI (vStorage APIs for Array Integration)
• High storage latency
• Time synchronization issues
• Outdated system firmware or drivers
• Suboptimal storage network configuration

Resolution

Step 1: Analyze Key Log Files

• Check VAAI Status in /var/log/hostd.log:
  
  YYYY-MM-DDTHH:MM.SSZ Unable to connect to vaai-nasd socket [No such file or directory]
  
  This indicates VAAI is not functioning properly.
  
  
• Monitor Storage Latency through /var/run/log/vmkernel.log:
  
YYYY-MM-DDTHH:MM.SSZ cpu##:20982##)WARNING: NFS: 5015: NFS volume #######_01 performance has deteriorated. I/O latency increased from average value of 50##(us) to 1170##(us).
  
  This indicates a significant degradation in storage performance.

High latency indicators:

• • Max latency > 1000ms (1000000 µs)indicates severe performance issues and can crash VMs
  • High latency > 40ms (40,000 µs) provides noticeably slow performance for intensive operations such as storage vMotion
  • Median latency > 25ms (25,000 µs) suggests consistent performance problems
  • Lower latencies < 20ms (20,000 µs) are usually noticeable only with very high input/output applications
• Examine Migration Progress in /var/run/log/vpxa.log:
  
  YYYY-MM-DDTHH:MM.SSZ verbose vpxa[#####] Immigrating VM at path /vmfs/volumes/<volume-id>/<vm-name>/<vm-name>.vmx has vmid ##
YYYY-MM-DDTHH:MM.SSZ verbose vpxa[#####] Finished tracking destination of migration

These entries help track migration duration and completion status.

• Review Available Storage Space:
  NFS         Public    volume-id    volume_name    total    used    free   %used
                                                 256#.00  230#.91  25#.09  9#.#%
  Low free space can impact performance.

Step 2: Performance Metrics Analysis

• Calculate throughput using these formulas:
  • Transfer Duration = (Migration End Time - Start Time)
    Average Throughput = (Total Data Size / Transfer Duration)
  • Example calculation:
    For a 100GB VM with 27-minute transfer
    • Throughput ≈ 62 MB/s (indicates poor performance if using 10GbE networking)
• This detailed log analysis helps identify:
• • Storage array performance issues
  • Network bottlenecks
  • Resource constraints
  • Configuration problems
  • Consider this an expected baseline for 10GbE networks:
    • Good performance: > 500 MB/s
    • Acceptable performance: 200-500 MB/s
    • Poor performance: < 100 MB/s

Step 3: Implement Performance Optimizations

Based on log analysis results:

• If VAAI issues are detected:
  • Engage storage vendor to enable and configure VAAI support
  • Verify storage configuration
• If high latency is observed:
  • Review storage network configuration
  • Check for network or storage path congestion
  • Verify storage array performance
• If system-level issues are found:
  • Update system firmware and drivers
  • Configure proper time synchronization
  • Perform system maintenance as needed

Additional Information

• Frequently Asked Questions for vStorage APIs for Array Integration
• Performance Best Practices for VMware vSphere 8.0
• Migration with Storage vMotion