Live storage vMotion fails when Software Memory Tiering is enabled in vSphere 8.x

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Unable to perform a live Storage vMotion for virtual machines (VMs) while they are powered on.
• Storage vMotion for powered-off VMs (cold migration) works as expected.
• Combined compute and storage vMotion (Cross-host Storage vMotion) may succeed, but storage-only migration fails.
• VM Snapshots with Memory fails with error "Snapshots are not yet supported on a system with Software Memory Tiering enabled"
• Fault Tolerance (FT) cannot be enabled or fails to function.
  
  
• VMs vmware.log shows following information during storage vMotion prepare task:
  
  /vmfs/volumes/###########/VM/vmware.log

YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - VigorTransportProcessClientPayload: opID=###########: Receiving Migrate.PrepareSource request.
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - MigrateVMXdrToSpec: type: 2 srcIp=<#####> dstIp=<#####> mid=########### priority=no checksumMemory=no maxDowntime=0 latencyAware=yes diskOpFile= srcLogIp=<<unknown>> dstLogIp=<<unknown>> ftPrimaryIp=<<unknown>> ftSecondaryIp=<<unknown>>
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - MigrateVMXdrToSpec: scsi:0:0 -> /vmfs/volumes/###########/VM/VM.vmdk, parent:null, parentShared: 0, rdmDevice: null,transform: 1, numSnapshots: 0, policy: 'null', independentFilters: ''
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - MigrateVMXdrToSpec: encryptedVMotion: 0
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - MigrateVMXdrToSpec: type 2 unsharedSwap 1 memMinToTransfer 0 cpuMinToTransfer 0 numDisks 1 numStreamIps 0 numFtStreamIps 0
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - Received migrate 'to' request for mid id ########, src ip <#####>, dst ip <######>(invalidate source config).
YYYY-MM-DDTHH:MM:SS.176Z In(05) vmx - Migrate: FSR is not yet supported on a system with Software Memory Tiering enabled.

Environment

VMware vSphere 8.x

Cause

In vSphere 8.x, Software Memory Tiering does not currently support the Fast Suspend and Resume (FSR) mechanism. FSR is a core technology used by vSphere to perform live storage migrations, memory-inclusive snapshots, and Fault Tolerance operations.

When these operations are initiated, the VMX process identifies the incompatibility and terminates the task to prevent instability.

Resolution

Support for FSR based operations (including live storage vMotion) on systems with Software Memory Tiering enabled is introduced in vSphere 9.0.

To resolve this limitation and enable live Storage vMotion with memory tiering, upgrade to vSphere 9.0 (or VMware Cloud Foundation 9.0).

Workaround:

If an upgrade is not immediately possible, use one of the following methods to migrate storage:

1. Cold Migration: Power off the virtual machine before initiating the Storage vMotion.
   
   
2. Change Both Compute and Storage: Perform a migration that moves both the compute resource (host) and the storage simultaneously. 
   
   
3. Disable Memory Tiering: If live storage-only vMotion is critically required and the performance benefits of memory tiering are not needed, disable Software Memory Tiering on the affected ESXi hosts. Note that this requires a host reboot.

Additional Information

Using the "Memory Tiering over NVMe" feature in vSphere 8.0 Update 3.