SMP-FT turns off during high I/O operations for a VM

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• During high I/O operations for a virtual machine with SMP-FT enabled, you see Fault Tolerance is turned off.
• In the /var/run/log/vmkernel.log, you see the entries similar to:
  
  YYYY-MM-DD T15:43:34.759Z cpu14:32917)WARNING: FTCpt: 1024: (514736172521 pri) Error allocating SCSI page (32768 alloc 32768 max): Out of memory
  YYYY-MM-DD T15:43:34.759Z cpu14:32917)WARNING: FTCpt: 1054: (514736172521 pri) Error cloning SCSI command: Out of memoryYYYY-MM-DD T15:43:34.759Z cpu14:32917)WARNING: FTCpt: 1064: (514736172521 pri) Remote not responding

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSphere 6.x

Cause

This issue occurs because the default disk replication buffer is less for high IO activity for SMP-FT. If the FT replication for the virtual machine cannot keep up with the IO operations, FT will get disabled.

 

Note: Default disk replication buffer is 128 MB.

Resolution

This issue is resolved in ESXi 6.0, Patch Release ESXi600-201611001

 

To work around this issue, increase the disk replication buffer size.

 

To increase the replication buffer size to 2 GB, add the following parameter to the vmx file of the VM by editing it using vi:

ftcpt.maxDiskBufferPages=262144
 

Note: Both the VM and FT need to be turned off to apply this setting

If you are unable to power off the VM you may also be able to apply the new buffer size via PowerCLI using the Set-AdvancedSetting cmdlet e.g.
Modify the buffer size and apply.
Get-AdvancedSetting -Entity (Get-VM -Name $vm-name) -Name "ftcpt.maxDiskBufferPages" | Set-AdvancedSetting -Value $new-buffer-size -Confirm:$false
Add the setting to a VM and apply
Get-VM -Name $vm-name| New-AdvancedSetting -Name "ftcpt.maxDiskBufferPages" -value $new-buffer-size