SMP-FT turns off during high I/O operations for a VM

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• During high I/O operations for a virtual machine with SMP-FT enabled, you see Fault Tolerance is turned off.
• In the /var/run/log/vmkernel.log, you see the entries similar to:
  
  

  YYYY-MM-DD THH:MM:SS.759Z cpu14:32917)WARNING: FTCpt: 1024: (5147###72521 pri) Error allocating SCSI page (32768 alloc 32768 max): Out of memory
  YYYY-MM-DD THH:MM:SS.759Z cpu14:32917)WARNING: FTCpt: 1054: (5147###72521 pri) Error cloning SCSI command: Out of memoryYYYY-MM-DD T15:43:34.759Z cpu14:32917)WARNING: FTCpt: 1064: (5147###72521 pri) Remote not responding

  
  

  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

• VMware vSphere 6.x
• VMware vSphere 7.x
• VMware vSphere 8.x

Cause

This issue occurs because the default disk replication buffer is less for high IO activity for SMP-FT. If the FT replication for the virtual machine cannot keep up with the IO operations, FT will get disabled.

 

Note: Default disk replication buffer is 128 MB.

Resolution

 

To work around this issue, increase the disk replication buffer size.

 

To increase the replication buffer size to 2 GB, add the following parameter to the vmx file of the VM by editing it using vi:

ftcpt.maxDiskBufferPages=262144

 

Note: Both the VM and FT need to be turned off to apply this setting

Steps to add parameter in vmx file:

1. Power off the virtual machine
2. SSH to ESXi host using root credentials
3. cd to VM directory
4. Edit the vmx file using vi editor (vi VM_name.vmx)
5. Press Insert button (i)
6. Add below line:
   

   ftcpt.maxDiskBufferPages=262144

7. Press ESC and type wq! to save and exit the file.
   
   

If you are unable to power off the VM you may also be able to apply the new buffer size via PowerCLI using the Set-AdvancedSetting cmdlet e.g.
Modify the buffer size and apply.

$vm-name = <Provide VM name>
$new-buffer-size = 262144

Get-AdvancedSetting -Entity (Get-VM -Name $vm-name) -Name "ftcpt.maxDiskBufferPages" 

Add required setting to a VM and apply

Get-VM -Name $vm-name| New-AdvancedSetting -Name "ftcpt.maxDiskBufferPages" -value $new-buffer-size