Error: "vim.fault.GenericVmConfigFault" when hot-adding PVSCSI controllers on VM

Products

• VMware vSphere ESXi

Issue/Introduction

• When you attempt to hot-add a Paravirtual SCSI (PVSCSI) controller to a virtual machine (VM) running on VMware ESXi 8.0 Update 3, the operation fails with the error vim.fault.GenericVmConfigFault
• In ESXi /vmfs/volumes/Datastore/<VM Name>/vmware.log for the affected VM, you see the following entries:

[YYYY-MM-DDTHH:MM:SS]Er(02) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a FSR operation is not yet supported on a system with Software Memory Tiering enabled.
[YYYY-MM-DDTHH:MM:SS]In(05) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a HotPlug: Unable to initiate hot-plug operation...
[YYYY-MM-DDTHH:MM:SS]In(05) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a Devices: Migration failed.  Aborting hot-plug.
[YYYY-MM-DDTHH:MM:SS]In(05) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a HotPlug: Hot-add of scsi# failed.
[YYYY-MM-DDTHH:MM:SS]No(00) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a ConfigDB: Setting scsi#.pciSlotNumber = "-1"
[YYYY-MM-DDTHH:MM:SS]No(00) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a ConfigDB: Setting scsi#.present = "FALSE"
[YYYY-MM-DDTHH:MM:SS]In(05) vmx mm3yew38-########-auto-3m1ui-h5:########-ef-01-85-b02a HotPlug: Hot-add of scsi# failed.

• In ESXi /var/run/log/hostd.log, you see the following entries:

[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2099599]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/Datastore/<VM Name>/<VM Name>.vmx] Commit Vigor hot-plug batch 3553545624655 failed: vim.fault.GenericVmConfigFault
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2099599]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/Datastore/<VM Name>/<VM Name>.vmx] Commit Vigor hot-plug batch 3553545624655 message: The hot-plug operation failed.
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2099570]: --> Hot-add of 'scsi#, scsi#' failed.
[YYYY-MM-DDTHH:MM:SS] Db(167) Hostd[2099570]: --> Hot plug operation failed to start.

Environment

VMware vSphere 8.x

Cause

In vSphere 8.0 Update 3, hot-adding a controller or device triggers a Fast Suspend and Resume (FSR) operation to perform a live hardware reconfiguration of the VM. However, Software Memory Tiering (Memory Tiering over NVMe) does not currently support the FSR mechanism in vSphere 8.x. When the VMX process attempts the hot-plug operation, it detects this incompatibility and aborts the request.

Resolution

Note: Support for FSR based operations on systems with Software Memory Tiering enabled is introduced in vSphere 9.0.

Option 1: Disable Memory Tiering per Virtual Machine

1. Power off the target virtual machine.
2. Log in to the vSphere Client and navigate to the VM.
3. Right-click the VM and select Edit Settings.
4. Click the VM Options tab and expand Advanced.
5. Click Edit Configuration next to Configuration Parameters.
6. Click Add Configuration Params and enter the following:
   • Name: sched.mem.enableTiering

   • Value: FALSE
7. Click OK to save the configuration.
8. Power on the VM. You can now execute the Vigor hot-add sequence successfully.

 

Option 2: Disable Memory Tiering via vSphere Client (Host Level)

1. Select the affected ESXi host in the vSphere Client inventory.
2. Place the host into Maintenance mode.
3. Go to the Configure tab.
4. Under System, select Advanced System Settings.
5. Click Edit and search for memoryTiering.
6. Locate VMkernel.Boot.memoryTiering and change the value to FALSE.
7. Click OK and reboot the ESXi host.

Additional Information

For more information on configuring memory tiering, see Memory Tiering Configuration.