VM stunned when device was hot removed.

Products

• VMware vSphere ESXi

Issue/Introduction

All hot-add/hot-remove device causes this stun, because before the reconfig it creates a checkpoint, after the hot-add/hot-remove it recovers the checkpoint with the new config.

A disk access error may occur on the GuestOS.
In the vmware.log you will see output similar to:
2023-07-06T09:50:31.302Z Wa(03) vmx - Requesting hot-remove of 'usb'.
2023-07-06T09:50:31.307Z In(05) vcpu-0 - UHCI: HCReset
2023-07-06T09:50:31.310Z No(00) vcpu-0 - ConfigDB: Setting usb.pciSlotNumber = "-1"
2023-07-06T09:50:31.312Z In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=6195962770890788
2023-07-06T09:50:31.312Z In(05) vcpu-0 - VMMon_VSCSIStopVports: No such target on adapter
2023-07-06T09:50:31.312Z No(00) vcpu-0 - ConfigDB: Unsetting all entries with prefix "usb:0."
2023-07-06T09:50:31.315Z In(05) vcpu-0 - USB: Disconnecting device 0x200000040e0f0003
2023-07-06T09:50:31.315Z No(00) vcpu-0 - ConfigDB: Unsetting all entries with prefix "usb:1."
2023-07-06T09:50:31.318Z In(05) vcpu-0 - USB: Disconnecting device 0x10e0f0002
2023-07-06T09:50:31.318Z In(05) vcpu-0 - Closing all the disks of the VM.
2023-07-06T09:50:31.318Z In(05) vcpu-0 - Closing disk 'scsi0:0'
2023-07-06T09:50:31.318Z In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe-flat.vmdk" : closed.
2023-07-06T09:50:31.318Z No(00) vcpu-0 - ConfigDB: Setting usb.present = "FALSE"
2023-07-06T09:50:31.322Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 11223 us
2023-07-06T09:50:31.323Z No(00) vcpu-0 - ConfigDB: Setting scsi0:0.redo = ""
2023-07-06T09:50:31.323Z In(05) vcpu-0 - DISK: OPEN scsi0:0 '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' persistent R[]
2023-07-06T09:50:31.324Z In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe-flat.vmdk" : open successful (10) size = 5368709120, hd = 6334981. Type 3
2023-07-06T09:50:31.324Z In(05) vcpu-0 - DISK: Disk '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' has UUID '60 00 c2 9a 13 03 52 85-a1 96 59 75 55 0a 5c b8'
2023-07-06T09:50:31.324Z In(05) vcpu-0 - DISK: OPEN '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' Geo (652/255/63) BIOS Geo (0/0/0)
2023-07-06T09:50:31.324Z In(05) vcpu-0 - Creating virtual dev for 'scsi0:0'.
2023-07-06T09:50:31.324Z In(05) vcpu-0 - DumpDiskInfo: scsi0:0 createType=11, capacity = 10485760, numLinks = 1
2023-07-06T09:50:31.324Z In(05) vcpu-0 - , allocationType = 1
2023-07-06T09:50:31.324Z In(05) vcpu-0 -
2023-07-06T09:50:31.324Z In(05) vcpu-0 - SCSIDiskESXPopulateVDevDesc: Using FS backend
2023-07-06T09:50:31.324Z In(05) vcpu-0 - DISKUTIL: scsi0:0 : geometry=652/255/63
2023-07-06T09:50:31.324Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC filter attach.
2023-07-06T09:50:31.324Z In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: Invalidating all hashes for this digest
2023-07-06T09:50:31.324Z In(05) vcpu-0 - SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.
2023-07-06T09:50:31.324Z In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 17104896 gosBits = 2 gosType = 1, gosVer = 0, gosMisc = 0
2023-07-06T09:50:31.325Z In(05) vcpu-0 - CPT: vm was stunned for 14130 us
2023-07-06T09:50:31.325Z In(05) vcpu-0 - Powering off usb
2023-07-06T09:50:31.325Z In(05) vcpu-0 - USB: Connecting device desc:name:VMware\ Virtual\ USB\ Mouse vid:0e0f pid:0003 speed:full family:hid deviceType:virtual-hid info:0000004 version:4 id:0x200000040e0f0003
2023-07-06T09:50:31.325Z Wa(03) vcpu-0 - VUsbHID: Failed to connect the virtual HID (1): VMware Virtual USB Mouse
2023-07-06T09:50:31.325Z In(05) vcpu-0 - Hot removal done.

Environment

VMware vSphere ESXi 7.0.2

Resolution

This sub-second VM stun is expected behavior during hot add/remove of devices.