VMware virtual machine is "stunned" during the hot removal of a USB device

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

 

When a VMware virtual machine (VM) is "stunned" during the hot removal of a USB device, it refers to a brief pause or freeze in the VM's execution. This process occurs because VMware must temporarily halt the VM to maintain consistency in its virtual hardware configuration and ensure that the removal is safely processed without causing corruption or disruption to the system.

 

In the vmware.log you will see output similar to:

YYYY-MM-DD HH-MM Wa(03) vmx - Requesting hot-remove of 'usb'.
YYYY-MM-DD HH-MM In(05) vcpu-0 - UHCI: HCReset
YYYY-MM-DD HH-MM No(00) vcpu-0 - ConfigDB: Setting usb.pciSlotNumber = "-1"
YYYY-MM-DD HH-MM In(05) vcpu-0 - Destroying virtual dev for scsi0:0 vscsi=6195962770890788
YYYY-MM-DD HH-MM In(05) vcpu-0 - VMMon_VSCSIStopVports: No such target on adapter
YYYY-MM-DD HH-MM No(00) vcpu-0 - ConfigDB: Unsetting all entries with prefix "usb:0."
YYYY-MM-DD HH-MM In(05) vcpu-0 - USB: Disconnecting device 0x200000040e0f0003
YYYY-MM-DD HH-MM No(00) vcpu-0 - ConfigDB: Unsetting all entries with prefix "usb:1."
YYYY-MM-DD HH-MM In(05) vcpu-0 - USB: Disconnecting device 0x10e0f0002
YYYY-MM-DD HH-MM In(05) vcpu-0 - Closing all the disks of the VM.
YYYY-MM-DD HH-MM In(05) vcpu-0 - Closing disk 'scsi0:0'
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe-flat.vmdk" : closed.
YYYY-MM-DD HH-MM No(00) vcpu-0 - ConfigDB: Setting usb.present = "FALSE"
YYYY-MM-DD HH-MM In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for 11223 us
YYYY-MM-DD HH-MM No(00) vcpu-0 - ConfigDB: Setting scsi0:0.redo = ""
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISK: OPEN scsi0:0 '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' persistent R[]
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe-flat.vmdk" : open successful (10) size = 5368709120, hd = 6334981. Type 3
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISK: Disk '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' has UUID '60 00 c2 9a 13 03 52 85-a1 96 59 75 55 0a 5c b8'
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISK: OPEN '/vmfs/volumes/635a4c9e-e414b8d6-e11d-005056010651/test-gEGe/test-gEGe.vmdk' Geo (652/255/63) BIOS Geo (0/0/0)
YYYY-MM-DD HH-MM In(05) vcpu-0 - Creating virtual dev for 'scsi0:0'.
YYYY-MM-DD HH-MM In(05) vcpu-0 - DumpDiskInfo: scsi0:0 createType=11, capacity = 10485760, numLinks = 1
YYYY-MM-DD HH-MM In(05) vcpu-0 - allocationType = 1
YYYY-MM-DD HH-MM In(05) vcpu-0 - SCSIDiskESXPopulateVDevDesc: Using FS backend
YYYY-MM-DD HH-MM In(05) vcpu-0 - DISKUTIL: scsi0:0 : geometry=652/255/63
YYYY-MM-DD HH-MM In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: CBRC not enabled or opened without filters,skipping CBRC filter attach.
YYYY-MM-DD HH-MM In(05) vcpu-0 - SCSIFilterESXAttachCBRCInt: Invalidating all hashes for this digest
YYYY-MM-DD HH-MM In(05) vcpu-0 - SCSIFilterSBDAttachCBRC: device scsi0:0 is not SBD. Skipping CBRC attach SBD way.
YYYY-MM-DD HH-MM In(05) vcpu-0 - VMXNET3 user: Ethernet0 Driver Info: version = 17104896 gosBits = 2 gosType = 1, gosVer = 0, gosMisc = 0
YYYY-MM-DD HH-MM In(05) vcpu-0 - CPT: vm was stunned for 14130 us
YYYY-MM-DD HH-MM In(05) vcpu-0 - Powering off usb
YYYY-MM-DD HH-MM In(05) vcpu-0 - USB: Connecting device desc:name:VMware\ Virtual\ USB\ Mouse vid:0e0f pid:0003 speed:full family:hid deviceType:virtual-hid info:0000004 version:4 id:0x200000040e0f0003
YYYY-MM-DD HH-MM Wa(03) vcpu-0 - VUsbHID: Failed to connect the virtual HID (1): VMware Virtual USB Mouse
YYYY-MM-DD HH-MM In(05) vcpu-0 - Hot removal done.

Environment

VMware vSphere ESXi 7.x

Cause

Here’s why this happens:

1. Checkpoint Creation: Before making a change to the VM's hardware (like removing a USB device), VMware creates a checkpoint. This snapshot of the VM's current state ensures that if anything goes wrong, it can recover to a stable state.
2. Consistency of Configuration: When the USB device is removed, the VM's hardware configuration is updated. The stun allows VMware to reconcile the changes and ensure that all virtual hardware is accurately represented post-change.
3. Data Integrity: During this brief stun period, VMware ensures that no ongoing operations involving the USB device are disrupted, preventing data corruption or other errors.
4. Resuming the VM: After the USB device is removed and the configuration is updated, the VM is quickly "unstunned," and normal operation resumes with the new hardware configuration.

Resolution

This stun is generally very short (sub-second) but may be more noticeable if there are high I/O operations or intensive workloads on the VM at the time of the hot removal.

Additional Information

While the ESXi host supports hot-removal of a USB controller, the guest operating system must also support this functionality. Refer to the vendor's documentation to confirm if hot-removal is supported by the guest OS. If the guest OS does not support this feature, the virtual machine will need to be powered off to safely remove the USB controller.

For further details, please refer to Steps to remove a USB controller from a VMware ESXi virtual machine.