Virtual machines freeze intermittently or goes unresponsive under heavy I/O load

Products

• VMware vSphere ESXi

Issue/Introduction

• Running the "ps -s | grep vm-name" command on the ESXi host running the affected virtual machine returns similar to:

4313969 vmm0: vm-name COSTOP NONE 0-63
4313971 vmm1:vm-name WAIT SCSI 0-63
4313972 4313957 vmx-vthread-5:vm-name WAIT UFUTEX 0-63 /bin/vmx
4314204 4313957 vmx-vthread-6:vm-name WAIT UFUTEX 0-63 /bin/vmx
4314205 4313957 vmx-vthread-7:vm-name WAIT UFUTEX 0-63 /bin/vmx
4314206 4313957 vmx-vthread-8:vm-name WAIT UFUTEX 0-63 /bin/vmx
4314210 4313957 vmx-mks:vm-name WAIT UPOL 0-63 /bin/vmx
4314212 4313957 vmx-svga:vm-name WAIT SEMA 0-63 /bin/vmx
4314214 4313957 vmx-vcpu-0:vm-name COSTOP NONE 0-63 /bin/vmx
4314215 4313957 vmx-vcpu-1:vm-name WAIT SCSI 0-63 /bin/vmx

Note: The vmm1 is blocked on a SCSI call.

• The below error may appear:

Unable to connect to the MKS: Error connecting to /bin/vmx process.

• Virtual machines are unreachable over the network
• Virtual machines may report an invalid state
• Virtual machines are unresponsive

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

A virtual machine can be unresponsive when:

• Taking quiesced snapshots or using a custom quiescing script.
• A heavy I/O load on the ESXi hosts
• Storage performance issues  at the device, storage pool and/or LUN level.
• One of the Virtual Machine Monitor (VMM) thread is blocked on a VSCSI call, the other VMM threads are co-stopped, waiting for the blocked thread to make progress.

Resolution

Caution: Ensure that there are no snapshot consolidation tasks or backups are running on the VMs during this time. 

 

Workaround

To recover the virtual machine from its locked-up state:

1. Run this command to find the process list for the virtual machine and check the cartel ID:

ps -s | grep vm-name

 

Note: Refer to the ps -s output mentioned in the cause section of this Knowledge Base article.

2. Find the vmx-vcpu value that is waiting on SCSI event.

Note: The number in the second column of the output is the cartel ID.

3. Run "kill -18 <cartel-ID>" (SIGCONT) signal to the cartel to continue the process that has stopped.
   
   
4. After running the above steps the virtual machine may need to be reloaded. For more information see Reloading a vmx file without removing the virtual machine from inventory (broadcom.com).

Notes:

• Above mentioned steps is a workaround to recover VM from locked-up state.
• For more information on SIGCONT, see Sending signal to Processes.

Additional Information

• Troubleshooting hosted disk I/O performance problems (broadcom.com)
• Investigating hosted virtual machine resources (broadcom.com)
• Check the Virtual Machine Compatibility in the VMware Host Client
• Configuring disks to use VMware Paravirtual SCSI (PVSCSI) controllers (broadcom.com)
• Opening a console to a virtual machine fails with the error: Unable to connect to the MKS: Error connecting to /bin/vmx process. (broadcom.com)
• Reloading a vmx file without removing the virtual machine from inventory (broadcom.com)