Virtual machine hangs and issues ESXi host based VSCSI reset

Products

• VMware vSphere ESXi

Issue/Introduction

In virtual machines with disk size larger than 256GB on VMFS5 and larger than 2.5TB on VMFS6, the following symptoms may appear:

• Host becomes unresponsive
• Unable to restart specific hung virtual machines
• Host requires a reboot to restart the virtual machines
• ESXi - /var/log/vmkernel.log shows:
  
  ..... cpu46:712802)WARNING: VSCSI: 3488: handle 19192(vscsi0:1):WaitForCIF: Issuing reset; number of CIF:1
..... cpu38:65988)VSCSI: 2891: handle 19192(vscsi0:1):Reset [Retries: 0/0] from (vmm0:wp-02)
..... cpu46:712802)WARNING: VSCSI: 3488: handle 19192(vscsi0:1):WaitForCIF: Issuing reset; number of CIF:1
..... cpu51:712223)PVSCSI: 2642: scsi0:0: ABORT ctx=0xed
..... cpu36:173183)VSCSI: 2611: handle 19086(vscsi0:0):Reset request on FSS handle 9173930292 (0 outstanding commands) from (vmm0:wp-02)
..... cpu61:65988)VSCSI: 2679: handle 19086(vscsi0:0):Completing reset (0 outstanding commands)
..... cpu13:66610)ScsiDeviceIO: 2948: Cmd(0x439690e250c0) 0x85, CmdSN 0x3022 from world 68172 to dev "naa.61866xxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
..... cpu57:173183)PVSCSI: 2642: scsi0:0: ABORT ctx=0xee
..... cpu57:173183)VSCSI: 2611: handle 19086(vscsi0:0):Reset request on FSS handle 9173930292 (0 outstanding commands) from (vmm0:wp-02)
..... cpu37:1100750)WARNING: VSCSI: 3497: handle 8263(vscsi0:0):WaitForCIF: Issuing reset; number of CIF:33
..... cpu37:1100750)WARNING: VSCSI: 2645: handle 8263(vscsi0:0):Ignoring double reset
..... cpu5:65891)VSCSI: 2891: handle 8263(vscsi0:0):Reset [Retries: 1/0] from (vmm0:vmOS_7.3)
..... cpu5:65891)VSCSI: 2891: handle 8263(vscsi0:0):Reset [Retries: 2/0] from (vmm0:vmOS_7.3)
..... cpu5:65891)VSCSI: 2891: handle 8263(vscsi0:0):Reset [Retries: 3/0] from (vmm0:vmOS_7.3)
..... cpu5:65891)VSCSI: 2891: handle 8263(vscsi0:0):Reset [Retries: 4/0] from (vmm0:vmOS_7.3)

Environment

VMware vSphere ESXi 6.5

Cause

Virtual machine IO requires accessing VMFS metadata associated with the virtual disks. Write IO may require updating of this metadata due to allocation of new data blocks.

 

There is a synchronization issue while accessing this file block metadata from multiple threads causing a deadlock between threads.

Resolution

This issue is resolved in VMware ESXi650-201712001.