Various VMs freezing primarily during snapshot consolidations or removals

Products

• VMware vSphere ESXi

Issue/Introduction

Multiple VMs sharing the same backing thin provisioned storage experience freezes, occurring more often during snapshot consolidation. 

Environment

ESXi all versions

Cause

The thin provisioned LUN used as datastore for the VMs has reached quota and further writes are prevented until storage is freed. This can be verified in the vmkernel log with the following messages:

H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0  -->  H=VMK_SCSI_HOST_OK  D=VMK_SCSI_DEVICE_CHECK_CONDITION  P=VMK_SCSI_PLUGIN_GOOD  SK=MISCOMPARE  ASD=MISCOMPARE DURING VERIFY OPERATION
H:0x0 D:0x2 P:0x7 Valid sense data: 0x6 0x38 0x7  -->  H=VMK_SCSI_HOST_OK  D=VMK_SCSI_DEVICE_CHECK_CONDITION  P=VMK_SCSI_PLUGIN_THINPROV_ATQUOTA  SK=UNIT ATTENTION  ASD=THIN PROVISIONING SOFT THRESHOLD REACHED

Resolution

Free space/reduce utilization on the storage backing the datastore. Once available space increases the VMs stop freezing.