Thin-provisioned storage devices met soft threshold

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms

• The thin-provisioned storage devices meet their soft threshold.
• You receive a warning with an indication that the LUN reached a thin provisioning soft threshold.
• In the /var/log/vobd.log file of the ESXi host, you see entries similar to:
  
  Space utilization on the thin-provisioned device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx exceeded the configured threshold.

Cause

This issue occurs when you are using the thin-provisioned device and the storage array is running out of physical storage.
 

Impact

The system logs capture the threshold information. ESXi retries the SCSI command that triggered the response. The storage devices might run out of physical space which could lead the host to be not responding and disconnected.

At a specific threshold, the LUN will go into a READ ONLY state (read-only) and all VMs hosted on this LUN will be impacted. For example:

Some thinly-provisioned storage systems allow an administrator to set a soft limit or threshold that indicates storage backup for the thinly-provisioned device is limited The message indicates the device informed ESXi that the device reached its threshold.

Environment

 

VMware vSphere ESXi 8.0

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Resolution

To resolve this issue, check other storage devices for physical storage space, and add more physical storage if needed. Consult with your storage vendor if you are not familiar with the Storage Management Interface.

Additional Information

In the /var/log/vmkernel.log file of the ESXi host, you see entries similar to:
 
2019-02-01T03:05:56.091Z cpu59:33773)NMP: nmp_ThrottleLogForDevice:3333: Cmd 0x8a (0x43be1e8451c0, 81679675) to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" on path "vmhba65:C0:T0:L22" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x6 0x38 0x7. Act:NONE 
2019-02-01T03:05:56.091Z cpu59:33773)WARNING: ScsiDeviceIO: 2420: Space utilization on thin-provisioned device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx exceeded configured threshold 

This is not a storage connectivity issue but a notification from the storage. For additional information, see the HPE 3PAR VMware ESX/ESXi Implementation Guide.

Notes:

• When the warning limit is reached, the array sends a soft threshold error asc/q: 0x38/0x7 and the ESXi host continues to write.
• When the array runs out of disk space, a hard permanent error asc/q: 0x27/0x7 is sent.