VMs freeze due to out of space issue with thin-provisioned LUNs

Products

• VMware vCenter Server

Issue/Introduction

Symptoms : 

• VMs freeze/ VMs are down due to out of space issue with thin-provisioned LUNs.
• Unable to remove any files to free up space on the datastore

Validation Steps: 

• Login to SSH session for the host and run the command "df -h" to verify the storage usage.
  
  # df -h
Filesystem     Size      Used       Available     Used%     Mounted on
VMFS-6        100.T      9.7T       258.4G         97%       /vmfs/volumes/<datastorename>

• Removing unwanted files fails with error : "Failed - Error caused by file /vmfs/volumes/<Datastore UUID>/<VM NAME>/vmdk"
  
  

Note: Kindly ensure that the files are no longer required before proceeding with their removal. Additionally, please confirm that there is a restorable backup available to avoid any impact. 

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The issue is caused by the LUN backing the datastore being thinly provisioned at the array level. The pool associated with this LUN is out of space, causing the datastore to enter read-only mode.

Cause validation:

• The /var/run/log/vobd.log file indicates that Volume will remain in read-only metadata mode with limited write support until journal can be created.
  
  
  YYYY-MM-DDTHH:MM.SSSZ : [vmfsCorrelator] 90436842us: [vob.vmfs.journal.createfailed] No space for the journal on volume yyyyyyyy-yyyyyyyy-yyyy-yyyyyyyyyyyy ("Datastore name"). Volume will remain in read-only metadata mode with limited write support until journal can be created.
  YYYY-MM-DDTHH:MM.SSSZ : [vmfsCorrelator] 172749660us: [vob.vmfs.journal.createfailed] No space for the journal on volume yyyyyyyy-yyyyyyyy-yyyy-yyyyyyyyyyyy ("Datastore name"). Volume will remain in read-only metadata mode with limited write support until journal can be created.

• The /var/run/log/vmkernel.log file indicates below mentioner errors.
  
  YYYY-MM-DDTHH:MM.SSSZ cpu6:2097875)ScsiDeviceIO: 4167: Cmd(0x45d901400ac8) 0x2a, CmdSN 0x566e from world 0 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x8 Valid sense data: 0x7 0x27 0x7
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097875)ScsiDeviceIO: 4167: Cmd(0x45d9014b6dc8) 0x2a, CmdSN 0x5673 from world 0 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x8 Valid sense data: 0x7 0x27 0x7
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097875)ScsiDeviceIO: 4167: Cmd(0x45d916b14ac8) 0x2a, CmdSN 0x5679 from world 0 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x8 Valid sense data: 0x7 0x27 0x7
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097875)ScsiDeviceIO: 4167: Cmd(0x45d915ad57c8) 0x2a, CmdSN 0x567e from world 0 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x8 Valid sense data: 0x7 0x27 0x7
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097875)ScsiDeviceIO: 4167: Cmd(0x45d915adb6c8) 0x2a, CmdSN 0x5683 from world 0 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x8 Valid sense data: 0x7 0x27 0x7

SCSI sense codes reported :
P:0x8 :        Backing pool for thin provisioned LUN is out of space.
0x7 :           Data Protect
0x27/0x7 : Space Allocation Failed Write Protect
  
  Refer VMware ESXi SCSI Sense Code Decoder

Resolution

The backing pool for the thin-provisioned LUN is out of space. Please engage the storage vendor to assist with either releasing existing space or increasing the storage pool size.