The vmkernel logs in ESXi hosts report the error: Replay of journal on vol '' failed: No space left on device in vmkernel logs

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When you browse the datastore to view existing files, you do not see any files in the datastore
• Virtual machines on the affected datastore fail to power on
• When you try to access the virtual machine's folder on the datastore from the command line, you see the error:
  
  /vmfs/volumes/4f563d2f-1e72ecef-5dcc-002481eddf96 # cd VM/
  sh: getcwd: Device or resource busy
  (unknown) #
  
  
• Running the pwd command also reports the same error:
  
  (unknown) # pwd
  /vmfs/volumes/datastore/VM
  sh: getcwd: Device or resource busy
  
  
• You see this error in vmkernel logs:
  
  naa.600601603d402800b0fede8df0fbe111" failed H:0x0 D:0x2 P:0x8 Possible sense data: 0x7 0x27 0x7.
  cpu12:5630)J3: 3497: Replaying transaction failed: No space left on device
  cpu12:5630)J3: 3510: checksum 12169bb972298899, length 7168, CID 0xc1d00001, hbGen 9, ser# 1467629
  cpu12:5630)J3: 3512: 3 lockActions, first at 48
  cpu12:5630)J3: 3514: 6 logActions, first at 848
  cpu12:5630)J3: 3520: Locks
  cpu12:5630)J3: 3524: #0: 189034496 v 1608
  cpu12:5630)J3: 3524: #1: 174558208 v 640
  cpu12:5630)J3: 3524: #2: 552684544 v 532
  cpu12:5630)J3: 3526: Actions
  cpu12:5630)J3: 3530: #0: type 1 reqLk 1, fr 0 to 174558720, len 512
  cpu12:5630)J3: 3530: #1: type 1 reqLk 2, fr 0 to 552685056, len 512
  cpu12:5630)J3: 3530: #2: type 1 reqLk 0, fr 0 to 563101696, len 512
  cpu12:5630)J3: 3530: #3: type 1 reqLk 0, fr 0 to 189274112, len 2048
  cpu12:5630)J3: 3530: #4: type 1 reqLk 0, fr 0 to 562738176, len 512
  cpu12:5630)J3: 3530: #5: type 1 reqLk 0, fr 0 to 189035008, len 1536
  cpu12:5630)WARNING: J3: 3207: Replaying slot 0 for journal at <FB 257800>: No space left on device
  cpu12:5630)WARNING: HBX: 4336: Replay of journal <FB 257800> on vol 'SAN-Datastore5' failed: No space left on device
  

Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable

Cause

The VMFS filesystem fails to replay the journal (read/write) on the affected volume as the volume would be full and no free space is left on the physical storage.

For example, if the storage admin presents LUN from a thin pool of storage, and if there is no free space left on this thin pool capacity disk, the virtual machines fail to read/write from this thin pool and the vmkernel logs display the error detailed in the Symptoms field.

Resolution

Make sure that there is sufficient amount of free space on the storage device. To verify this information, involve your SAN vendor.

If there is no free space on the device, increase the amount of space on the storage device, unpresent and then represent the affected LUN, and rescan for datastore on ESXi.

Additional Information

Impact/Risks:
This scenario may corrupt the virtual machine files (if snapshots exist on the virtual machine). You should ensure there is a sufficient amount of space on the thin pool of storage.