The vmkernel logs in ESXi hosts report the error: Replay of journal on vol '' failed: No space left on device in vmkernel logs
search cancel

The vmkernel logs in ESXi hosts report the error: Replay of journal on vol '' failed: No space left on device in vmkernel logs

book

Article ID: 311556

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • When you browse the datastore to view existing files, you do not see any files in the datastore
  • Virtual machines on the affected datastore fail to power on
  • When you try to access the virtual machine's folder on the datastore from the command line, you see the error:

    /vmfs/volumes/4f563d2f-1e72ecef-5dcc-002481eddf96 # cd VM/
    sh: getcwd: Device or resource busy
    (unknown) #


  • Running the pwd command also reports the same error:

    (unknown) # pwd
    /vmfs/volumes/datastore/VM
    sh: getcwd: Device or resource busy


  • You see this error in vmkernel logs:

    naa.600601603d402800b0fede8df0fbe111" failed H:0x0 D:0x2 P:0x8 Possible sense data: 0x7 0x27 0x7.
    cpu12:5630)J3: 3497: Replaying transaction failed: No space left on device
    cpu12:5630)J3: 3510: checksum 12169bb972298899, length 7168, CID 0xc1d00001, hbGen 9, ser# 1467629
    cpu12:5630)J3: 3512: 3 lockActions, first at 48
    cpu12:5630)J3: 3514: 6 logActions, first at 848
    cpu12:5630)J3: 3520: Locks
    cpu12:5630)J3: 3524: #0: 189034496 v 1608
    cpu12:5630)J3: 3524: #1: 174558208 v 640
    cpu12:5630)J3: 3524: #2: 552684544 v 532
    cpu12:5630)J3: 3526: Actions
    cpu12:5630)J3: 3530: #0: type 1 reqLk 1, fr 0 to 174558720, len 512
    cpu12:5630)J3: 3530: #1: type 1 reqLk 2, fr 0 to 552685056, len 512
    cpu12:5630)J3: 3530: #2: type 1 reqLk 0, fr 0 to 563101696, len 512
    cpu12:5630)J3: 3530: #3: type 1 reqLk 0, fr 0 to 189274112, len 2048
    cpu12:5630)J3: 3530: #4: type 1 reqLk 0, fr 0 to 562738176, len 512
    cpu12:5630)J3: 3530: #5: type 1 reqLk 0, fr 0 to 189035008, len 1536
    cpu12:5630)WARNING: J3: 3207: Replaying slot 0 for journal at <FB 257800>: No space left on device
    cpu12:5630)WARNING: HBX: 4336: Replay of journal <FB 257800> on vol 'SAN-Datastore5' failed: No space left on device



Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware ESXi 4.0.x Embedded
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable

Cause

The VMFS filesystem fails to replay the journal (read/write) on the affected volume as the volume would be full and no free space is left on the physical storage.

For example, if the storage admin presents LUN from a thin pool of storage, and if there is no free space left on this thin pool capacity disk, the virtual machines fail to read/write from this thin pool and the vmkernel logs display the error detailed in the Symptoms field.

Resolution

Make sure that there is sufficient amount of free space on the storage device. To verify this information, involve your SAN vendor.
If there is no free space on the device, increase the amount of space on the storage device, unpresent and then represent the affected LUN, and rescan for datastore on ESXi.


Additional Information

Impact/Risks:
This scenario may corrupt the virtual machine files (if snapshots exist on the virtual machine). You should ensure there is a sufficient amount of space on the thin pool of storage.