VMkernel logs report the error: Replay of journal on vol 'DATASTORE_NAME' failed I/O error

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Various operations, such as Power operations, Edit Settings, and Migrate, on virtual machines are greyed out

• You are unable to register the virtual machine because the option to register the VMX file is greyed out

• There is no current process involving the virtual machine

• The virtual machine is not accessed by any other host or third-party product

• ESXi is unable to acquire a lock on a volume or file
   
• On reboot, a virtual machine connected to a vDS switch is unable to gain network due to issues locking the .dvsData folder.
   
• In the /var/log/vmkernel.log file on the ESXi host, you see similar entries such as:
  
  Apr 25 15:16:17 vmkernel: 0:01:18:01.347 cpu1:20191)FS3: 6109: Marking HB [HB state abcdef04 offset 3461120 gen 191 stamp 4589142916 uuid 5358e060-bbe2####-####-########64a jrnl <FB 18200> drv 8.46] on vol 'DATASTORE_NAME'

Apr 25 15:16:17 vmkernel: 0:01:18:01.355 cpu1:20191)J3: 2648: Replaying journal at <FB 18200>, gen 191

Apr 25 15:16:19 vmkernel: 0:01:18:04.119 cpu1:20191)WARNING: FS3: 6224: Replay of journal <FB 18200> on vol 'DATASTORE_NAME' failed I/O error

Environment

VMware vSphere ESXi 6.x, 7.x, 8.x

Cause

The issue occurs because the FS3 journal cannot be replayed due to an I/O error.

 

In this case, accessing the File Block 18200 fails while replaying the journal and the VMFS heartbeat at offset 3461120 cannot be updated. As a result, ESXi cannot acquire a lock on a file.

 

Resolution

To resolve this issue, you must investigate the cause of the I/O error.

To find additional information regarding the I/O error, investigate the VMkernel logs.

 

In the sample log in the Symptoms section, you see entries similar to:

 

Apr 25 15:16:19 vmkernel: 0:01:18:04.119 cpu0:4100)ScsiDeviceIO: 1688: Command 0x28 to device "naa.###############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0.

 

In this case, the failure occurred because of a SCSI read failure (Command 0x28) on the device due to hardware error:

MEDIUM ERROR / UNRECOVERED READ ERROR

 

For more information, see Interpreting SCSI sense codes in VMware ESXi.

Additional Information