VMkernel logs report the error: Replay of journal on vol 'DATASTORE_NAME' failed I/O error

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Various operations, such as Power operations, Edit Settings, and Migrate, on virtual machines are greyed out

• You are unable to register the virtual machine because the option to register the VMX file is greyed out

• There is no current process involving the virtual machine

• The virtual machine is not accessed by any other host or third-party product

• ESXi is unable to acquire a lock on a volume or file
   
• On reboot, a virtual machine connected to a vDS switch is unable to gain network due to issues locking the .dvsData folder.
   
• In the /var/log/vmkernel.log file on the ESXi host, you see similar entries such as:
  
  Apr 25 15:16:17 vmkernel: 0:01:18:01.347 cpu1:20191)FS3: 6109: Marking HB [HB state abcdef04 offset 3461120 gen 191 stamp 4589142916 uuid 5358e060-bbe27e1b-e193-3440b589864a jrnl <FB 18200> drv 8.46] on vol 'DATASTORE_NAME'

Apr 25 15:16:17 vmkernel: 0:01:18:01.355 cpu1:20191)J3: 2648: Replaying journal at <FB 18200>, gen 191

Apr 25 15:16:19 vmkernel: 0:01:18:04.119 cpu1:20191)WARNING: FS3: 6224: Replay of journal <FB 18200> on vol 'DATASTORE_NAME' failed I/O error

 

 

 

Environment

VMware vSphere ESXi 5.0
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Installable
VMware vSphere ESXi 6.0

Cause

The issue occurs because the FS3 journal cannot be replayed due to an I/O error.

 

In this case, accessing the File Block 18200 fails while replaying the journal and the VMFS heartbeat at offset 3461120 cannot be updated. As a result, ESXi cannot acquire a lock on a file.

 

Resolution

To resolve this issue, you must investigate the cause of the I/O error.

To find additional information regarding the I/O error, investigate the VMkernel logs.

 

In the sample log in the Symptoms section, you see entries similar to:

 

Apr 25 15:16:19 vmkernel: 0:01:18:04.119 cpu0:4100)ScsiDeviceIO: 1688: Command 0x28 to device "naa.600508e0000000001bb2016e9948700a" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0.

 

In this case, the failure occurred because of a SCSI read failure (Command 0x28) on the device due to hardware error:

MEDIUM ERROR / UNRECOVERED READ ERROR

 

For more information, see Interpreting SCSI sense codes in VMware ESXi and ESX (289902).

Additional Information

VMkernel ログに次のエラーが記録される： Replay of journal on vol 'DATASTORE_NAME' failed I/O error
VMkernel 日志报告以下错误：Replay of journal on vol 'DATASTORE_NAME' failed I/O error