Error when accessing VMFS volume: WARNING: J3: 1644: Error freeing journal block

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When accessing VMFS volumes, the ESX /var/log/vmkernel file or ESXi /var/log/messages file contain an entry similar to:
  
  WARNING: J3: 1644: Error freeing journal block (returned 0) <FB 731582> for 4a5c8acd-53af0650-4f86-002481a5c7e4: Invalid metadata
  
  
• You may experience filesystem corruption.
• Virtual machines simultaneously fail on multiple hosts.

Environment

VMware ESXi 4.0.x Installable
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable

Cause

This is a known issue which affects versions of ESX 4.0 prior to Update 2.

Note: This article refers to having the error WARNING: J3: 1644: Error freeing journal block with Invalid metadata. It is possible to receive the error WARNING: J3: 1644: Error freeing journal block, but it is referring to another issue such as Lock was not free.

Resolution

This issue is resolved in ESX 4.0 Update 2. You can download ESX 4.0 Update 2 from the vSphere Download Center.

Prior to ESX 4.0 Update 2, this issue occurs when different ESX hosts have NICs with the same MAC address. The MAC address of one NIC is used to construct the System ID of the host. Hosts use this ID to identify what is performing operations on the metadata of the datastore and therefore should be unique.

To resolve this issue, ensure that the System UUID of every host is unique and that each NIC has a unique MAC address.

To determine the System UUID, run this command from the service console of each host:

# esxcfg-info -y | grep -i "System UUID"

The output appears similar to:

|----System UUID.................................................4b27ad9b-2120-4013-99ac- 00237da6fc54

In this example, the MAC address is 00237da6fc54.

To determine the physical MAC addresses of the ESX hosts accessing the volume, run the following command from the service console for ESX or the CLI for ESXi and compare the output to verify that each NIC has a unique MAC address:

esxcfg-nics -l

For example:

# esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 42:00.00 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic1 44:00.00 bnx2 Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Broadcom Corporation Broadcom NetXtreme II BCM5708 1000Base-T
vmnic2 4b:00.00 e1000e Up 1000Mbps Full xx:xx:xx:xx:xx:xx 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic3 4b:00.01 e1000e Down 0Mbps Half xx:xx:xx:xx:xx:xx 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

In this example, the MAC address is xx:xx:xx:xx:xx:xx.

To resolve this issue, eliminate the NIC that has the same MAC address or reconfigure the hosts so they do not share NICs.

Additional Information

For ESX 3.x, use this command to find the MAC address:

# ifconfig | grep -i HWaddr
inet addr:xx.xx.xx.xx Mask:255.0.0.0
vmnic0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic1 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic2 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vmnic3 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
vswif0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx