ESX VMkernel log and ESXi messages log contain the messages: WRITE10/READ 10 past end of virtual device

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

The /var/log/vmkernel log on ESX or /var/log/messages log on ESXi contains entries similar to:

• Feb 5 15:44:46 <hostname> vmkernel: 63:05:31:58.181 cpu3:1129)WARNING: VSCSI: 5292: WRITE10 past end of virtual device with 33554432 numBlocks, offset 33554351, length 128
• Feb 12 17:03:04 <hostname> vmkernel: 156:05:50:47.889 cpu0:1174)WARNING: VSCSI: 3430: READ10 past end of virtual device with 20971520 numBlocks, offset 20980737, length 16

Environment

VMware ESXi 3.5.x Embedded
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.1
VMware ESX 4.1.x
VMware ESX Server 3.0.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESX 4.0.x
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Installable
VMware ESXi 3.5.x Installable
VMware ESXi 4.1.x Installable

Cause

These messages indicate that I/O outside the boundaries of the virtual disk is being attempted. Specifically, this issue may occur if the end value in the partition table exceeds the number of cylinders set in the descriptor file. The messages do not indicate an error, but if data is written beyond the limits of the disk, it is possible that you may experience data loss.

Resolution

While this issue is not resolved in VMware ESXi 5.1, Patch ESXi510-201304201-UG: Updates VMware ESXi 5.1 esx-base vib (2041612) it does identify which virtual machine is causing the error by writing the virtual machine name in the error log.

The virtual machine causing the issue is noted in the error message in the /var/log/vmkernel log.

To find out the virtual machine is causing error messages in older builds, generate the log bundle from the ESXi/ESX host on the virtual machine is running. Then contact VMware Technical Support.

Once the virtual machine causing the messages found out by VMware Technical Support, verify the source of the error:

1. Open the descriptor file for the virtual machine that is triggering the messages.
   
2. Look for the entry that lists the number of cylinders for the virtual disk. In this example, the virtual disk has 5221 cylinders:
   
   ddb.geometry.cylinders = "5221"
   
   
3. Run this command against the flat file of the virtual disk:
   
   fdisk -l
   
   Note: To ensure that the -flat file is not locked, only run this command on a virtual machine that is powered off or has at least one snapshot.
   
   The output appears similar to:
   
   Disk ANSGOOD-flat.vmdk: 0 MB, 0 bytes
   255 heads, 63 sectors/track, 0 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
   
   Device Boot Start End Blocks Id System
   ANSGOOD-flat.vmdk1 * 1 5222 41940992 7 HPFS/NTFS
   Partition 1 has different physical/logical endings:
   phys=(1023, 254, 63) logical=(5221, 137, 36)
   
   In this example, the end value for the partition table is 5222. This value exceeds the number of cylinders set in the descriptor file (in this case, 5221).

To resolve this issue, extend the disk with the vmkfstools -X command to, or beyond, the discovered boundary from above. For more information, see Adding space to an ESX host virtual disk (994).

Alternatively if there has been no perceivable data loss or problems in the guest operating system, you may opt to create a new virtual disk of the intended size, attach it to the virtual machine, and migrate data to the new disk manually within the Guest operating system.

This issue commonly occurs with Windows 7 and Windows 2008 virtual machines. If you run the command msinfo32., and expand Components > Storage and highlight Disks, you can see results similar to this for the same virtual machine:

Size: 39.99 Gb (42,944,186,880 bytes)

Total Cylinders: 5,221

Total Sectors: 83,875,365

Total Tracks: 1,331,355

Tracks/Cylinder: 255

Partition: Disk #0, Partition #0

Partition Size: 40.00 Gb (42,947,575,808 bytes)

Partition Starting Offset: 1,048,576 bytes

You can work out the over allocation by adding the Partition Starting Offset to the Partition Size and subtracting this from the Size. For example:

42,944,186,880 - (1,048,576+42,947,575,808) = -4,437,504 bytes

Additional Information

VMware ESXi 5.1, Patch ESXi510-201304201-UG: Updates VMware ESXi 5.1 esx-base vib

Impact/Risks:
Small possibility of data loss, if data is written beyond the limits of the disk.