Virtual machines using EFI firmware fails to start from certain storage devices

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When powering on a Windows virtual machine using EFI from an Dell EMC PowerStore and EMC XtremIO storage arrays, you see these events:

• Virtual machine fails to start, reports an I/O error or an error indicating that boot files are missing or corrupt.
• In the /var/log/vmkernel.log file, you see this error indicating disk I/O failures on the storage array:
  
  Cmd(0x#####) 0x##, CmdSN 0x##### from world ##### to dev "####" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0
   
• Running the esxcli system settings advanced list -o "/Disk/DiskMaxIOSize" command on the host containing the virtual machine reports the default disk IO block size:
  
  Path: /Disk/DiskMaxIOSize
Type: integer
Int Value: 32767
Default Int Value: 32767
Min Value: 32
Max Value: 32767
String Value:
Default String Value:
Valid Characters:
Description: Max Disk READ/WRITE I/O size before splitting (in KB)
   
• When running the esxcli storage core path list command, you see that the storage array reports a default IO block size not equal to 32767.
• You may also encounter an issue where after installing some vibs in a boot from SAN environment  ESXi fails to  with message 

       Loading /sb.v00
      Error loading /sb.v00
      Compressed MD%: beb813d57fd3215416e8b87a4db2e4a2
      Decompressed MD5: 00000000000000000000000000000000
      Fatal erorr: 6 (Buffer too small)

• You may also encounter an issue when snapshot is attempted on a VM it fails with message "An error occurred while saving: 38 (Function not implemented)"

Environment

VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5

Cause

The default DiskMaxIOSize in ESXi is 32767 (32 MB). Even if the storage array reports a different block size to the host, ESXi continues to use the default DiskMaxIOSize of 32 MB. Because of this, block transfers between the virtual machine and the storage array may be interrupted or incomplete.The result is that the virtual machine may fail to boot using EFI.

Resolution

This issue is resolved in VMware ESXi 6.0, Patch Release ESXi600-201909001, VMware ESXi 6.5, Patch Release ESXi650-201811002 and VMware ESXi 6.7 Update 1 Release.

Refer also to release notes at VMware Release Notes

To download patches go to Broadcom Support

 

Workaround:

To work around this issue, you must set the ESXi DiskMaxIOSize parameter on each host that sees the storage array's LUNs to the IO block size value advertised by the array.

Note: With Dell EMC PowerStore storage, this parameter should be set to 1 MB . With XtremIO storage, this parameter should be set to 4 MB.

To set the ESXi DiskMaxIOSize parameter:

1. Put the host in the Maintenance Mode.
2. Set the ESXi DiskMaxIOSize parameter to 1 MB (for PowerStore) and 4 MB(for XtremIO) by running this command:

 

For PowerStore:

esxcli system settings advanced set -o "/Disk/DiskMaxIOSize" --int-value 1024
 

For XtremIO:

esxcli system settings advanced set -o "/Disk/DiskMaxIOSize" --int-value 4096

3. Run this command to verify if the parameter is 1 MB or 4 MB:
   
   esxcli system settings advanced list -o "/Disk/DiskMaxIOSize"
   
   You see output similar to:

   Path: /Disk/DiskMaxIOSize
Type: integer
Int Value: 1024 (for PowerStore) or 4096(for XtremIO)
Default Int Value: 32767
Min Value: 32
Max Value: 32767
String Value:
Default String Value:
Valid Characters:
Description: Max Disk READ/WRITE I/O size before splitting (in KB)

Alternatively, you can reduce the size of the IO requests passed to the storage device using the VMware vSphere Client program:

1. Navigate to Host > Configuration.
2. Click Advanced Settings.
3. Navigate to Disk.
4. Change the Disk.DiskMaxIOSize value to 1024 (for PowerStore) or 4096(for XtremIO).

Note: You can change this setting without rebooting the host or putting it in the maintenance mode.

 

Additional Information

Impact/Risks:
If there are other devices on the host which are capable of servicing large IOs, the IOs to these devices will also be split as per DiskMaxIOSize setting and can  affect all VMs on the ESXi host and possibly affect the performance