ESXi/ESX hosts disconnect from vCenter Server with timeout and reservation errors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi/ESX hosts disconnect from vCenter Server.
• You experience performance issues even when the storage is not busy.
• ESXi/ESX logs report timeouts and reservation errors.
• You see intermittent All-Paths-Down (APD) messages.
• Some of the fibre channel switch ports connected to the ESXi/ESX hosts report large number of errors.
• You are using FC HBA BIOS versions 1.24 or 1.26.
• When you run the cat /proc/scsi/qlaxxx/1 command on an ESX host, you see an output similar to:
  
  Firmware version 4.04.09 [IP] [Multi-ID] [84XX] , Driver version 8.02.01-k1-vmw43
  FCODE version 1.24
  Flash FW version 4.00.26
  ISP: ISP2432
  
  
• The /var/log/messages file on the ESXi 4.1 host contains entries similar to:
  
  vmkernel: 10:08:26:09.601 cpu4:4262)WARNING: NMP: nmp_DeviceUpdatePathStates: Could not drop reservation on failover for NMP device "naa.6006016002801a00648ee133eda2de11".
  vmkernel: 10:08:26:10.600 cpu7:4261)NMP: nmp_DeviceUpdatePathStates: Activated path "NULL" for NMP device "naa.6006016002801a00648ee133eda2de11".
  vmkernel: 10:08:26:10.600 cpu3:4263)NMP: nmp_DeviceUpdatePathStates: Activated path "NULL" for NMP device "naa.6006016002801a000c6e1c7cd5aade11".
  vmkernel: 10:08:26:44.937 cpu7:4204)WARNING: NMP: nmp_DeviceStartLoop: NMP Device "naa.6006016002801a000c6e1c7cd5aade11" is blocked.Not starting I/O from device.
  vmkernel: 10:08:26:44.937 cpu3:4116)WARNING: FS3: 6650: Reservation error: Timeout
  vmkernel: 10:08:26:44.937 cpu3:4116)WARNING: NMP: nmp_IssueCommandToDevice: I/O could not be issued to device "naa.6006016002801a005484e68ad5aade11" due to Not found
  vmkernel: 10:08:26:44.937 cpu3:4116)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.6006016002801a005484e68ad5aade11": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.

Environment

VMware ESXi 4.0.x Installable
VMware ESX Server 3.5.x
VMware ESXi 3.5.x Embedded
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Embedded
VMware ESX 4.0.x
VMware ESX 4.1.x
VMware ESXi 3.5.x Installable
VMware ESXi 4.1.x Installable

Resolution

This issue occurs if you are using BIOS version 1.24 or 1.26.

To resolve this issue, update the FC HBA BIOS to the latest firmware version.

Notes:

• Consult your storage and fibre channel switch hardware vendor compatibility matrix before updating the QLogic FC HBA BIOS.
• Contact the hardware vendor for vendor-specific instructions regarding downloading and updating the HBA BIOS.

If your issue continues after upgrading the BIOS, it may be due to a storage IO contention problem. Contact the hardware vendor to check the storage and fabric for issues.