Local Datastore Inaccessibility due to disk medium read errors and firmware fault
search cancel

Local Datastore Inaccessibility due to disk medium read errors and firmware fault

book

Article ID: 450566

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • A Broadcom/LSI MegaRAID storage controller issues a predictive replacement alert for a Hard Disk.

  • The local datastore accessed though the storage controller becomes intermittently inaccessible.

  • Rebooting the host temporarily restores access to the local datastore.

  • Following a reboot, the datastore only remains accessible for a short period of time before dropping again.

  • Storage-dependent operations, such as Zerto replication or backups, fail to complete.

  • Storage paths routed through the local controller transition to a "dead" state.

Environment

VMware vSphere ESXi 8.x

Cause

  • This issue occurs when physical drives experience unrecoverable medium errors, which triggers a firmware fault on the Broadcom/LSI MegaRAID storage adapter.

  • When the lsi_mr3 driver encounters these hardware errors, the storage controller firmware attempts an Online Controller Reset (OCR) to recover.

  • If the controller is unable to successfully complete the OCR, the adapter enters a permanent "critical" state. This halts all I/O operations, resulting in inaccessiable datastores and storage path failure.

  • The following excerpts from the host logs (e.g., vmkernel.log) illustrate this sequence of events. Starting with the drive errors, followed by the firmware fault, the failed reset attempt, and finally the critical adapter state:

ESXi: /var/log/vmkernel.log
####-##-##T##:##:.###Z In(182) vmkernel: cpu0:2097786)lsi_mr3_0000:81:00.0: megasas_hotplug_work: 583: Event : Unrecoverable medium error during recovery on Drive 05(e251/Port 1I Box 3 Bay 3) at 709d4980
####-##-##T##:##:.###Z In(182) vmkernel: cpu0:2097786)lsi_mr3_0000:81:00.0: megasas_hotplug_work: 581: event code: 0x61.
####-##-##T##:##:.###Z In(182) vmkernel: cpu0:2097786)lsi_mr3_0000:81:00.0: megasas_hotplug_work: 583: Event : Puncturing bad block on Drive 05(e251/Port 1I Box 3 Bay 3) at 709d4980

####-##-##T##:##:.###Z In(182) vmkernel: cpu0:2097786)lsi_mr3_0000:81:00.0: megasas_hotplug_work: 583: Event : Unrecoverable medium error during recovery on Drive 06(e250/Port 1I Box 2 Bay 2) at 709d4980

####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu2:2097441)WARNING: lsi_mr3_0000:81:00.0: fusionWaitForOutstanding: 4202: FW in FAULT state. Fault code 0x10000 Sub code 0x0

####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu5:2097441)WARNING: lsi_mr3_0000:81:00.0: mfi_fwReadyStateCheck: 407: Waiting for FW to come to ready state

####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu2:2097441)WARNING: lsi_mr3_0000:81:00.0: fusionReset: 4495: Failed to transition controller to ready.
####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu2:2097441)WARNING: lsi_mr3_0000:81:00.0: fusionReset: 4583: Reset failed, stopping adapter.

####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu9:2121636)WARNING: lsi_mr3_0000:81:00.0: mfiIoctl_FWCmd_SGL64: 422: Adapter in critical state
####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu9:2121636)WARNING: lsi_mr3_0000:81:00.0: mfiIoctl_FWCmd_SGL64: 422: Adapter in critical state
####-##-##T##:##:.###Z Wa(180) vmkwarning: cpu9:2121636)WARNING: lsi_mr3_0000:81:00.0: mfiIoctl_FWCmd_SGL64: 422: Adapter in critical state

Resolution

Contact Hardware Vendor Support to investigate the specific physical drives reporting "Unrecoverable medium errors." and check the integrity of the storage controller.




Powered by Wolken Software