Unable to Create New Datastore Due to Corrupted or Inaccessible Storage Device
search cancel

Unable to Create New Datastore Due to Corrupted or Inaccessible Storage Device

book

Article ID: 415116

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms

  •  While data corruption often manifests as an inability to create new datastores, it can also impact the usability of existing virtual machines residing on the affected LUN due to corrupted VMFS metadata, often resulting in I/O errors or the inability to register/power on VMs.     
  • Unable to create a VMFS datastore from the vSphere Client or Host Client.— the operation did not progress beyond the initialization stage
  • Rescan storage operations fail or take an unusually long time to complete.
  • Host fail to rejoin back to cluster 
  • VMs appear in an "INVALID" state following an abrupt ESXi host reboot.I
  • Inability to register or power on specific VMs because of stale exclusive lock (.lck) files.
  • Log analysis shows vob.vmfs.metadata.corruptondisk errors if VMFS volume metadata corruption is also present.
  • Affected datastore was showing 0 byte. 
  • VMFS volumes report metadata corruption warnings in /var/log/vmkwarning.log or /var/log/vmksummary.log.

    2025-10-21T01:06:00.599Z Wa (180) vmkwarning: cpu0:2101033) WARNING: FS3: 636: VMFS volume <Datastore-name>/681281af-68c7d235-#####-####### on naa.############:1 has been detected corrupted
    2025-10-21T01:06:36.624Z Wa (180) vmkwarning: cpu0:2101033) WARNING: Fil3: 10590: Found invalid object on 681281af-68c7d235-#####-######### <FD c0 r0> expected <FD c28 r0>
    2025-10-21T01:06:36.624Z Wa(180) vmkwarning: cpu0:2101033) WARNING: FS3: 636: VMFS volume <Datastore name>/681281af-######-####-######## on naa.#############:1 has been detected corrupted
  • ESXi hosts report repeated READ/WRITE command failures (ScsiDeviceIO errors) in /var/log/vmkernel.log.

    2025-10-21T01:32:57.631Z In(182) vmkernel: cpu274:2100454)ScsiDeviceIO: 4619: Cmd(0x45db2c9b1fc0) 0x28, CmdSN 0x1f32de1 from world 3086524 to dev "naa.##############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0
    2025-10-21T01:32:57.634Z In(182) vmkernel: cpu274:2100454)ScsiDeviceIO: 4619: Cmd(0x45db2c9b1fc0) 0x28, CmdSN 0x1f32de1 from world 3086524 to dev "naa.##############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0
    2025-10-21T01:32:57.637Z In(182) vmkernel: cpu274:2100454)ScsiDeviceIO: 4619: Cmd(0x45cb29563940) 0x28, CmdSN 0x1771 from world 2101307 to dev    "naa.###############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xb 0x44 0x0

     

  • Existing virtual machines on the affected datastore appear as 'Invalid' or 'Inaccessible' in the vSphere Client, often following an abrupt host reboot or storage path event.

Environment

VMware vSphere ESXi 

Cause

Due to data corruption on one of the backend storage arrays, all read and write I/O operations directed to the affected datastores began to fail. These repeated I/O failures caused the hostd service on the ESXi host to become overloaded while continuously attempting to process unresponsive I/O requests. As a result, hostd entered a busy state and was unable to perform datastore creation task.

Resolution

Note: If existing virtual machines on the datastore are impacted, investigate if VM files are held by stale host locks (e.g., following an abrupt reboot) using command-line utilities. Correlate this with vmkernel.log and vobd.log to confirm if VMFS metadata corruption is the root cause rather than a stale lock.

To resolve the issue, perform the following steps:

  1. Identify and isolate affected LUNs:
    Review /var/log/vmkernel.log for repeated read/write or SCSI sense key errors on specific device IDs (naa.*).

  2. Validate storage health:
    Engage your storage administrator to check for array-side issues such as LUN corruption, path errors, or controller faults.

  3. Attempt to storage vmotion the VMs to other datastores to reduce the impact of the storage issues affecting the remaining VMs.
    Note: If storage vmotions fail due to data corruption and there are no other datastores availabel to move VMs to, check the Additional Information section in this article.

  4. Unpresent or unzone the affected LUNs from the ESXi hosts to prevent further access attempts.

  5. Clean up stale devices:

    • Run esxcli storage core adapter rescan --all

    • If rescan fails or stale entries persist, restart management agents using: Restarting Management Agents in ESXi

    • Reboot the affected ESXi hosts if the stale devices still remain after cleanup.

  6. Revalidate connectivity:

    • Perform a rescan after reboot to ensure healthy storage paths are visible.

    • Verify that accessible LUNs are correctly detected.

  7. Create the VMFS datastore on the healthy storage device once the environment stabilizes. Follow steps from the article Create a VMFS Datastore

Additional Information

In situations where VMs cannot be storage vmotioned and there is observed datastore corruption, you can attempt to repair the corruption using VOMA tool. However, data corruption can happen in a variety of ways, and not in all cases it can be used. Please contact Broadcom support for the feasibility of using VOMA in the particular situation encountered.

 

 

Powered by Wolken Software