The vSAN disk appears as Absent in vSAN Disk Management
search cancel

The vSAN disk appears as Absent in vSAN Disk Management

book

Article ID: 326548

calendar_today

Updated On:

Products

VMware vSAN VMware vSAN 8.x VMware vSAN 7.x VMware vSAN 6.x

Issue/Introduction

  • vSAN Health service reports a warning about disk(s)
  • Unable to Remove an Absent vSAN disk from vCenter UI
  • Unable to remove vSAN disk reference where disk has already been replaced
  • After physically replacing a failed vSAN Capacity drive: Part has been replaced, but it is in unclaimed condition
  • The vCenter vSphere UI, vCenter > Host and Cluster View > vSAN cluster >  Monitor > Skyline Health > RETEST > Operational Health : will display "Absent Disk" with overall Health with red mark
  • General vSAN error. vSAN disk data evacuation resource check has failed for disk or disk-group vsan:########-####-####-####-############ (########-####-####-####-############) with mode noAction on host 10.x.x.4. Go to vSAN Data Migration Pre-Check page for more details..




  • In vCenter vSphere UI, vCenter > Host and Cluster View > vSAN cluster > Configure > vSAN > Disk Management, you see a Disk group with a red exclamation diamond and, in the detail window below, it is marked as Absent vSAN Disk.

  • Multiple vSAN nodes reporting Absent vSAN disk error at the same time.


  • When trying to remove absent disk it fails with an error " A General system error occurred

  • You may also see inaccessible objects being reported in Cluster -> Monitor -> Virtual objects

Environment

  • VMware vSAN (All Versions)

Cause

This issue occurs due to one or all of these reasons:
  • A physical disk has failed.
  • The physical disk has been removed from server, before removing the disk from vSAN disk group.
  • If multiple vSAN nodes are reporting "Absent vSAN disk error" for multiple disk groups, then the issue can be from the backend enclosure. To validate that you can compare the nunber physical disks on a host with disks the disk group showing in healthy state on the host(s) with absent error.

  • To further verify the absent disks, you can check the host's vmkernel.log for "device not found" messages:

[from /var/run/log/vmkernel.log]

2025-09-27T10:32:00.210Z Wa(180) vmkwarning: cpu77:3335255)WARNING: StorageDeviceVsi: 426: Device vsan:###-####-##-##-####not found.
2025-09-27T10:35:56.435Z Wa(180) vmkwarning: cpu54:3336825)WARNING: StorageDeviceVsi: 426: Device vsan:###-####-##-##-###not found.
2025-09-27T10:35:56.457Z Wa(180) vmkwarning: cpu56:2101826 opID=4ddeb394)WARNING: StorageDeviceVsi: 426: Device vsan:###-###-####-##-##### not found.
2025-09-27T10:36:43.030Z Wa(180) vmkwarning: cpu27:3337218)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-##-####-### not found.
2025-09-27T10:40:56.568Z Wa(180) vmkwarning: cpu91:3338863)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-####-##-#### not found.
2025-09-27T10:40:56.583Z Wa(180) vmkwarning: cpu60:2101826 opID=532dd31b)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-###-####-### not found
 
or NVMe errors as described in KB: vSAN NVMe disk report read only critical warning
  • From the command line, run this command:

    $ esxcli vsan storage list | less

    output for the device:

         

     Unknown
             Device: Unknown
             Display Name: Unknown
             Is SSD: false
             VSAN UUID: ########-####-####-####-############ 
             VSAN Disk Group UUID:
             VSAN Disk Group Name:
             Used by this host: false
             In CMMDS: false
             On-disk format version: -1
             Deduplication: false
             Compression: false
             Checksum:
             Checksum OK: false
             Is Capacity Tier: false
             Encryption Metadata Checksum OK: true
             Encryption: false
             DiskKeyLoaded: false
             Is Mounted: false
             Creation Time: Unknown
  • In the ESXI host logs
    /var/run/log/vobd.log will show below messages 

    vobd.log:YYYY-MM-ddThh:mm:ss.342Z: [scsiCorrelator] 4079214243471us: [esx.problem.scsi.device.state.permanentloss] Device: naa.############### has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.
    vobd.log:YYYY-MM-ddThh:mm:ss.374Z: [scsiCorrelator] 4104631067753us: [vob.scsi.device.state.permanentloss] Device :naa.############### has been removed or is permanently inaccessible.
    

    /var/run/log/vmkernel.log, check for the SCSI sense code against the device identifier:
    YYYY-MM-ddThh:mm:ss.sssZ In(182) vmkernel: cpu52:2098242)ScsiDeviceIO: 4672: Cmd(0x45de37490640) 0x25, CmdSN 0x113d38b from world 0 to dev "naa.###############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0xe2

    Sense Key [0x4] in the above example is a hardware error


  • When reviewing smart data for the device you may see it reporting a health status of failed and/or offline 
    # esxcli storage core device smart get -d naa.xxxxxxxxxxxxxxxx
    Parameter                     Value              Threshold  Worst
    ----------------------------  -----------------  ---------  -----
    Health Status                 FAILED/OFFLINE     N/A        N/A
    Media Wearout Indicator       N/A                N/A        N/A
    Write Error Count             0                  N/A        N/A
    Read Error Count              1638320548         N/A        N/A
    Power-on Hours                N/A                N/A        N/A
    Power Cycle Count             621                N/A        N/A
    Reallocated Sector Count      N/A                N/A        N/A
    Raw Read Error Rate           N/A                N/A        N/A
    Drive Temperature             31                 N/A        N/A
    Driver Rated Max Temperature  N/A                N/A        N/A
    Write Sectors TOT Count       N/A                N/A        N/A
    Read Sectors TOT Count        N/A                N/A        N/A
    Initial Bad Block Count       N/A                N/A        N/A
  • Running the command vdq -Hi will show the impacted disk(s) as UUIDs and not a device
    Mappings:
       DiskMapping[0]:
               SSD:  naa.################
                MD:  naa.################
                MD:  naa.################
                MD:  naa.################
                MD:  ########-####-####-####-############    <-- Instead of showing the NAA ID, the vSAN UUID of the disk is shown
    
     



Resolution

  • Please contact  the hardware vendor to replace the faulty disk.

    • NOTE: Before removing the the Absent disk ensure that correct DISK_UUID.
      1. Get the UUID of the absent disk either from vCenter > Configure > vSAN > Disk Management or by running
        $ esxcli vsan storage list |grep "Device: Unknown"|grep <DISK_UUID:>
        Device : Unknown
        Device: Unknown
        Display Name: Unknown
        VSAN UUID: ########-####-####-####-############ 



      2. Use this command to check if the drive contains any data.

        $ cmmds-tool find -u DISK_UUID -f json


      3. Output similar to this will be displayed, indicating there is no data contained in the storage device:
        {
        "entries":
        [
        ]
        }
        

      4. Use this command to check if any of the vSAN objects claim to have an association with the volume named as VSAN_UUID: 

        $ cmmds-tool find -t DOM_OBJECT -f json |grep <DISK_UUID>


      5. If there is no data associated with the DISK_UUID , it should not display results and return the command prompt indicating that no objects have data associated to the VSAN_UUID.

 

  • If you have inaccessible objects, please open a case with Broadcom Support (see KB: Creating and managing Broadcom support request (SR) cases) for assistance in determining if the objects would be recoverable after replacement. Be especially mindful of this in multiple disk failure scenarios. 
  • The following article explain the replacement of the cache and capacity disk 
  • In the vCenter vSphere UI, if the above operation fails then follow the below steps to remove the disk from the disk group:
    • Replace the physical disk from the hardware level by powering off the ESXi 
      • If the server hardware supports Hot Swap of the disk , the faulty disk can be replaced after putting the ESXi host in Maintenance Mode
        • The newly added disk should be visible in Host and Clusters View -> ESXi Host -> Configure -> Storage Devices 

      • If Hot Swap is not supported, the reboot of the ESXI  will show the new disk  in the vCenter vSphere UI -> Host and Cluster view -> ESXI Host -> Configure -> Storage Devices.


  • To remove the failed disk from the vSAN cluster after it has been validated :
    1. After the device is determined to be an empty reference, remove the DISK_UUID using command below:

      $ esxcli vsan storage remove -u DISK_UUID


    2. Re-run the same command to verify if the listing for this volume is removed:

      $ esxcli vsan storage list|grep DISK_UUID


    3. Refresh the view in the vCenter vSphere UI and the volume will also be removed there. ( vCenter > Host and Cluster View > vSAN cluster > Configure > vSAN > Disk Management )

    4. If the command to remove the drive fails with this error:

      Unable to remove device: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Sysinfo error: Not found See VMkernel log for details.

      1. Check if the vSAN cluster is configured with dedup and compression , you need to destroy and re-create the entire Disk Group which contains the affected disk UUID 
      2. If the vSAN cluster does not use dedup and compression, the disk can be added directly to the respective Disk Group( Add Devices to the Disk Group in vSAN Cluster )

Additional Information

  • The Hardware is identified at BIOS level by the installed firmware on the server. The SCSI code handling changes from a vendor to vendor and hence some of the vendors may provide the hot swappable HDD option for failed disk
  • However, the ESXi OS requires a reboot of the host to identify the new disk ( hardware changes in the system )  when the existing failed disk is replaced in the same slot to refresh the driver information in the ESXi kernel.
  • A newly added disk in the empty slot will be identified in real time. However, this disk replacement and identification behavior changes from vendor to vendor.
  • Adding and Removing Disks in vSAN via GUI and CLI - YouTube