The vSAN disk appears as Absent in vSAN Disk Management
search cancel

The vSAN disk appears as Absent in vSAN Disk Management

book

Article ID: 326548

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptom:

  • The vSAN Skyline Health report warning about disk(s)

  • The vCenter vSphere UI, vCenter > Host and Cluster View > vSAN cluster >  Monitor > Skyline Health > RETEST > Operational Health : will display "Absent Disk" with overall Health with red mark




  • In vCenter vSphere UI, vCenter > Host and Cluster View > vSAN cluster > Configure > vSAN > Disk Management, you see a Disk group with a red exclamation diamond and, in the detail window below, it is marked as Absent vSAN Disk.

  • Multiple vSAN nodes reporting Absent vSAN disk error at the same time.


  • When trying to remove absent disk it fails with an error " A General system error occurred. Disk is not mounted to this host" 

  • From the command line, run this command:

    $ esxcli vsan storage list | less

    output for the device:

      Unknown
         Device: Unknown
         Display Name: Unknown
         Is SSD: false
         VSAN UUID: 52fdc833-b925-8454-c88b-###########
         VSAN Disk Group UUID:
         VSAN Disk Group Name:
         Used by this host: false
         In CMMDS: false
         On-disk format version: -1
         Deduplication: false
         Compression: false
         Checksum:
         Checksum OK: false
         Is Capacity Tier: false
         Encryption Metadata Checksum OK: true
         Encryption: false
         DiskKeyLoaded: false
         Is Mounted: false
         Creation Time: Unknown

  • The ESXI host /var/run/log/vobd.log will show below messages 

    vobd.log:YYYY-MM-ddThh:mm:ss.342Z: [scsiCorrelator] 4079214243471us: [esx.problem.scsi.device.state.permanentloss] Device: naa.############### has been removed or is permanently inaccessible. Affected datastores (if any): Unknown.
    vobd.log:YYYY-MM-ddThh:mm:ss.374Z: [scsiCorrelator] 4104631067753us: [vob.scsi.device.state.permanentloss] Device :naa.############### has been removed or is permanently inaccessible.


    In vmkernel logs, check for the SCSI sense code against the device identifier:
     
    2025-03-14T10:25:46.444Z In(182) vmkernel: cpu52:2098242)ScsiDeviceIO: 4672: Cmd(0x45de37490640) 0x25, CmdSN 0x113d38b from world 0 to dev "naa.###############" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x4 0x44 0xe2
     
    Sense Key [0x4] --> HARDWARE ERROR


    # esxcli storage core device smart get -d naa.xxxxxxxxxxxxxxxx
    Parameter                     Value              Threshold  Worst
    ----------------------------  -----------------  ---------  -----
    Health Status                 FAILED/OFFLINE     N/A        N/A
    Media Wearout Indicator       N/A                N/A        N/A
    Write Error Count             0                  N/A        N/A
    Read Error Count              1638320548         N/A        N/A
    Power-on Hours                N/A                N/A        N/A
    Power Cycle Count             621                N/A        N/A
    Reallocated Sector Count      N/A                N/A        N/A
    Raw Read Error Rate           N/A                N/A        N/A
    Drive Temperature             31                 N/A        N/A
    Driver Rated Max Temperature  N/A                N/A        N/A
    Write Sectors TOT Count       N/A                N/A        N/A
    Read Sectors TOT Count        N/A                N/A        N/A
    Initial Bad Block Count       N/A                N/A        N/A

 

Validation :

Ensure that correct VSAN_UUID is noted before removing the disk that is showing as absent.

  1. Get the UUID of the absent disk either from vCenter > Configure > vSAN > Disk Management or by running

    $ esxcli vsan storage list |grep "Device: Unknown"|grep "VSAN UUID:"
    Device : Unknown
    Device: Unknown
    Display Name: Unknown
    VSAN UUID: 529a152a-6e87-d8dc-c890-############

  2. Use this command to check if the drive contains any data.

    $ cmmds-tool find -u VSAN_UUID -f json

  3. Output similar to this will be displayed, indicating there is no data contained in the storage device:

    {
    "entries":
    [
    ]
    }

  4. Use this command to check if any of the vSAN objects claim to have an association with the volume named as VSAN_UUID: 

    $ cmmds-tool find -t DOM_OBJECT -f json |grep VSAN_UUID

  5. If there is no data associated with the vSAN_UUID , it should not display results and return the command prompt indicating that no objects have data associated to the VSAN_UUID.

 

Environment

VMware vSAN (All Versions)

Cause

This issue occurs due to one or all of these reasons:
  • A physical disk has failed.
  • The physical disk has been removed from server, before removing the disk from vSAN disk group.
  • If multiple vSAN nodes are reporting "Absent vSAN disk error" for multiple disk groups, then the issue can be from the backend enclosure. To validate the same you can compare the physical disks on a host with all the disk group in healthy state to a host with absent error.

Cause Validation 

vmkernel.log reports device not found. 

Log path : /var/run/log/vmkernel.log 

2025-09-27T10:32:00.210Z Wa(180) vmkwarning: cpu77:3335255)WARNING: StorageDeviceVsi: 426: Device vsan:###-8848-##-##-####not found.
2025-09-27T10:35:56.435Z Wa(180) vmkwarning: cpu54:3336825)WARNING: StorageDeviceVsi: 426: Device vsan:###-8848-##-##-###not found.
2025-09-27T10:35:56.457Z Wa(180) vmkwarning: cpu56:2101826 opID=4ddeb394)WARNING: StorageDeviceVsi: 426: Device vsan:###-###-9b0d-##-##### not found.
2025-09-27T10:36:43.030Z Wa(180) vmkwarning: cpu27:3337218)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-##-59d8-### not found.
2025-09-27T10:40:56.568Z Wa(180) vmkwarning: cpu91:3338863)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-9b0d-##-#### not found.
2025-09-27T10:40:56.583Z Wa(180) vmkwarning: cpu60:2101826 opID=532dd31b)WARNING: StorageDeviceVsi: 426: Device vsan:###-##-###-59d8-### not found
 
 
2025-09-29T07:29:30.667Z Wa(180) vmkwarning: cpu50:2098186)WARNING: NVMEIO:2644 command 0x45ba9e7a0800 failed: ctlr 257, queue 2, psaCmd 0x45ba82012080, status 0x182, opc 0x1, cid 446, nsid 1
2025-09-29T07:29:30.667Z Wa(180) vmkwarning: cpu50:2098186)WARNING: NVMEPSA:217 Complete vmkNvmeCmd: 0x45ba9e7a0800, vmkPsaCmd: 0x45ba82012080, cmdId.initiator=0x430b1dda68c0, CmdSN: 0x2, status: 0x182
2025-09-29T07:29:30.667Z Wa(180) vmkwarning: cpu50:2098186)WARNING: HPP: HppNvmeThrottleLogForDevice:600: NVMe Cmd 0x1 (0x45ba82012080, 2104939) to dev "t10.NVMe____####_7###_#####_______________#######" on path "vmhba3:C0:T0:L0" Failed:
2025-09-29T07:29:30.667Z Wa(180) vmkwarning: cpu50:2098186)WARNING: HPP: HppNvmeThrottleLogForDevice:608: Error status H:0x0 D:0x182 P:0x0 hppAction = 1
2025-09-29T07:29:30.667Z Wa(180) vmkwarning: cpu50:2098186)WARNING: NvmeUtil: 151: Error on Cmd(0x45ba82012080) 0x1, CmdSN 0x2 from world 2104939 to component "t10.NVMe____###_7450_####______________######"  H:0x0 D:0x182 P:0x0
 

Resolution

Please contact  the hardware vendor to replace the faulty disk.

The following article explain the replacement of the cache and capacity disk 

In the vCenter vSphere UI, if the above operation fails then follow the below steps to remove the disk from the disk group 

      • Replace the physical disk from the hardware level by powering off the ESXi 
        • If the server hardware supports Hot Swap of the disk , the faulty disk can be replaced after putting the ESXi host in Maintenance Mode
        • The newly added disk should be visible in Host and Cluster view > ESXi Host > Configure > Storage Devices 

      • If Hot Swap is not supported, the reboot of the ESXI  will show the new disk  in the vCenter vSphere UI > Host and Cluster view > ESXI Host > Configure > Storage Devices.

The Disk removal from the vSAN cluster after it has been validated :

    1. After the device is determined to be an empty reference, remove the vSAN_UUID using command below:

      $ esxcli vsan storage remove -u VSAN_UUID

    2. Re-run the same command to verify if the listing for this volume is removed:

      $ esxcli vsan storage list|grep VSAN_UUID

    3. Refresh the view in the vCenter vSphere UI and the volume will also be removed there. ( vCenter > Host and Cluster View > vSAN cluster > Configure > vSAN > Disk Management )

    4. If the command to remove the drive fails with this error:

      Unable to remove device: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Sysinfo error: Not found See VMkernel log for details.
      1. Check if the vSAN cluster is configured with dedup and compression , you need to destroy and re-create the entire Disk Group which contains the affected disk UUID 
      2. If the vSAN cluster does not use dedup and compression, the disk can be added directly to the respective Disk Group( Add Devices to the Disk Group in vSAN Cluster ) 

Additional Information

  • The Hardware is identified at BIOS level by the installed firmware on the server. The SCSI code handling changes from a vendor to vendor and hence some of the vendors may provide the hot swappable HDD option for failed disk
  • However, the ESXi OS requires a reboot of the host to identify the new disk ( hardware changes in the system )  when the existing failed disk is replaced in the same slot to refresh the driver information in the ESXi kernel.
  • A newly added disk in the empty slot will be identified in real time. However, this disk replacement and identification behavior changes from vendor to vendor.
  • Adding and Removing Disks in vSAN via GUI and CLI - YouTube



Powered by Wolken Software