VM summary page display a warning "There is no more space for virtual disk ########.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session"
search cancel

VM summary page display a warning "There is no more space for virtual disk ########.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session"

book

Article ID: 413125

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms : 

  • Virtual machine's summary page displays the following warning:

    "There is no more space for virtual disk ########.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session"

     
  • The /var/run/log/hostd.log file confirms the above mentioned error. This log file also records VM's heartbeat status changing from green to red. 

    YYYY-MM-DDTHH:MM.SSSZ verbose hostd[#######] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/###########.vmx opID=########-##-##-####] Handling vmx message #########: There is no more space for virtual disk '###########.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.
    -->
    YYYY-MM-DDTHH:MM.SSSZ info hostd[#######] [Originator@#### sub=Vimsvc.ha-eventmgr opID=########-##-##-####] Event 21949 : Message on ########### on ####-### in ha-datacenter: There is no more space for virtual disk '###########.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.
    YYYY-MM-DDTHH:MM.SSSZ verbose hostd[#######] [Originator@#### sub=Vmsvc.vm:/vmfs/volumes/vsan:################-################/########-####-####-####-############/###########.vmx opID=########-##-##-####] Updating current heartbeatStatus: green -> red

  • Virtual machines may fail to power on.
  • VMware vSAN objects display a status of "Reduced Availability" within the vSphere Client interface under vSAN Cluster > Monitor > Virtual Objects.

  • Clicking View Component Placement on an affected object reveals components residing in an Absent or Degraded state.



  • The output of the below command will confirm that one or more objects are in reduced availability state. 

    esxcli vsan debug object health summary get
    Health Status                                              Number Of Objects
    ---------------------------------------------------------  -----------------
    remoteAccessible                                                           0
    inaccessible                                                               0
    reduced-availability-with-no-rebuild                                       1

  • Use the following command to verify the health of the affected object. The Health of the affected object will show as "Reduce availability".

    Example:
    esxcli vsan debug object list -u <object UUID>
    Object UUID: ########-####-####-####-############
       Version: ##
       Health: reduced-availability-with-no-rebuild
       Owner: #################
       Size: 250.00 GB
       Used: 484.11 GB

Environment

VMware vSAN 7.x
VMware vSAN 8.x
VMware vSAN 9.x

Cause

This issue occurs when underlying vSAN storage devices lack sufficient free capacity to process object modifications. When the CLOMD attempts to merge vSAN object concatenations, the operation fails. Consequently, subsequent component reactive rebalancing taksks also fail to complete.

Cause validation:

  • vSAN Skyline Health displays capacity warnings such as:

    • "Disk Capacity"
    • "Storage space”
    • “What if most consumed host fails”





  • The /var/run/log/clomd.log file confirms that reconfiguration tasks fail due to capacity exhaustion:

    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1809797581] CLOMProcessWorkItem: Op CONCAT starts:1809797581
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1809797581] CLOMReconfigure: Reconfiguring ########-####-####-####-############ workItem type CONCAT
    YYYY-MM-DDTHH:MM.SSSZ warning clomd[#######] [Originator@6876 opID=1809797581] CLOMCreateOneDiskList: Disk dropped due to decomDisks:0 decomNodes:0 maxCompLimit:0 capFull:69 unhealthySSD:0 unhealthyNodes:0 unhealthyDisks:0 ReadCacheResvFailures:0 noMinFreeSpace: 0 Nodes with no disk:0 deltaCreateLimit: 0 numNoMinFreeSpace: 0 numIncompatibleStorageType: 0, numIncompatibleEncryptedDisk: 0 numIncompatibleDedupDisk: 0, numIncompatibleDiskVersion: 0
    YYYY-MM-DDTHH:MM.SSSZ error clomd[#######] [Originator@6876 opID=1809797581] CLOMAssignDisksToConfig: Failed to assign disks: Not found
    YYYY-MM-DDTHH:MM.SSSZ warning clomd[#######] [Originator@6876 opID=1809797581] CLOMReconfigure: WorkItem failed ignoring disk capacity/transient capacity/component limit: Not found.
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1809797581] CLOM_PublishResyncBytes: About to fail the workItem(Underlying device has no free space), reset queued resync bytes to 0
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1809797581] CLOMProcessWorkItem: Op ends:1809797581

  • The /var/run/log/clomd.log file also confirms that the component reactive rebalance operations fail due to capacity issues:

    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1804312346] CLOMProcessWorkItem: Op REACTIVE_REBALANCE starts:1804312346
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1804312346] CLOMReconfigure: Reconfiguring ########-####-####-####-############ workItem type REACTIVE_REBALANCE
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1804312346] CLOMReplCompPreWorkRebalance: Moving component ########-####-####-####-############ for rebalancing from disk ########-####-####-####-############
    YYYY-MM-DDTHH:MM.SSSZ error clomd[#######] [Originator@6876 opID=1804312346] CLOMBalance_CheckMoveGoodnessV2:  Failed to move comp ########-####-####-####-############:########-####-####-####-############ size: 258525993369 initSD: 0.047563, finSD: 0.126757 srcFF 0.807888 destFF 0.900998
    YYYY-MM-DDTHH:MM.SSSZ warning clomd[#######] [Originator@6876 opID=1804312346] CLOMBalance_CheckMoveGoodnessV2: Unsetting disk assignment!
    YYYY-MM-DDTHH:MM.SSSZ error clomd[#######] [Originator@6876 opID=1804312346] CLOMReplaceComponentsWork: Partial fix supported but failed to find any successful fixes
    YYYY-MM-DDTHH:MM.SSSZ error clomd[#######] [Originator@6876 opID=1804312346] CLOMReplaceComponentsWork: Could not replace components for ########-####-####-####-############: Not found.
    YYYY-MM-DDTHH:MM.SSSZ warning clomd[#######] [Originator@6876 opID=1804312346] CLOMReconfigure: WorkItem failed ignoring disk capacity/transient capacity/component limit: Not found.
    YYYY-MM-DDTHH:MM.SSSZ error clomd[#######] [Originator@6876 opID=1804312346] CLOMReconfigure: exit: obj ########-####-####-####-############ transiantCapGenerated - total: 0, site1: 0, site2: 0, workItem type REACTIVE_REBALANCE configDelay 0 newConfigGenerated 0 status Not found
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1804312346] CLOM_PublishResyncBytes: About to fail the workItem(Failure), reset queued resync bytes to 0
    YYYY-MM-DDTHH:MM.SSSZ info clomd[#######] [Originator@6876 opID=1804312346] CLOMProcessWorkItem: Op ends:1804312346

Resolution

To resolve this issue, free or expand storage resources across the vSAN cluster using one of the following methods:

  1. Add physical capacity by adding disks or new nodes to the vSAN cluster.
  2. Delete unused snapshots or decommission unnecessary VMs to reclaim space.
  3. Power off non critical VMs to reduce active swap file overhead. 
Powered by Wolken Software