vSAN OSA clusters running on 8.0 U2 or 8.0U2b may hit a rare condition where the vSAN datastore reaches out of logical space due to frequent deletion of VMs.
search cancel

vSAN OSA clusters running on 8.0 U2 or 8.0U2b may hit a rare condition where the vSAN datastore reaches out of logical space due to frequent deletion of VMs.

book

Article ID: 312822

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

This KB provides a workaround to the “Out Of Logical Space” condition and/or virsto create vdisk  failure with "no space left in XLog registry" for vSAN OSA clusters running vSphere 8.0U2 and 8.0U2b

Symptoms:
  • VMs fail to deploy/create with the error "There is no more space", yet there is plenty of space on the vSAN datastore.
  • vSAN Health "physdiskcapacity" reports out of logical space
  • Example1: Hosts might run into error  “Out Of Logical Space” errors seen in the Logs, not specific to any Host(s):
2024-03-06T20:41:02.679Z Wa(180) vmkwarning: cpu64:2561511 opID=ac4b68ff)WARNING: 

2024-03-06T22:57:53.736Z Wa(180) vmkwarning: cpu124:2099922)WARNING: LSOM: LSOMOutOfLogical:12361: Throttled: cp 4194304 cu 0

2024-03-06T23:07:05.161Z Wa(180) vmkwarning: cpu71:2099290)WARNING: VSAN: VsanSparseWriteDone:4590: Throttled: Write error on '########-########-####-####-####-########3b50': token status 'No space left on device',SCSI status 8 (OK:BUSY)
 
  • Example2: Hosts which might run into  VirstoCreateVDiskV4:1528: Create failed, status=Limit exceeded after seeingXLogAllocateVDisk:1111: no space left in XLog registry 
    • ou.0:2024-08-12T05:13:06.039Z Wa(180) vmkwarning: cpu102:2101296)WARNING: Service: XLogAllocateVDisk:1111: no space left in XLog registry - 2097172 2097152
      vmkernel.0:2024-08-12T05:16:47.204Z Wa(180) vmkwarning: cpu52:2109598)WARNING: Service: XLogAllocateVDisk:1111: no space left in XLog registry - 2097172 2097152

      vmkernel.0:2024-08-12T05:13:06.039Z Wa(180) vmkwarning: cpu102:2101296)WARNING: Service: VirstoCreateVDiskV4:1528: Create failed, status=Limit exceeded
      vmkernel.0:2024-08-12T05:16:47.204Z Wa(180) vmkwarning: cpu52:2109598)WARNING: Service: VirstoCreateVDiskV4:1528: Create failed, status=Limit exceeded
  • vSAN device mointor serivice will attempt to unmount/remount the DiskGroup after encountering "XLogAllocateVDisk:1111: no space left in XLog registry"   
    • 2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device eui.01000000000000005cd2e4db0a545551 state is DISKGROUP_UNDER_REPAIR
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device ########-########-####-####-####-########0bb5 not found.
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device eui.3656323054a065140025384500000003 state is DISKGROUP_UNDER_REPAIR
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device eui.3656323054a043630025384500000002 state is DISKGROUP_UNDER_REPAIR
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device eui.3656323054a028980025384500000002 state is DISKGROUP_UNDER_REPAIR
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device eui.3656323054a029570025384500000002 state is DISK_UNDER_REPAIR
      2024-08-22T21:51:29Z In(14) vsandevicemonitord[2101901]: Device ########-########-####-####-####-########3767 not found.

      2024-08-22T22:05:49Z In(14) vsandevicemonitord[2101901]: Unmounting succeeded on VSAN diskgroup with Cache tier : eui.01000000000000005cd2e4db0a545551.
      2024-08-22T22:05:49Z In(14) vsandevicemonitord[2101901]: Device eui.3656323054a043630025384500000002 was already unmounted.
      2024-08-22T22:08:18Z In(14) vsandevicemonitord[2101901]: Mounting succeeded on VSAN diskgroup with Cache tier : eui.01000000000000005cd2e4db0a545551.
      2024-08-22T22:08:18Z In(14) vsandevicemonitord[2101901]: Repair successful for device ########-########-####-####-####-########92d3
  • In certain conditions when vsan device mointor tries to repair the disk group, the unmount operation might get stuck after encountering
    • "XLogAllocateVDisk:1111: no space left in XLog registry"  
      
    • 2024-05-13T06:54:55.474Z Er(163) vsanesxcmd[13546550]: [Originator@6876 sub=Libs] VsanUtil: Disk with vSAN uuid ########-########-####-####-####-########ea6f failed to appear in LSOM
      2024-05-13T06:54:55Z In(14) vsandevicemonitord[2102553]: stderr Errors:
      2024-05-13T06:54:55Z In(14)[+] vsandevicemonitord[2102553]: Unable to mount: Disk with vSAN uuid ########-########-####-####-####-########ea6f failed to appear in LSOM
      2024-05-13T06:54:55Z In(14)[+] vsandevicemonitord[2102553]: , stdout  from command vsan storage diskgroup mount -s eui.01000000000000005cd2e4bda85e5551.
      2024-05-13T06:54:55Z In(14) vsandevicemonitord[2102553]: Mounting failed on VSAN device eui.01000000000000005cd2e4bda85e5551.
      2024-05-13T06:54:55Z In(14) vsandevicemonitord[2102553]: Repair attempt 1 for device ########-########-####-####-####-########e0b6


Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.




Environment

VMware vSAN 8.0 Update 2
VMware vSAN 8.0 Update 2b

Cause

When a large number of VMs are deleted and space is not reclaimed fast enough while new VMs are being provisioned  (VDI like workloads) we might encounter this out of logical space scenario and/or no space error in xlog registry while creating virsto vdisks.

Resolution

This is a known issue affects ONLY vSAN OSA 8.0 U2 and vSAN OSA 8.0 U2b under very specific conditions explained above.

VMware recommends applying the configuration changes suggested in the "Workaround" section to mitigate the risk of encountering this issue proactively on vSAN OSA 8.0 U2 / vSAN OSA 8.0 U2b releases.

It is high recommended to apply the below workaround and upgrade to vSphere 8.0 U3b which contains the fixes.


Workaround:
Apply the below advanced configuration on all hosts in the cluster and run auto-backup.sh:

esxcfg-advcfg -s 0 /LSOM/lsomPlogZeropV2;/sbin/auto-backup.sh

Powered by Wolken Software