Error "The ramdisk 'vsantraces' is full. As a result the file /vsantraces/vsantraces-########-####.####.####-############ could not be written".
Article ID: 326990
Updated On:
Products
VMware vSAN
Issue/Introduction
Symptoms:
- The vsantraces files are used for vSAN diagnostics, so if vsantraces stops logging due to no space, then historical data is lost for troubleshooting vSAN issues.
- Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails.
- Trying to view the cluster's vSAN services,
- vSAN Health in vCenter fails to load and/or Virtual Objects, and you will see an error "The request is rejected due to many pending requests."
- vSAN Health error "Unable to query vSAN Health information. Check vSphere Client logs for details."
- vSAN Service error: "Unable to retrieve the cluster configuration. Check vSphere client logs for details."
/var/log/vsantraced.log
YYYY-MM-DDTHH:MM:SS.252Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memoryYYYY-MM-DDTHH:MM:SS253Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory
OR the following events are seen YYYY-MM-DDTHH:MM:SS.062Z Er(11) chardevlogger[67289337]: Failed to write during flush: No space left on device YYYY-MM-DDTHH:MM:SS.063Z Er(11) chardevlogger[67289337]: Failed to write during flush: No space left on device
/var/log/vobd.log
[VisorfsCorrelator] 98549997090us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantraces--YYYY-MM-DDThhmmss166.gz because its ramdisk (vsantraces) is full.[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--YYYY-MM-DDThhmmss166.gz could not be written.[vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantracesUrgent--2026-02-25T17h54m56s985--xxxxxxxx-ee6b-a7b5-1052-xxxxxxxxxxx.zst because its ramdisk (vsantraces) is full.[vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantracesIODiag--2026-02-27T20h11m20s989--xxxxxxxx-ee6b-a7b5-1052-xxxxxxxxxxx.zst because its ramdisk (vsantraces) is full.
Run the following command to verify the use of the vsantraced partition.
vdf -h-----
Ramdisk Size Used Available Use% Mounted on
root 32M 2M 29M 8% --
etc 28M 232K 27M 0% --
opt 32M 0B 32M 0% --
var 48M 516K 47M 1% --
tmp 256M 12K 255M 0% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
hostdstats 114M 2M 111M 1% --
vsantraces 300M 287M 12M 95% --
vsantraces ramdisk appears to be full or nearly full on the ESXi:
Environment
VMware vSAN 7.x
VMware vSAN 8.x
Cause
- If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
- The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
- Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.
Resolution
Workaround:
- VMware recommends setting the desired size of retaining vSAN traces to around 600 MB. But additional storage is not available, so limit vsantraces to 200MB to allow a bit of overhead for additional files and avoid the ramdisk from becoming full.
- If you are using vSAN 8.x this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set --urgentnumfiles=4 --urgentsize=8 --numfiles=10 --size=8 - If you are using vSAN 7.x, this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10 - These changes are immediate, and existing files will be rolled over as newly created vSANtrace files are created over time. No services require a restart to enable this change.
Option A: Redirect vsantraces to a persistent scratch location using the steps below.
- Connect to each ESXi server in the cluster using SSH.
- Get the current vsantrace config by running.
esxcli vsan trace getvSAN Traces Directory: /vsantracesNumber Of Files To Rotate: 8Maximum Trace File Size: 180 MBLog Urgent Traces To Syslog: true - Redirect vsantraces to the new location using the below command.
esxcli vsan trace set
Cmd options:-f|--numfiles: Log file rotation for vSAN trace files.-p|--path: Path to store vSAN trace files.-r|--reset: When set to true, reset defaults for VSAN trace files.-s|--size: Maximum size of vSAN trace files in MB.
Note: Log bundles should be placed in their own host folders; otherwise, once the first Host acquires locks on the scratch location, it prevents the other hosts in the cluster from logging.
- If the Scratch partition has already been configured, vsantraces can be redirected to it. There may be unique folders for each Host; create a folder called vsantraces to separate traces per Host.
- If not, refer to Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696 to create a folder specific to each Host in the cluster.
- Once there is a folder for each Host and the logging is separated, redirect traces using these commands:
mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/ - This automatically begins sending trace files to the new location; no further configuration is needed.
Option B
Delete or move older vsantraces files from /vsantraces/on the ramdisk.
- Navigate to the vsantraces folder by running
cd var/log/vsantraces - List the contents of the folder by running ls -ahl
ls -ahl-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--YYYY-MM-DDT07h56m21s853.gz-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--YYYY-MM-DDT16h51m11s433.gz-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--YYYY-MM-DDT02h54m10s389.gz-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--YYYY-MM-DDT05h55m47s005.gz-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--YYYY-MM-DDT18h09m27s740.gz-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--YYYY-MM-DDT17h23m25s376.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--YYYY-MM-DDT17h10m06s623.gz-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--YYYY-MM-DDT15h56m16s897.gz-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--YYYY-MM-DDT17h23m25s299.gz-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--YYYY-MM-DDT13h09m23s998.gz-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--YYYY-MM-DDT17h23m25s333.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--YYYY-MM-DDT12h01m34s418.gz-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--YYYY-MM-DDT10h11m46s702.gz-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index - Delete or move the old vsantrace files.
Deletionrm vsantraces*20##-05*.gzrm vsanObserver--20##-05*.gzrm vsantracesUrgent--20##-05*.gz
Move vSAN traces to a new location:
mv vsantraces*20##-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--20##-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--20##-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--20##-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--20##-05*.gz /<new-location-of-vsantraces>
Note: Please engage Broadcom Support if you are unsure whether to delete the files or move them to a new location.
Additional Information
VMware ESXi 7.0.3 build-23307199 , the generated file size was of 16 MB after using the commad
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10We had to use the command, which is specified for ESXi 8.x version, to limit vsantraces as per the desired limit.
esxcli vsan trace set --urgentnumfiles=4 --urgentsize=8 --numfiles=10 --size=8" Feedback
Yes
No
Powered by
