vSAN -- Error: The ramdisk 'vsantraces' is full on ESXi

Products

VMware vSAN

Issue/Introduction

vSAN diagnostic traces are stored on a ramdisk by default.

If a persistent scratch location is not configured, these traces can exhaust the 300MB ramdisk capacity, leading to logging failures and Aria Operations alarms.

A.) Experiencing one or more of the following symptoms:

Error: vsantraceFailover partition full

vSAN diagnostic logging stops, resulting in loss of historical troubleshooting data
Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails
vSphere Client: vSAN Virtual Objects fails to load with error "The request is rejected due to many pending requests"
vSphere Client: vSAN Health / Healthcheck Error: "Unable to query vSAN Health information. Check vSphere Client logs for details."
vSphere Client: vSAN Service error: "Unable to retrieve the cluster configuration. Check vSphere client logs for details."
vSphere Client: Events do show error "vsantraces is full"
Aria Operations: Error/Alarm: "ESXi: RAM disk / inode table is full"

On the ESXi Host: /var/log/vsantraced.log shows one or both of the following messages:

YYYY-MM-DDTHH:MM:SS.ZZ Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS.ZZ Er(11) chardevlogger[67289337]: Failed to write during flush: No space left on device

On the ESXi Host: /var/log/vobd.log shows one or both of the following messages:

[vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantracesUrgent-####-####-###-####.zst because its ramdisk (vsantraces) is full.

   [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantracesIODiag-####.zst because its ramdisk (vsantraces) is full.

On a ESX Host running vSAN/vSphere 9.1: /var/log/vmkwarning.log shows:

WARNING vmkwarning - [esx@####] cpu###:######) VisorFSRam: 251: Cannot extend visorfs file /vsantraces/vsantracesLSOMVerbose--####.zst because its ramdisk (vsantraceFailover) is full.

B.) Via SSH/Putty: Confirm the space issue by checking the related partition via command "vdf -h" :

ESXi/vSAN Version 7.x or 8.x: vsantraces

Example output:

vdf -h

Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        2M       29M   8% --
etc                        28M      232K       27M   0% --
opt                        32M        0B       32M   0% --
var                        48M      516K       47M   1% --
tmp                       256M       12K      255M   0% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
hostdstats                114M        2M      111M   1% --
vsantraces                300M      287M       12M  95% --    >>>>>> Partition is nearly full

ESXi/vSAN Version 9.1: vsantraceFailover

Example output:

vdf -h

Ramdisk Size Used Available Use% Mounted on
root 32M 8M 23M 27% --
etc 28M 3M 24M 11% --
opt 32M 0B 32M 0% --
var 48M 952K 47M 1% --
tmp 256M 4K 255M 0% --
memfd 1024M 0B 1024M 0% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
crx 1024M 0B 1024M 0% --
configstore 32M 416K 31M 1% --
configstorebkp 32M 388K 31M 1% --
vsantraceFailover 300M 300M 0B 100% -- >>>>>> Partition is full
hostdstats 2053M 8M 2044M 0%

Environment

VMware vSAN 8.x

VMware vSAN 9.x

VMware ESXi 8.x

VMware ESXi 9.x

VCF 9.1
VMware Cloud Foundation 9.1

Cause

If no persistent scratch location is configured, these traces exhaust the 300MB ramdisk capacity. High logging can accelerate this exhaustion.

Resolution

VMware recommends setting the desired size of retaining vSAN traces to around 600 MB.

Option A: If no additional space is available other than the local ramdisk:

But if additional storage is not available, limit vsantraces to 200MB to allow a bit of overhead for additional files and avoid the ramdisk from becoming full,
by running the following command via SSH/Putty on all vSAN Hosts in the Cluster:

ESXi/vSAN Version 8.x: esxcli vsan trace set --urgentnumfiles=4 --urgentsize=8 --numfiles=10 --size=8
ESXi/vSAN Version 7.x: esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10

These changes are immediate, and existing files will be rolled over as newly created vSANtrace files are created over time. No restart of any services/reboot required.

Option B: If additional space is available other than the local ramdisk and no persistent scratch location exists

Refer to Creating a persistent scratch location for ESXi 8.x/7.x to create a folder for each vSAN Host in the Cluster

Once there is a folder for each Host and the logging is separated, redirect traces using these commands:

mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/

This automatically begins sending trace files to the new location; no further configuration is needed.

Option C: Redirect vsantraces to an existing persistent scratch location:

By running the following command via SSH/Putty on all vSAN Hosts in the Cluster:

1.) Get the current vsantrace config by running. esxcli vsan trace get

vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true

2.) Redirect vsantraces to the new location using the below command.

esxcli vsan trace set

Cmd options:

        -f|--numfiles: Log file rotation for vSAN trace files.
        -p|--path: Path to store vSAN trace files.
        -r|--reset: When set to true, reset defaults for VSAN trace files.
        -s|--size: Maximum size of vSAN trace files in MB.

In regards to existing persistent Scratch Location:

Log bundles should be placed in their own host folders; otherwise, once the first Host acquires locks on the scratch location, it prevents the other hosts in the cluster from logging
To separate traces per Host: Consider creating a folder for each Host called vsantraces

Option D: Free up space in the affected partition (depending on version: vsantraces or vsantraceFailover)

By deleting and/or moving older vsantraces files:

1.) Navigate to the vsantraces folder by running
cd var/log/vsantraces

2.) List the contents of the folder by running

ls -ahl

Example Output:
-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--YYYY-MM-DDTHHMMs853.gz
-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--YYYY-MM-DDTHHMMs433.gz
-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--YYYY-MM-DDTHHMMs389.gz
-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--YYYY-MM-DDTHHMMs005.gz
-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--YYYY-MM-DDTHHMMs740.gz
-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--YYYY-MM-DDTHHMMs418.gz
-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--YYYY-MM-DDTHHMMs702.gz
-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index

3.) Delete or move the old vsantrace files.

Example: Deletion:

rm vsantraces*20##-05*.gz
rm vsanObserver--20##-05*.gz
rm vsantracesUrgent--20##-05*.gz

Example: Move of files to new location:

mv vsantraces*20####-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--20####-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--20####-05*.gz /<new-location-of-vsantraces>

If assistance is required, open a Ticket with VMware by Broadcom Support.