Error "The ramdisk 'vsantraces' is full. As a result the file /vsantraces/vsantraces-########-####.####.####-############ could not be written".

Products

VMware vSAN

Issue/Introduction

The vsantraces files are used for vSAN diagnostics, so if vsantraces stops logging due to no space, then historical data is lost for troubleshooting vSAN issues..
Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails.
Trying to view the cluster's vSAN services,
- vSAN Health in vCenter fail to load and/or Virtual Objects, will see an error"The request is rejected due to many pending requests"
- vSAN Health error "Unable to query vSAN Health information. Check vSphere Client logs for details"
- vSAN Service error "Unable to retrieve the cluster configuration. Check vSphere client logs for details"

/var/log/vsantraced.log from the ESXi host

YYYY-MM-DDTHH:MM:SS.252Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS253Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory

/var/log/vobd.log from the ESXi host,

[VisorfsCorrelator] 98549997090us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantraces--YYYY-MM-DDThhmmss166.gz because its ramdisk (vsantraces) is full.
[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--YYYY-MM-DDThhmmss166.gz could not be written.

Execute the following command to validate the usage of the vsantraced partition.

vdf -h

-----
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        2M       29M   8% --
etc                        28M      232K       27M   0% --
opt                        32M        0B       32M   0% --
var                        48M      516K       47M   1% --
tmp                       256M       12K      255M   0% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
hostdstats                114M        2M      111M   1% --
vsantraces                300M      287M       12M  95% --

vsantraces ramdisk appears to be full or nearly full on the ESXi:

Environment

7.x, 8.x

Cause

If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.

Resolution

Option A : Redirect vsantraces to presistent scratch location using these steps below.

Connect to each ESXi server in the cluster using SSH.
Get the current vsantrace config by running esxcli vsan trace get

vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true
Redirect vsantraces to new location using the below command

    esxcli vsan trace set

    Cmd options:

        -f|--numfiles: Log file rotation for vSAN trace files.
        -p|--path: Path to store vSAN trace files.
        -r|--reset: When set to true, reset defaults for VSAN trace files.
        -s|--size: Maximum size of vSAN trace files in MB.

Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.

If the Scratch partition is been configured already, vsantraces can be redirected to it. There may be unique folders for each host, create a folder called vsantraces, to separate out the traces per-Host.
If not, refer to Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696 to create a folder specific to each host in the cluster.
Once there is a folder for each Host and the logging is separated, redirect traces using these commands:

mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
This automatically begins sending trace files to the new location, no further configuration is needed.

Option B

Delete or move older vsantraces files from /vsantraces/ located on the ramdisk

Navigate to the vsantraces folder by running cd var/log/vsantraces
List the contents of the folder by running ls -ahl

[root@esx:/vsantraces] ls -ahl

-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--YYYY-MM-DDT07h56m21s853.gz
-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--YYYY-MM-DDT16h51m11s433.gz
-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--YYYY-MM-DDT02h54m10s389.gz
-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--YYYY-MM-DDT05h55m47s005.gz
-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--YYYY-MM-DDT18h09m27s740.gz
-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index
-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--YYYY-MM-DDT17h23m25s376.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index
-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--YYYY-MM-DDT17h10m06s623.gz
-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--YYYY-MM-DDT15h56m16s897.gz
-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index
-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--YYYY-MM-DDT17h23m25s299.gz
-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--YYYY-MM-DDT13h09m23s998.gz
-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index
-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--YYYY-MM-DDT17h23m25s333.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index
-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--YYYY-MM-DDT12h01m34s418.gz
-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--YYYY-MM-DDT10h11m46s702.gz
-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index
Delete or move the old vsantrace files

Deletion

rm vsantraces*20##-05*.gz
rm vsanObserver--20##-05*.gz
rm vsantracesUrgent--20##-05*.gz

Move vSAN traces to a new location:

mv vsantraces*20##-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--20##-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--20##-05*.gz /<new-location-of-vsantraces>

Note: Please engage Broadcom Support if you are not certain about deleting the files or moving it to a new location.

Workaround:

VMware recommends to set the desired size of retaining vSAN traces around 600 MB.
But additional storage is not available, limit the size of vsantraces to 200MB to allow a bit of overhead for additional files and avoid the the ramdisk becoming full.

- If you are using vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:
  
  esxcli vsan trace set -f 10 -s 20
- If you are uisng vSAN 7.x and this can be done by running this command on all vSAN hosts in the cluster:
  
  esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
- If you are using vSAN 8.x this can be done by running this command on all vSAN hosts in the cluster:
  
  esxcli vsan trace set --urgentnumfiles=4 --urgentsize=8 --numfiles=10 --size=8

These changes are immediate and existing files will be rolled over as newly created vSANtrace files are created over time.
No services require a restart to enable this change.

Additional Information

VMware ESXi 7.0.3 build-23307199 , the generated file size was of 16 MB after using the commad "esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10".

We had to use the command "esxcli vsan trace set --urgentnumfiles=4 --urgentsize=8 --numfiles=10 --size=8" which is specified for ESXi 8.x version to limit vsantraces as per the desired limit.