vCenter is repeatedly reporting "The ramdisk 'vsantraces' is full"

Products

• VMware vSAN

Issue/Introduction

The following event of failure is logged by the vCenter, repetitively:
The ramdisk 'vsantraces' is full. As a result the file /vsantraces/vsantraces--####-##-##T##h##m##s###--########-####.####.####-############.zst could not be written.

Environment

vSAN
ESXi 7.0 and later

Cause

• If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
• The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
• Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.

Resolution

To resolve the issue, configure a persistent scratch location and follow steps from this article to resolve the issue The ramdisk 'vsantraces' is full

If a dedicated VMFS/NFS datastore cannot be created, we may create a directory and move redirect vsantraces and urgenttraces to Scratch in OSData partition of the ESXi and set it to grow to a maximum of 200MB by taking the following steps:

1.  Create a directory named vsantraces in scratch
   mkdir -p /scratch/vsantraces
2. Set vsantraces to the /scratch/vsantraces directory
   esxcli vsan trace set -p /scratch/vsantraces
   
   
3. Run the following command
   
   esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
   This would set the max number of files to '10' and the file size to 20 MB. With these settings, the vSAN trace logs would only grow to a maximum of 200 MB
   Note: If File Services is enabled in vSAN, these files will grow upto the double of what's defined i.e., 200*2=400MB

If there are many hosts that you need to make this change on, you may run the following command on each host:
mkdir -p /scratch/vsantraces;esxcli vsan trace set -p /scratch/vsantraces;esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10

Additional Information

To check whether scratch is contained in the OSData Partition, take the following steps:

1. To obtain path to the partition on which Scratch is configured, run the following command:
   vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation 
   
   Sample Output:
   [root@esx###:~] vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation 
(vim.option.OptionValue) [
 (vim.option.OptionValue) { 
 key " ScratchConfig . ConfiguredScratchLocation" , 
 value = "/vmfs/volumes/########-########-####-############"

2. Verify that the label of the partition starts with OSDATA, by running the following command:
   vmkfstools -Ph <path to partition from step 1>
   
   Sample Output:
   [root@esx###:~] vmkfstools -Ph /vmfs/volumes/########-########-####-############
VMFSOS-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): OSDATA-########-########-####-############
Mode: public
Capacity 23.8 GB, 21.6 GB available, file block size 1 MB, max supported file size 64 TB
Media type: Local HDD
Disk Block Size: 512/512/0
UUID: ########-########-####-############
Partitions spanned (on "lvm"):
            naa.#########################
Is Native Snapshot Capable: NO