vCenter is repeatedly reporting "The ramdisk 'vsantraces' is full"

Products

• VMware vSAN

Issue/Introduction

Symptoms:

The following event of failure is logged by the vCenter, repetitively:

• The ramdisk 'vsantraces' is full. As a result the file /vsantraces/vsantraces--####-##-##T##h##m##s###--########-####.####.####-############.zst could not be written.
  
  
• When executing the vdf -h command via SSH on the affected ESXi host, the vsantraceFailover ramdisk is reported at 100% capacity (300 MB Used / 300 MB Size).
  
  

Environment

VMware vSAN 8.x
VMware vSAN 9.1

Cause

• If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
• The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
• Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.

Resolution

To resolve the issue, configure a persistent scratch location and follow steps from this article to resolve the issue The ramdisk 'vsantraces' is full

If a dedicated VMFS or NFS datastore is unavailable, you can redirect the standard and urgent vSAN traces to the Scratch directory within the ESXi OSData partition. To configure this directory and limit the total trace size to 200 MB, perform the following steps:

1.  Create a directory named vsantraces in scratch
   mkdir -p /scratch/vsantraces
   
   
2. Set vsantraces to the /scratch/vsantraces directory
   esxcli vsan trace set -p /scratch/vsantraces
   
   
3. Run the following command
   
   esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
   
   This would set the max number of files to '10' and the file size to 20 MB. With these settings, the vSAN trace logs would only grow to a maximum of 200 MB
   
   Note: If File Services is enabled in vSAN, these files will grow upto the double of what's defined i.e., 200*2=400MB

To apply this configuration across multiple hosts, execute the following command on each ESXi host:

mkdir -p /scratch/vsantraces;esxcli vsan trace set -p /scratch/vsantraces;esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10

Additional Information

To check whether scratch is contained in the OSData Partition, take the following steps:

1. To obtain path to the partition on which Scratch is configured, run the following command:
   vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation 
   
   Sample Output:
   [root@esx###:~] vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation 
(vim.option.OptionValue) [
 (vim.option.OptionValue) { 
 key " ScratchConfig . ConfiguredScratchLocation" , 
 value = "/vmfs/volumes/########-########-####-############"

2. Verify that the label of the partition starts with OSDATA, by running the following command:
   vmkfstools -Ph <path to partition from step 1>
   
   Sample Output:
   [root@esx###:~] vmkfstools -Ph /vmfs/volumes/########-########-####-############
VMFSOS-6.82 (Raw Major Version: 24) file system spanning 1 partitions.
File system label (if any): OSDATA-########-########-####-############
Mode: public
Capacity 23.8 GB, 21.6 GB available, file block size 1 MB, max supported file size 64 TB
Media type: Local HDD
Disk Block Size: 512/512/0
UUID: ########-########-####-############
Partitions spanned (on "lvm"):
            naa.#########################
Is Native Snapshot Capable: NO