The ramdisk 'vsantraces' is full
search cancel

The ramdisk 'vsantraces' is full

book

Article ID: 326990

calendar_today

Updated On:

Products

VMware vSAN

Issue/Introduction

Symptoms:
  • The ESXi host reports not responding in vCenter Server
  • Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails.
  • In the vobd.log file, you see the entries similar to:
[VisorfsCorrelator] 98549997090us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz because its ramdisk (vsantraces) is full.
[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz could not be written.
  • vsantraces ramdisk appears to be full or nearly full on the ESXi:
vdf -h
-----
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        2M       29M   8% --
etc                        28M      232K       27M   0% --
opt                        32M        0B       32M   0% --
var                        48M      516K       47M   1% --
tmp                       256M       12K      255M   0% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
hostdstats                114M        2M      111M   1% --
vsantraces                300M      287M       12M  95% --

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vSAN 6.2.x
VMware vSAN 6.5.x
VMware vSAN 6.1.x
VMware vSAN 6.7.x
VMware vSAN 6.6.x

Cause

  • If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
  • The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
  • Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.

Resolution

It is possible in vSAN that the ramdisk for vSAN trace information may grow and fill the allotted ramdisk. vsantraces can be redirected to another location for logging.

vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.

If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.

1) Connect to each ESXi server in the cluster using SSH.
2) Get the current vsantrace config by running esxcli vsan trace for vSAN 6.7 and lower

You see output similar to:

vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true

For vSAN 7.x and higher run cat /etc/vmware/vsan/vsantraced.conf | grep -v "#"

You see output similar to:
VSANTRACED_ROTATE_MAX_FILES=10
VSANTRACED_ROTATE_FILE_SIZE=8
VSANTRACED_URGENT_ROTATE_MAX_FILES=4
VSANTRACED_URGENT_ROTATE_FILE_SIZE=8
VSANTRACED_LAST_SELECTED_VOLUME="/vsantraces"
VSANOBSERVER_MAX_MB_SIZE="38"

3) To configure vsantraces to log elsewhere, run this command:

    [root@localhost:~] esxcli vsan trace set
    Usage: esxcli vsan trace set [cmd options]

    Note: This command is not thread safe.

    Cmd options:

        -f|--numfiles: Log file rotation for vSAN trace files.
        -p|--path: Path to store vSAN trace files.
        -r|--reset: When set to true, reset defaults for VSAN trace files.
        -s|--size: Maximum size of vSAN trace files in MB.

If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)

Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.

After we have a folder for each Host and the logging is separated, we can redirect traces using these commands:

    mkdir /vsantraces
    esxcli vsan trace set -f 10 -s 60 -p /vsantraces

For example:

If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands:

    mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
    esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/

Note: The preceding commands are just an example. The location may vary based on the environment.

This automatically begins sending trace files to the new location, no further configuration is needed.

Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk

1) Navigate to the vsantraces folder by running cd var/log/vsantraces
2) List the contents of the folder by running ls-ahl

Sample output:
[root@esxi1:/vsantraces] ls -ahl
total 264512
drwxr-xr-x    1 root     root         512 Jun  4 20:04 .
drwxr-xr-x    1 root     root         512 Jun  4 20:16 ..
-rw-r--r--    1 root     root      254.8K Jun  4 17:22 vsanObserver--2023-06-04T17h02m00s.gz
-rw-r--r--    1 root     root      254.9K Jun  4 17:44 vsanObserver--2023-06-04T17h24m00s.gz
-rw-r--r--    1 root     root      254.7K Jun  4 18:06 vsanObserver--2023-06-04T17h46m00s.gz
-rw-r--r--    1 root     root      254.6K Jun  4 18:28 vsanObserver--2023-06-04T18h08m00s.gz
-rw-r--r--    1 root     root      254.8K Jun  4 18:50 vsanObserver--2023-06-04T18h30m00s.gz
-rw-r--r--    1 root     root      254.6K Jun  4 19:12 vsanObserver--2023-06-04T18h52m00s.gz
-rw-r--r--    1 root     root      254.6K Jun  4 19:34 vsanObserver--2023-06-04T19h14m00s.gz
-rw-r--r--    1 root     root      254.7K Jun  4 19:56 vsanObserver--2023-06-04T19h36m00s.gz
-rw-r--r--    1 root     root      208.5K Jun  4 20:16 vsanObserver--2023-06-04T19h58m01s.gz
-rw-r--r--    1 root     root       20.0M Jun  2 07:56 vsantraces--2023-06-01T23h01m33s187.gz
-rw-r--r--    1 root     root       20.0M Jun  2 16:51 vsantraces--2023-06-02T07h56m21s853.gz
-rw-r--r--    1 root     root       20.0M Jun  3 01:48 vsantraces--2023-06-02T16h51m11s433.gz
-rw-r--r--    1 root     root       20.0M Jun  3 10:45 vsantraces--2023-06-03T01h48m01s609.gz
-rw-r--r--    1 root     root       20.0M Jun  3 19:39 vsantraces--2023-06-03T10h45m44s438.gz
-rw-r--r--    1 root     root       20.0M Jun  4 04:28 vsantraces--2023-06-03T19h39m32s771.gz
-rw-r--r--    1 root     root       20.0M Jun  4 13:24 vsantraces--2023-06-04T04h28m00s889.gz
-rw-r--r--    1 root     root       15.4M Jun  4 20:17 vsantraces--2023-06-04T13h24m46s788.gz
-rw-r--r--    1 root     root         192 Jun  4 13:24 vsantraces.index
-rw-r--r--    1 root     root        8.0M May 15 03:33 vsantracesClom--2023-05-11T15h15m00s791.gz
-rw-r--r--    1 root     root        8.0M May 18 15:48 vsantracesClom--2023-05-15T03h33m29s948.gz
-rw-r--r--    1 root     root        8.0M May 22 04:02 vsantracesClom--2023-05-18T15h48m59s091.gz
-rw-r--r--    1 root     root        8.0M May 25 15:50 vsantracesClom--2023-05-22T04h02m55s293.gz
-rw-r--r--    1 root     root        8.0M May 29 02:54 vsantracesClom--2023-05-25T15h50m08s307.gz
-rw-r--r--    1 root     root        8.0M Jun  1 05:55 vsantracesClom--2023-05-29T02h54m10s389.gz
-rw-r--r--    1 root     root        8.0M Jun  4 18:09 vsantracesClom--2023-06-01T05h55m47s005.gz
-rw-r--r--    1 root     root      206.3K Jun  4 20:16 vsantracesClom--2023-06-04T18h09m27s740.gz
-rw-r--r--    1 root     root         192 Jun  4 18:09 vsantracesClom.index
-rw-r--r--    1 root     root      620.8K Jun  4 03:29 vsantracesDOMObj--2023-03-13T17h23m25s376.gz
-rw-r--r--    1 root     root          24 Mar 13 17:23 vsantracesDOMObj.index
-rw-r--r--    1 root     root        1.0M Jun  3 17:10 vsantracesIODiag--2023-06-02T18h31m21s576.gz
-rw-r--r--    1 root     root        1.0M Jun  4 15:56 vsantracesIODiag--2023-06-03T17h10m06s623.gz
-rw-r--r--    1 root     root      195.4K Jun  4 20:17 vsantracesIODiag--2023-06-04T15h56m16s897.gz
-rw-r--r--    1 root     root          72 Jun  4 15:56 vsantracesIODiag.index
-rw-r--r--    1 root     root        2.0M May 17 13:09 vsantracesLSOM--2023-03-13T17h23m25s299.gz
-rw-r--r--    1 root     root      575.1K Jun  4 20:11 vsantracesLSOM--2023-05-17T13h09m23s998.gz
-rw-r--r--    1 root     root          48 May 17 13:09 vsantracesLSOM.index
-rw-r--r--    1 root     root          15 Mar 13 17:23 vsantracesLSOMVerbose--2023-03-13T17h23m25s333.gz
-rw-r--r--    1 root     root          24 Mar 13 17:23 vsantracesLSOMVerbose.index
-rw-r--r--    1 root     root       10.0M May 29 00:22 vsantracesUrgent--2023-05-25T05h05m25s105.gz
-rw-r--r--    1 root     root       10.0M May 31 12:01 vsantracesUrgent--2023-05-29T00h22m35s242.gz
-rw-r--r--    1 root     root       10.0M Jun  4 10:11 vsantracesUrgent--2023-05-31T12h01m34s418.gz
-rw-r--r--    1 root     root        1.0M Jun  4 20:17 vsantracesUrgent--2023-06-04T10h11m46s702.gz
-rw-r--r--    1 root     root          96 Jun  4 10:11 vsantracesUrgent.index

3) To delete older files run the following:
rm vsantraces*2023-05*.gz
rm vsanObserver--2023-05*.gz
rm vsantracesUrgent--2023-05*.gz


4) To move them run the following:
mv vsantraces*2023-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--2023-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--2023-05*.gz /<new-location-of-vsantraces>

Note: If you're not comfortable performing the delete/move of older vsantraces logs engage vSAN support for assistance.


Workaround:
If no additional storage is available, limit the size of vsantraces to 200MB. This allows a bit of overhead for additional files and ensures that we do not fill the ramdisk.

For vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:

esxcli vsan trace set -f 10 -s 20
This would make the max files setting '10' and the file size setting 20 MBs. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs

For vSAN 7.x and higher this can be done by running this command on all vSAN hosts in the cluster:

esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
This would make the max files setting '10' and the file size setting 10 MBs for both the vsantraces and the urgenttraces. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs

These changes are immediate and older files will be rolled off as smaller ones take their places over time.

No services require a restart to enable this change.

Delete older vsantraces files from the vsantraces folder /vsantraces/ as per the Resolution section.

Additional Information

Impact/Risks:
  • The ramdisk gets full which could lead to a hung host or the host becoming non-responsive in vCenter
  • The vsantraces files are used for vSAN diagnostics so if vsantraces stops logging due to no space then historical data is lost for troubleshooting vSAN issues.