The ramdisk 'vsantraces' is full
Article ID: 326990
Updated On:
Products
Issue/Introduction
- The ESXi host reports not responding in vCenter Server.
- Memory leak events are reported in the vsantraced log.
- Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails.
Validation Steps:
- Review the /var/log/vsantraced.log from the ESXi host. The following entries will be identified
YYYY-MM-DDTHH:MM:SS.252Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS253Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS.253Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS.254Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory
- Review the /var/log/vobd.log from the ESXi host, you see the entries similar to:
[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--YYYY-MM-DDThhmmss166.gz could not be written.
- Execute the following command to validate the usage of the vsantraced partition.
vdf -h
-----
Ramdisk Size Used Available Use% Mounted on
root 32M 2M 29M 8% --
etc 28M 232K 27M 0% --
opt 32M 0B 32M 0% --
var 48M 516K 47M 1% --
tmp 256M 12K 255M 0% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
hostdstats 114M 2M 111M 1% --
vsantraces 300M 287M 12M 95% --vsantraces ramdisk appears to be full or nearly full on the ESXi:
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware vSAN 8.x
Cause
- If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
- The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
- Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.
Resolution
It is possible in vSAN that the ramdisk for vSAN trace information may grow and fill the allotted ramdisk. vsantraces can be redirected to another location for logging.
vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.
If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.
1) Connect to each ESXi server in the cluster using SSH.
2) Get the current vsantrace config by running esxcli vsan trace get
You see output similar to:
vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true
3) To configure vsantraces to log elsewhere, run this command:
[root@localhost:~] esxcli vsan trace set Usage: esxcli vsan trace set [cmd options]
Note: This command is not thread safe.
Cmd options:
-f|--numfiles: Log file rotation for vSAN trace files.
-p|--path: Path to store vSAN trace files.
-r|--reset: When set to true, reset defaults for VSAN trace files.
-s|--size: Maximum size of vSAN trace files in MB.
If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)
Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.
After we have a folder for each Host and the logging is separated, we can redirect traces using these commands:
mkdir /vsantraces
esxcli vsan trace set -f 10 -s 60 -p /vsantraces
For example:
If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands:
mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/ esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
Note: The preceding commands are just an example. The location may vary based on the environment.
This automatically begins sending trace files to the new location, no further configuration is needed.
Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk
1) Navigate to the vsantraces folder by running cd var/log/vsantraces
2) List the contents of the folder by running ls-ahl
Sample output:
[root@esx:/vsantraces] ls -ahltotal 264512drwxr-xr-x 1 root root 512 Jun 4 20:04 .drwxr-xr-x 1 root root 512 Jun 4 20:16 ..-rw-r--r-- 1 root root 254.9K Jun 4 17:44 vsanObserver--YYYY-MM-DDT17h24m00s.gz-rw-r--r-- 1 root root 254.7K Jun 4 18:06 vsanObserver--YYYY-MM-DDT17h46m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 18:28 vsanObserver--YYYY-MM-DDT18h08m00s.gz-rw-r--r-- 1 root root 254.8K Jun 4 18:50 vsanObserver--YYYY-MM-DDT18h30m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 19:12 vsanObserver--YYYY-MM-DDT18h52m00s.gz-rw-r--r-- 1 root root 254.6K Jun 4 19:34 vsanObserver--YYYY-MM-DDT19h14m00s.gz-rw-r--r-- 1 root root 254.7K Jun 4 19:56 vsanObserver--YYYY-MM-DDT19h36m00s.gz-rw-r--r-- 1 root root 208.5K Jun 4 20:16 vsanObserver--YYYY-MM-DDT19h58m01s.gz-rw-r--r-- 1 root root 20.0M Jun 2 07:56 vsantraces--YYYY-MM-DDT23h01m33s187.gz-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--YYYY-MM-DDT07h56m21s853.gz-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--YYYY-MM-DDT16h51m11s433.gz-rw-r--r-- 1 root root 20.0M Jun 3 10:45 vsantraces--YYYY-MM-DDT01h48m01s609.gz-rw-r--r-- 1 root root 20.0M Jun 3 19:39 vsantraces--YYYY-MM-DDT10h45m44s438.gz-rw-r--r-- 1 root root 20.0M Jun 4 04:28 vsantraces--YYYY-MM-DDT19h39m32s771.gz-rw-r--r-- 1 root root 20.0M Jun 4 13:24 vsantraces--YYYY-MM-DDT04h28m00s889.gz-rw-r--r-- 1 root root 15.4M Jun 4 20:17 vsantraces--YYYY-MM-DDT13h24m46s788.gz-rw-r--r-- 1 root root 192 Jun 4 13:24 vsantraces.index-rw-r--r-- 1 root root 8.0M May 15 03:33 vsantracesClom--YYYY-MM-DDT15h15m00s791.gz-rw-r--r-- 1 root root 8.0M May 18 15:48 vsantracesClom--YYYY-MM-DDT03h33m29s948.gz-rw-r--r-- 1 root root 8.0M May 22 04:02 vsantracesClom--YYYY-MM-DDT15h48m59s091.gz-rw-r--r-- 1 root root 8.0M May 25 15:50 vsantracesClom--YYYY-MM-DDT04h02m55s293.gz-rw-r--r-- 1 root root 8.0M May 29 02:54 vsantracesClom--YYYY-MM-DDT15h50m08s307.gz-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--YYYY-MM-DDT02h54m10s389.gz-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--YYYY-MM-DDT05h55m47s005.gz-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--YYYY-MM-DDT18h09m27s740.gz-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--YYYY-MM-DDT17h23m25s376.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index-rw-r--r-- 1 root root 1.0M Jun 3 17:10 vsantracesIODiag--YYYY-MM-DDT18h31m21s576.gz-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--YYYY-MM-DDT17h10m06s623.gz-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--YYYY-MM-DDT15h56m16s897.gz-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--YYYY-MM-DDT17h23m25s299.gz-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--YYYY-MM-DDT13h09m23s998.gz-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--YYYY-MM-DDT17h23m25s333.gz-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index-rw-r--r-- 1 root root 10.0M May 29 00:22 vsantracesUrgent--YYYY-MM-DDT05h05m25s105.gz-rw-r--r-- 1 root root 10.0M May 31 12:01 vsantracesUrgent--YYYY-MM-DDT00h22m35s242.gz-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--YYYY-MM-DDT12h01m34s418.gz-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--YYYY-MM-DDT10h11m46s702.gz-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.indexrm vsantraces*20##-05*.gzrm vsanObserver--20##-05*.gzrm vsantracesUrgent--20##-05*.gz
mv vsanObserver--20##-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--20##-05*.gz /<new-location-of-vsantraces>
Note: If you're not comfortable performing the delete/move of older vsantraces logs engage vSAN support for assistance.
Workaround:
If no additional storage is available, limit the size of vsantraces to 200MB. This allows a bit of overhead for additional files and ensures that we do not fill the ramdisk.
For vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set -f 10 -s 20
This would make the max files setting '10' and the file size setting 20 MBs. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
For vSAN 7.x and higher this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
This would make the max files setting '10' and the file size setting 10 MBs for both the vsantraces and the urgenttraces. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
These changes are immediate and older files will be rolled off as smaller ones take their places over time.
No services require a restart to enable this change.
Delete older vsantraces files from the vsantraces folder /vsantraces/ as per the Resolution section.
Additional Information
- The ramdisk gets full which could lead to a hung host or the host becoming non-responsive in vCenter
- The vsantraces files are used for vSAN diagnostics so if vsantraces stops logging due to no space then historical data is lost for troubleshooting vSAN issues.
Feedback
