The ramdisk 'vsantraces' is full
Article ID: 326990
Updated On:
Products
VMware vSAN
Issue/Introduction
Symptoms:
- The ESXi host reports not responding in vCenter Server
- Logging in to the vSAN host through Log Insight or through the vSphere Web Client fails.
- In the vobd.log file, you see the entries similar to:
[VisorfsCorrelator] 98549997090us: [vob.visorfs.ramdisk.full] Cannot extend visorfs file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz because its ramdisk (vsantraces) is full.
[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz could not be written.
[VisorfsCorrelator] 98548366158us: [esx.problem.visorfs.ramdisk.full] The ramdisk 'vsantraces' is full. As a result, the file /vsantraces/vsantraces--2016-11-13T19h09m03s166.gz could not be written.
- vsantraces ramdisk appears to be full or nearly full on the ESXi:
vdf -h
----- Ramdisk Size Used Available Use% Mounted on root 32M 2M 29M 8% -- etc 28M 232K 27M 0% -- opt 32M 0B 32M 0% -- var 48M 516K 47M 1% -- tmp 256M 12K 255M 0% -- iofilters 32M 0B 32M 0% -- shm 1024M 0B 1024M 0% -- hostdstats 114M 2M 111M 1% -- vsantraces 300M 287M 12M 95% --
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware vSAN
Cause
- If no scratch location is configured for vsantraces files, the traces are stored on the ramdisk.
- The default ramdisk size for vsantraces is 300MB. When the usage reaches around 95%, additional files fail to be created.
- Under certain conditions, some vsantrace entries may get logged too frequently, eventually exhausting the capacity of the volume holding the trace files.
Resolution
It is possible in vSAN that the ramdisk for vSAN trace information may grow and fill the allotted ramdisk. vsantraces can be redirected to another location for logging.
vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.
If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.
1) Connect to each ESXi server in the cluster using SSH.
2) Get the current vsantrace config by running esxcli vsan trace get
You see output similar to:
vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true
3) To configure vsantraces to log elsewhere, run this command:
[root@localhost:~] esxcli vsan trace set
Usage: esxcli vsan trace set [cmd options]
Note: This command is not thread safe.
Cmd options:
-f|--numfiles: Log file rotation for vSAN trace files.
-p|--path: Path to store vSAN trace files.
-r|--reset: When set to true, reset defaults for VSAN trace files.
-s|--size: Maximum size of vSAN trace files in MB.
If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)
Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.
After we have a folder for each Host and the logging is separated, we can redirect traces using these commands:
mkdir /vsantraces
esxcli vsan trace set -f 10 -s 60 -p /vsantraces
For example:
If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands:
mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
Note: The preceding commands are just an example. The location may vary based on the environment.
This automatically begins sending trace files to the new location, no further configuration is needed.
Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk
1) Navigate to the vsantraces folder by running cd var/log/vsantraces
2) List the contents of the folder by running ls-ahl
Sample output:
[root@esxi1:/vsantraces] ls -ahl
total 264512
drwxr-xr-x 1 root root 512 Jun 4 20:04 .
drwxr-xr-x 1 root root 512 Jun 4 20:16 ..
-rw-r--r-- 1 root root 254.8K Jun 4 17:22 vsanObserver--2023-06-04T17h02m00s.gz
-rw-r--r-- 1 root root 254.9K Jun 4 17:44 vsanObserver--2023-06-04T17h24m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 18:06 vsanObserver--2023-06-04T17h46m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 18:28 vsanObserver--2023-06-04T18h08m00s.gz
-rw-r--r-- 1 root root 254.8K Jun 4 18:50 vsanObserver--2023-06-04T18h30m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:12 vsanObserver--2023-06-04T18h52m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:34 vsanObserver--2023-06-04T19h14m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 19:56 vsanObserver--2023-06-04T19h36m00s.gz
-rw-r--r-- 1 root root 208.5K Jun 4 20:16 vsanObserver--2023-06-04T19h58m01s.gz
-rw-r--r-- 1 root root 20.0M Jun 2 07:56 vsantraces--2023-06-01T23h01m33s187.gz
-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--2023-06-02T07h56m21s853.gz
-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--2023-06-02T16h51m11s433.gz
-rw-r--r-- 1 root root 20.0M Jun 3 10:45 vsantraces--2023-06-03T01h48m01s609.gz
-rw-r--r-- 1 root root 20.0M Jun 3 19:39 vsantraces--2023-06-03T10h45m44s438.gz
-rw-r--r-- 1 root root 20.0M Jun 4 04:28 vsantraces--2023-06-03T19h39m32s771.gz
-rw-r--r-- 1 root root 20.0M Jun 4 13:24 vsantraces--2023-06-04T04h28m00s889.gz
-rw-r--r-- 1 root root 15.4M Jun 4 20:17 vsantraces--2023-06-04T13h24m46s788.gz
-rw-r--r-- 1 root root 192 Jun 4 13:24 vsantraces.index
-rw-r--r-- 1 root root 8.0M May 15 03:33 vsantracesClom--2023-05-11T15h15m00s791.gz
-rw-r--r-- 1 root root 8.0M May 18 15:48 vsantracesClom--2023-05-15T03h33m29s948.gz
-rw-r--r-- 1 root root 8.0M May 22 04:02 vsantracesClom--2023-05-18T15h48m59s091.gz
-rw-r--r-- 1 root root 8.0M May 25 15:50 vsantracesClom--2023-05-22T04h02m55s293.gz
-rw-r--r-- 1 root root 8.0M May 29 02:54 vsantracesClom--2023-05-25T15h50m08s307.gz
-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--2023-05-29T02h54m10s389.gz
-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--2023-06-01T05h55m47s005.gz
-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--2023-06-04T18h09m27s740.gz
-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index
-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--2023-03-13T17h23m25s376.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index
-rw-r--r-- 1 root root 1.0M Jun 3 17:10 vsantracesIODiag--2023-06-02T18h31m21s576.gz
-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--2023-06-03T17h10m06s623.gz
-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--2023-06-04T15h56m16s897.gz
-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index
-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--2023-03-13T17h23m25s299.gz
-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--2023-05-17T13h09m23s998.gz
-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index
-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--2023-03-13T17h23m25s333.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index
-rw-r--r-- 1 root root 10.0M May 29 00:22 vsantracesUrgent--2023-05-25T05h05m25s105.gz
-rw-r--r-- 1 root root 10.0M May 31 12:01 vsantracesUrgent--2023-05-29T00h22m35s242.gz
-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--2023-05-31T12h01m34s418.gz
-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--2023-06-04T10h11m46s702.gz
-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index
3) To delete older files run the following:
rm vsantraces*2023-05*.gz
rm vsanObserver--2023-05*.gz
rm vsantracesUrgent--2023-05*.gz
4) To move them run the following:
mv vsantraces*2023-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--2023-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--2023-05*.gz /<new-location-of-vsantraces>
Workaround:
If no additional storage is available, limit the size of vsantraces to 200MB. This allows a bit of overhead for additional files and ensures that we do not fill the ramdisk.
For vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set -f 10 -s 20
This would make the max files setting '10' and the file size setting 20 MBs. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
For vSAN 7.x and higher this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
This would make the max files setting '10' and the file size setting 10 MBs for both the vsantraces and the urgenttraces. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
These changes are immediate and older files will be rolled off as smaller ones take their places over time.
No services require a restart to enable this change.
Delete older vsantraces files from the vsantraces folder /vsantraces/ as per the Resolution section.
vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.
If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.
1) Connect to each ESXi server in the cluster using SSH.
2) Get the current vsantrace config by running esxcli vsan trace get
You see output similar to:
vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true
3) To configure vsantraces to log elsewhere, run this command:
[root@localhost:~] esxcli vsan trace set
Usage: esxcli vsan trace set [cmd options]
Note: This command is not thread safe.
Cmd options:
-f|--numfiles: Log file rotation for vSAN trace files.
-p|--path: Path to store vSAN trace files.
-r|--reset: When set to true, reset defaults for VSAN trace files.
-s|--size: Maximum size of vSAN trace files in MB.
If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)
Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.
After we have a folder for each Host and the logging is separated, we can redirect traces using these commands:
mkdir /vsantraces
esxcli vsan trace set -f 10 -s 60 -p /vsantraces
For example:
If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands:
mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
Note: The preceding commands are just an example. The location may vary based on the environment.
This automatically begins sending trace files to the new location, no further configuration is needed.
Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk
1) Navigate to the vsantraces folder by running cd var/log/vsantraces
2) List the contents of the folder by running ls-ahl
Sample output:
[root@esxi1:/vsantraces] ls -ahl
total 264512
drwxr-xr-x 1 root root 512 Jun 4 20:04 .
drwxr-xr-x 1 root root 512 Jun 4 20:16 ..
-rw-r--r-- 1 root root 254.8K Jun 4 17:22 vsanObserver--2023-06-04T17h02m00s.gz
-rw-r--r-- 1 root root 254.9K Jun 4 17:44 vsanObserver--2023-06-04T17h24m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 18:06 vsanObserver--2023-06-04T17h46m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 18:28 vsanObserver--2023-06-04T18h08m00s.gz
-rw-r--r-- 1 root root 254.8K Jun 4 18:50 vsanObserver--2023-06-04T18h30m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:12 vsanObserver--2023-06-04T18h52m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:34 vsanObserver--2023-06-04T19h14m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 19:56 vsanObserver--2023-06-04T19h36m00s.gz
-rw-r--r-- 1 root root 208.5K Jun 4 20:16 vsanObserver--2023-06-04T19h58m01s.gz
-rw-r--r-- 1 root root 20.0M Jun 2 07:56 vsantraces--2023-06-01T23h01m33s187.gz
-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--2023-06-02T07h56m21s853.gz
-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--2023-06-02T16h51m11s433.gz
-rw-r--r-- 1 root root 20.0M Jun 3 10:45 vsantraces--2023-06-03T01h48m01s609.gz
-rw-r--r-- 1 root root 20.0M Jun 3 19:39 vsantraces--2023-06-03T10h45m44s438.gz
-rw-r--r-- 1 root root 20.0M Jun 4 04:28 vsantraces--2023-06-03T19h39m32s771.gz
-rw-r--r-- 1 root root 20.0M Jun 4 13:24 vsantraces--2023-06-04T04h28m00s889.gz
-rw-r--r-- 1 root root 15.4M Jun 4 20:17 vsantraces--2023-06-04T13h24m46s788.gz
-rw-r--r-- 1 root root 192 Jun 4 13:24 vsantraces.index
-rw-r--r-- 1 root root 8.0M May 15 03:33 vsantracesClom--2023-05-11T15h15m00s791.gz
-rw-r--r-- 1 root root 8.0M May 18 15:48 vsantracesClom--2023-05-15T03h33m29s948.gz
-rw-r--r-- 1 root root 8.0M May 22 04:02 vsantracesClom--2023-05-18T15h48m59s091.gz
-rw-r--r-- 1 root root 8.0M May 25 15:50 vsantracesClom--2023-05-22T04h02m55s293.gz
-rw-r--r-- 1 root root 8.0M May 29 02:54 vsantracesClom--2023-05-25T15h50m08s307.gz
-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--2023-05-29T02h54m10s389.gz
-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--2023-06-01T05h55m47s005.gz
-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--2023-06-04T18h09m27s740.gz
-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index
-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--2023-03-13T17h23m25s376.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index
-rw-r--r-- 1 root root 1.0M Jun 3 17:10 vsantracesIODiag--2023-06-02T18h31m21s576.gz
-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--2023-06-03T17h10m06s623.gz
-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--2023-06-04T15h56m16s897.gz
-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index
-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--2023-03-13T17h23m25s299.gz
-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--2023-05-17T13h09m23s998.gz
-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index
-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--2023-03-13T17h23m25s333.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index
-rw-r--r-- 1 root root 10.0M May 29 00:22 vsantracesUrgent--2023-05-25T05h05m25s105.gz
-rw-r--r-- 1 root root 10.0M May 31 12:01 vsantracesUrgent--2023-05-29T00h22m35s242.gz
-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--2023-05-31T12h01m34s418.gz
-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--2023-06-04T10h11m46s702.gz
-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index
3) To delete older files run the following:
rm vsantraces*2023-05*.gz
rm vsanObserver--2023-05*.gz
rm vsantracesUrgent--2023-05*.gz
4) To move them run the following:
mv vsantraces*2023-05*.gz /<new-location-of-vsantraces>
mv vsanObserver--2023-05*.gz /<new-location-of-vsantraces>
mv vsantracesUrgent--2023-05*.gz /<new-location-of-vsantraces>
Note: If you're not comfortable performing the delete/move of older vsantraces logs engage vSAN support for assistance.
Workaround:
If no additional storage is available, limit the size of vsantraces to 200MB. This allows a bit of overhead for additional files and ensures that we do not fill the ramdisk.
For vSAN 6.7 and lower this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set -f 10 -s 20
This would make the max files setting '10' and the file size setting 20 MBs. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
For vSAN 7.x and higher this can be done by running this command on all vSAN hosts in the cluster:
esxcli vsan trace set --urgentnumfiles=10 --urgentsize=10 --numfiles=10 --size=10
This would make the max files setting '10' and the file size setting 10 MBs for both the vsantraces and the urgenttraces. With these settings, the vSAN trace logs would only grow to a maximum of 200 MBs
These changes are immediate and older files will be rolled off as smaller ones take their places over time.
No services require a restart to enable this change.
Delete older vsantraces files from the vsantraces folder /vsantraces/ as per the Resolution section.
Additional Information
Impact/Risks:
- The ramdisk gets full which could lead to a hung host or the host becoming non-responsive in vCenter
- The vsantraces files are used for vSAN diagnostics so if vsantraces stops logging due to no space then historical data is lost for troubleshooting vSAN issues.
Feedback
Yes
No
Powered by
