Validation Steps:
YYYY-MM-DDTHH:MM:SS.252Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS253Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS.253Z Er(11) chardevlogger[8244481]: Failed to read trace: Cannot allocate memory
YYYY-MM-DDTHH:MM:SS.254Z Er(11) chardevlogger[8244544]: Failed to read trace: Cannot allocate memory
vdf -h
-----
Ramdisk Size Used Available Use% Mounted on
root 32M 2M 29M 8% --
etc 28M 232K 27M 0% --
opt 32M 0B 32M 0% --
var 48M 516K 47M 1% --
tmp 256M 12K 255M 0% --
iofilters 32M 0B 32M 0% --
shm 1024M 0B 1024M 0% --
hostdstats 114M 2M 111M 1% --
vsantraces 300M 287M 12M 95% --
vsantraces ramdisk appears to be full or nearly full on the ESXi:
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
It is possible in vSAN that the ramdisk for vSAN trace information may grow and fill the allotted ramdisk. vsantraces can be redirected to another location for logging.
vSAN traces can grow to any size you select, but VMware recommends around 600 MB if available.
If persistent logging is configured for the cluster, we can redirect vsantraces here using these steps.
1) Connect to each ESXi server in the cluster using SSH.
2) Get the current vsantrace config by running esxcli vsan trace get
You see output similar to:
vSAN Traces Directory: /vsantraces
Number Of Files To Rotate: 8
Maximum Trace File Size: 180 MB
Log Urgent Traces To Syslog: true
3) To configure vsantraces to log elsewhere, run this command:
[root@localhost:~] esxcli vsan trace set
Usage: esxcli vsan trace set [cmd options]
Note: This command is not thread safe.
Cmd options:
-f|--numfiles: Log file rotation for vSAN trace files.
-p|--path: Path to store vSAN trace files.
-r|--reset: When set to true, reset defaults for VSAN trace files.
-s|--size: Maximum size of vSAN trace files in MB.
If the Scratch folder has already been redirected, we can also redirect vsantraces into the same folder. There may be unique folders for each host that already exist on your storage for this if previously configured. In this case, create a folder called vsantraces so these can be separated out per-Host. If one folder per Host is not there, create one folder for each. To create one folder for each host, for the corresponding vSphere version, see Creating a persistent scratch location for ESXi 8.x/7.x/6.x (1033696)
Note: Log-bundles should be separated into their own host-folders, otherwise, once the first host gets the locks on the scratch location it prevents the other hosts in the cluster from logging.
After we have a folder for each Host and the logging is separated, we can redirect traces using these commands:
mkdir /vsantraces
esxcli vsan trace set -f 10 -s 60 -p /vsantraces
For example:
If files are located at /vmfs/volumes/Datastore/scratch/localhost/, run these commands:
mkdir /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
esxcli vsan trace set -f 10 -s 60 -p /vmfs/volumes/Datastore/scratch/localhost/vsantraces/
Note: The preceding commands are just an example. The location may vary based on the environment.
This automatically begins sending trace files to the new location, no further configuration is needed.
Delete or move older vsantraces files from the vsantraces folder /vsantraces/ located on the ramdisk
1) Navigate to the vsantraces folder by running cd var/log/vsantraces
2) List the contents of the folder by running ls-ahl
Sample output:
[root@esx:/vsantraces] ls -ahl
total 264512
drwxr-xr-x 1 root root 512 Jun 4 20:04 .
drwxr-xr-x 1 root root 512 Jun 4 20:16 ..
-rw-r--r-- 1 root root 254.9K Jun 4 17:44 vsanObserver--YYYY-MM-DDT17h24m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 18:06 vsanObserver--YYYY-MM-DDT17h46m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 18:28 vsanObserver--YYYY-MM-DDT18h08m00s.gz
-rw-r--r-- 1 root root 254.8K Jun 4 18:50 vsanObserver--YYYY-MM-DDT18h30m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:12 vsanObserver--YYYY-MM-DDT18h52m00s.gz
-rw-r--r-- 1 root root 254.6K Jun 4 19:34 vsanObserver--YYYY-MM-DDT19h14m00s.gz
-rw-r--r-- 1 root root 254.7K Jun 4 19:56 vsanObserver--YYYY-MM-DDT19h36m00s.gz
-rw-r--r-- 1 root root 208.5K Jun 4 20:16 vsanObserver--YYYY-MM-DDT19h58m01s.gz
-rw-r--r-- 1 root root 20.0M Jun 2 07:56 vsantraces--YYYY-MM-DDT23h01m33s187.gz
-rw-r--r-- 1 root root 20.0M Jun 2 16:51 vsantraces--YYYY-MM-DDT07h56m21s853.gz
-rw-r--r-- 1 root root 20.0M Jun 3 01:48 vsantraces--YYYY-MM-DDT16h51m11s433.gz
-rw-r--r-- 1 root root 20.0M Jun 3 10:45 vsantraces--YYYY-MM-DDT01h48m01s609.gz
-rw-r--r-- 1 root root 20.0M Jun 3 19:39 vsantraces--YYYY-MM-DDT10h45m44s438.gz
-rw-r--r-- 1 root root 20.0M Jun 4 04:28 vsantraces--YYYY-MM-DDT19h39m32s771.gz
-rw-r--r-- 1 root root 20.0M Jun 4 13:24 vsantraces--YYYY-MM-DDT04h28m00s889.gz
-rw-r--r-- 1 root root 15.4M Jun 4 20:17 vsantraces--YYYY-MM-DDT13h24m46s788.gz
-rw-r--r-- 1 root root 192 Jun 4 13:24 vsantraces.index
-rw-r--r-- 1 root root 8.0M May 15 03:33 vsantracesClom--YYYY-MM-DDT15h15m00s791.gz
-rw-r--r-- 1 root root 8.0M May 18 15:48 vsantracesClom--YYYY-MM-DDT03h33m29s948.gz
-rw-r--r-- 1 root root 8.0M May 22 04:02 vsantracesClom--YYYY-MM-DDT15h48m59s091.gz
-rw-r--r-- 1 root root 8.0M May 25 15:50 vsantracesClom--YYYY-MM-DDT04h02m55s293.gz
-rw-r--r-- 1 root root 8.0M May 29 02:54 vsantracesClom--YYYY-MM-DDT15h50m08s307.gz
-rw-r--r-- 1 root root 8.0M Jun 1 05:55 vsantracesClom--YYYY-MM-DDT02h54m10s389.gz
-rw-r--r-- 1 root root 8.0M Jun 4 18:09 vsantracesClom--YYYY-MM-DDT05h55m47s005.gz
-rw-r--r-- 1 root root 206.3K Jun 4 20:16 vsantracesClom--YYYY-MM-DDT18h09m27s740.gz
-rw-r--r-- 1 root root 192 Jun 4 18:09 vsantracesClom.index
-rw-r--r-- 1 root root 620.8K Jun 4 03:29 vsantracesDOMObj--YYYY-MM-DDT17h23m25s376.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesDOMObj.index
-rw-r--r-- 1 root root 1.0M Jun 3 17:10 vsantracesIODiag--YYYY-MM-DDT18h31m21s576.gz
-rw-r--r-- 1 root root 1.0M Jun 4 15:56 vsantracesIODiag--YYYY-MM-DDT17h10m06s623.gz
-rw-r--r-- 1 root root 195.4K Jun 4 20:17 vsantracesIODiag--YYYY-MM-DDT15h56m16s897.gz
-rw-r--r-- 1 root root 72 Jun 4 15:56 vsantracesIODiag.index
-rw-r--r-- 1 root root 2.0M May 17 13:09 vsantracesLSOM--YYYY-MM-DDT17h23m25s299.gz
-rw-r--r-- 1 root root 575.1K Jun 4 20:11 vsantracesLSOM--YYYY-MM-DDT13h09m23s998.gz
-rw-r--r-- 1 root root 48 May 17 13:09 vsantracesLSOM.index
-rw-r--r-- 1 root root 15 Mar 13 17:23 vsantracesLSOMVerbose--YYYY-MM-DDT17h23m25s333.gz
-rw-r--r-- 1 root root 24 Mar 13 17:23 vsantracesLSOMVerbose.index
-rw-r--r-- 1 root root 10.0M May 29 00:22 vsantracesUrgent--YYYY-MM-DDT05h05m25s105.gz
-rw-r--r-- 1 root root 10.0M May 31 12:01 vsantracesUrgent--YYYY-MM-DDT00h22m35s242.gz
-rw-r--r-- 1 root root 10.0M Jun 4 10:11 vsantracesUrgent--YYYY-MM-DDT12h01m34s418.gz
-rw-r--r-- 1 root root 1.0M Jun 4 20:17 vsantracesUrgent--YYYY-MM-DDT10h11m46s702.gz
-rw-r--r-- 1 root root 96 Jun 4 10:11 vsantracesUrgent.index
rm vsantraces*20##-05*.gz
rm vsanObserver--20##-05*.gz
rm vsantracesUrgent--20##-05*.gz
Note: If you're not comfortable performing the delete/move of older vsantraces logs engage vSAN support for assistance.