vSAN Traces Observed on ESXi RAM Disk in vSAN 8.0 Update 1 and Later

Products

• VMware vSAN
• VMware vSAN 8.x

Issue/Introduction

In VMware vSAN 8.0 Update 1 and later, users may observe that vSAN trace files are being written to the ESXi host RAM disk (typically located in /scratch/log/).
This behavior may lead to the assumption that vSAN traces are not being stored persistently or that the vSAN Native Traces feature is not functioning as expected.

Environment

VMware vSAN 8.1

Cause

• The vSAN Native Traces feature, introduced in vSAN 8.0 Update 1, utilizes the ESXi host RAM disk as a temporary buffer to collect trace data.
• This architecture is designed to optimize performance by minimizing direct I/O overhead on the vSAN datastore.
• Periodically, the system automatically flushes the collected trace data from the RAM disk buffer to a dedicated persistent namespace object on the vSAN datastore.
• The presence of trace files in the RAM disk is an expected intermediate state of this process and does not indicate a failure in data persistence.

Resolution

Native Traces object :

• To verify that vSAN Native Traces are correctly configured and persisting data to the vSAN datastore, perform the following validation steps:
• Log in to the ESXi host via SSH and run the following command to check the current trace configuration:

esxcli vsan trace get

• Verify Persistent Object Existence: Confirm the existence of the persistent vSAN Native Traces object by locating the .vsan.trace directory on the vSAN datastore:

ls -ld /vmfs/volumes/vsan:<vsan_datastore_uuid>/.vsan.trace

(Note: Replace <vsan_datastore_uuid> with the actual UUID of your vSAN datastore.)

• If the native traces object is healthy, the system will automatically handle the rotation and persistence of logs. No manual intervention is required unless you specifically need to redirect traces to an external persistent storage location (e.g., NFS), which would override the Native Traces default behavior.

 

Persistent External Storage:

• If there is a requirement to store traces on a specific external persistent volume rather than the default vSAN datastore, you may manually configure the path:
• Set the manual trace location:

esxcli vsan trace set --path /vmfs/volumes/<persistent_datastore>/<folder_name>

• Note: Manually configuring a trace path disables the default vSAN Native Traces behavior. The system will no longer use the vSAN datastore namespace for traces.
  
  Extending the retention period of diagnostic logs (clomd, vsantrace) in vSAN environments