Alarms "System logs are stored on non-persistent storage" triggered on ESXi hosts using USB/SD boot media
search cancel

Alarms "System logs are stored on non-persistent storage" triggered on ESXi hosts using USB/SD boot media

book

Article ID: 317690

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides guidance for resolving simultaneous alarms regarding non-persistent system logs and missing coredump targets on ESXi 8.x hosts. These symptoms typically occur following an upgrade or installation when the boot media does not meet the revised hardware requirements for the ESX-OSData partition.

System logs are required for auditing and diagnostic purposes. If system logs are not stored permanently on a datastore, the logs disappear after a reboot. The esx.problem.syslog.nonpersistent message triggers when the system stores log files on non-persistent storage.

This occurs under the following conditions:

  • The advanced system parameter Syslog.global.logDir is configured to a temporary location, for example [] /tmp.

  • The location for system logs is not configured, causing the system to use the scratch partition by default. If the scratch location is set to non-persistent storage, such as /tmp, the message triggers.

Symptoms:

  • ESXi hosts display the warning: System logs on host [hostname] are stored on non-persistent storage.

  • ESXi hosts display the warning: No coredump target has been configured. Host core dumps cannot be saved.

  • Advanced setting ScratchConfig.CurrentScratchLocation points to a temporary ramdisk directory (e.g., /tmp/_osdata...) instead of persistent storage.

Important Note for vSAN: vSAN datastores are not supported for persistent logging and must not be used to configure a scratch location. Redirecting system logs to a vSAN object causes an ESXi host lock up.

For additional context on vSAN environments and scratch locations, refer to the below KB articles: 

Environment

  • VMware vCenter Server 8.0.x

  • VMware vCenter Server 7.0.x

  • VMware vSphere ESXi 8.0.x

  • VMware vSphere ESXi 7.0.x

  • Applicable also to older and out-of-support versions

  • Hardware: USB/SD boot media < 32 GB

Cause

ESXi 7.x and 8.x system architecture requires a minimum of 32 GB (128 GB recommended) of persistent local storage to provision the ESX-OSData partition. This partition houses the scratch area for logs and coredumps.

  • If the boot media (USB/SD card) is 30 GB or less, the installer cannot create the OSData partition. The system defaults to a volatile RAMDisk for scratch space and coredumps, triggering the alarms.

  • Redirecting these partitions to a vSAN datastore is unsupported and causes host unresponsiveness. Reference: Redirecting system logs to a vSAN object causes an ESXi host lock up

Resolution

Verifying the Location of System Logs in vSphere Client (HTML5)

  1. In the vSphere Client navigator, select the Hosts and Clusters view.

  2. Select the host object in the vSphere Client navigator.

  3. Click the Configure tab, then the System expander.

  4. Under System, click Advanced System Settings.

  5. Ensure Syslog.global.logDir points to a persistent location.

  6. If the Syslog.global.logDir field is empty or points to a scratch partition, ensure the ScratchConfig.CurrentScratchLocation field shows a location on persistent storage.

  7. If the folder used as a scratch location is shared by multiple ESXi hosts, set the Syslog.global.logDirUnique field to avoid contention over log files.

Note: To log to a datastore, the Syslog.global.logDir entry format is [Datastorename]/foldername. To log to the scratch partition set in ScratchConfig.CurrentScratchLocation, the format is blank or []/foldername.

Verifying the Location of System Logs in vSphere Web Client

  1. Browse to the host in the vSphere Web Client navigator.

  2. Click the Manage tab, then click Settings.

  3. Under System, click Advanced System Settings.

  4. Ensure Syslog.global.logDir points to a persistent location.

  5. If the Syslog.global.logDir field is empty or points to a scratch partition, ensure the ScratchConfig.CurrentScratchLocation field shows a location on persistent storage.

  6. If the folder used as a scratch location is shared by multiple ESXi hosts, set the Syslog.global.logDirUnique field to avoid contention over log files.

Verifying the Location of System Logs in vSphere Client (vSphere 6.0 and earlier)

  1. In vSphere Client, select the host in the inventory panel.

  2. Click the Configuration tab, then click Advanced Settings under Software.

  3. Ensure Syslog.global.logDir points to a persistent location. Specify the directory as [datastorename] path_to_file where the path is relative to the datastore (for example, [datastore1] /systemlogs).

  4. If the Syslog.global.logDir field is empty or explicitly points to a scratch partition, ensure the ScratchConfig.CurrentScratchLocation field shows a location on persistent storage.

Note: Reboot the ESXi server for the Scratch configuration changes to update. The current configuration location must match the configured location to ensure successful changes.

Resolving System Log Errors on Non-Persistent Storage

To resolve the highlighted errors regarding system logs on non-persistent storage, ensure the ESXi host has compliant persistent local storage.

  1. Replace Boot Media: Replace USB/SD boot devices with persistent local storage (SSD/NVMe/HDD/SATADOM) of at least 32 GB (128 GB recommended for 8.x).

  2. Re-install/Reconfigure: Install ESXi on the new compliant media. The installer automatically provisions the ESX-OSData partition.

  3. Verify Scratch: Navigate to Host > Configure > System > Advanced System Settings and verify ScratchConfig.CurrentScratchLocation points to the persistent OSData volume.

  4. Re-register Coredump: If the coredump alarm persists, run the following command via SSH:

    esxcli system coredump file set --smart --enable true

Additional Information

To determine the friendly name of a datastore when the scratch location is in a UUID format (/vmfs/volumes/####-####-####-####/foldername):

  1. Connect to the working host via an SSH session and log in with root credentials.

  2. Run the following command:

    esxcli storage filesystem list

The output is similar to the following:

/vmfs/volumes/####-####-####-####  abc-lun3     ####-####-####-####     true  VMFS-5    805037932544   400613703680

In this example, abc-lun3 is the friendly name of the datastore displayed in vCenter or the host client. Configure the Syslog.global.logDir entry in the format of [abc-lun3]/foldername.

Powered by Wolken Software