System logs are stored on non-persistent storage

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

You receive this message when your system stores its log files on non-persistent storage.

Example

When you use the advanced system parameter Syslog.global.logDir to configure the system log location, if you configure the system log location as temporary, for example [] /tmp, you get the esx.problem.syslog.nonpersistent message.

When the location for system logs is not configured, your system uses the scratch partition by default. If the scratch location is set to nonpersistent storage, such as /tmp, you get the esx.problem.syslog.nonpersistent message.

Impact

System logs are required for auditing and diagnostic purposes. If you do not store system logs permanently, for example on a datastore, they disappear after a reboot.

IMPORTANT NOTE for vSAN.

vSAN datastores should not be used for persistent logging.
It is not supported to configure a scratch location on a vSAN datastore.

Redirecting system logs to a vSAN object causes an ESXi host lock up (2147541)
Storing ESXi coredump and scratch partitions in vSAN (2074026)
Persistent Logging in a Virtual SAN Cluster
Creating a persistent scratch location for ESXi 7.x/6.x/5.x/4.x (1033696)

Environment

VMware vCenter Server 6.5.x
VMware vCenter Server 7.0.x
VMware vSphere ESXi 6.5
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware vSphere ESXi 7.0.0
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server 5.5.x
VMware vCenter Server 5.1.x
VMware vCenter Server 5.0.x
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.0
VMware vCenter Server Appliance 6.7.x
VMware vSphere ESXi 6.7

Resolution

Verifying the Location of System Logs in vSphere Client (HTML5)

In the vSphere Client navigator, select the Hosts and Clusters view
Select the host object in the vSphere Client navigator.
Click the Configure tab, then the System expander
Under System, click Advanced System Settings.
Ensure that Syslog.global.logDir points to a persistent location.
If the field Syslog.global.logDir is empty or points to a scratch partition, make sure that the field ScratchConfig.CurrentScratchLocation shows a location on persistent storage.
If the folder being used as a scratch location is to be shared by multiple ESXi hosts, you should also set the field Syslog.global.logDirUnique to avoid contention over log files.

Note: To log to a datastore, the Syslog.global.logDir entry should be in the format of [Datastorename]/foldername. To log to the scratch partition set in the ScratchConfig.CurrentScratchLocation, the format is blank or []/foldername.

Verifying the Location of System Logs in vSphere Web Client

Browse to the host in the vSphere Web Client navigator.
Click the Manage tab, then click Settings.
Under System, click Advanced System Settings.
Ensure that Syslog.global.logDir points to a persistent location.
If the field Syslog.global.logDir is empty or points to a scratch partition, make sure that the field ScratchConfig.CurrentScratchLocation shows a location on persistent storage.
If the folder being used as a scratch location is to be shared by multiple ESXi hosts, you should also set the field Syslog.global.logDirUnique to avoid contention over log files.

Verifying the Location of System Logs in vSphere Client (vSphere 6.0 and earlier)

In vSphere Client, select the host in the inventory panel.
Click the Configuration tab, then click Advanced Settings under Software.
Ensure that Syslog.global.logDir points to a persistent location.
The directory should be specified as [datastorename] path_to_file where the path is relative to the datastore. For example, [datastore1] /systemlogs.
If the Syslog.global.logDir field is empty or explicitly points to a scratch partition, make sure that the field ScratchConfig.CurrentScratchLocation shows a location on persistent storage.

Note: ESXi server should be rebooted for the Scratch configuration changes to get updated.
The Current configuration location should match the Configured location to ensure the changes are successful.

Additional Information

If you see that a working host has its information for the scratch location in a UUID format:
/vmfs/volumes/xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx/foldername and want to find out what the "friendly" name is, as it would appear in your vCenter or host client view, you can:

Connect to the working host via SSH session, and log on with root credentials
Run the command:

# esxcli storage filesystem list
The output will look similar to this:

/vmfs/volumes/########-####-########c050 abc-lun3 ########-####-########c050 true VMFS-5 805037932544 400613703680

In this example, abc-lun3 is the friendly name of the datastore you would find in your vCenter or host client, and the Syslog.global.logDir entry should be in the format of [abc-lun3]/foldername