Creating a persistent scratch location for ESXi 8.x/7.x/6.x

Products

• VMware vSphere ESXi

Issue/Introduction

This article provides steps to create a persistent scratch location for ESXi.

VMware recommends that ESXi has a persistent scratch location available for storing temporary data including logs, diagnostic information, and system swap. (However, this is not a requirement) Persistent scratch space may be provisioned on a FAT 16, VMFS, or NFS partition accessible by the ESXi host.

Note: Pointing the scratch location to a partition on an SD card or USB stick is not recommended. These devices only have a limited amount of read/write cycles.

Note: Scratch space is configured automatically during installation or first boot of an ESXi host, and is not required to be manually configured. ESXi Installable creates a 4 GB Fat16 partition on the target device during installation if there is sufficient space, and if the device is considered Local.

Only ESXi systems with swap enabled can be added to an HA cluster. For more information, see ESXi hosts without swap enabled cannot be added to a VMware High Availability Cluster .

Note: For ESXi versions prior to 7.0, scratch space is configured automatically during installation or first boot of an ESXi host, and is not required to be manually configured. ESXi Installer creates a 4 GB Fat16 partition on the target device during installation if there is sufficient space, and if the device is considered Local.

From ESXi 7.0, a VMFS-L based ESX-OSData volume (where logs, coredumps and configuration are stored) replaces the traditional scratch partition. During upgrade, the configured scratch partition is converted to ESX-OSData. The settings described in this article are still applicable for cases where you want to point the scratch path to another location.

Symptoms:

• When you run the esxcli software vib list command, you see error similar to:
  
  Traceback (most recent call last):
  File "/usr/lib/vmware/esxcli-software", line 432, in module
  setupLogging()
  File "/usr/lib/vmware/esxcli-software", line 78, in setupLogging
  handler = logging.handlers.SysLogHandler(address='/dev/log')
  File "/build/mts/release/bora-504890/bora/build/esx/release/python-
  2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
  line 704, in __init__
  File "/build/mts/release/bora-504890/bora/build/esx/release/python-
  2.6-lib-zip-stage/504890/visor/pylib/python2.6/logging/handlers.py",
  line 719, in _connect_unixsocket
  File "string", line 1, in connect
  socket.error: [Errno 2] No such file or directory
   
• Commands related to esxcli software or esxupdate do not work.
• Scanning the host using Update Manager fails with the error:
  
  General System Error
   
• Rebooting the host initially resolves the issue, but it reoccurs in a few days.
• Log files are missing in /var/log/ file or are not being updated.
• You cannot install HA agent on affected ESXi hosts.
• ESXi is installed on a SD card or USB stick. 

Environment

VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 5.0
VMware vSphere ESXi 6.5
VMware ESXi 4.0.x Embedded
VMware vSphere ESXi 5.5
VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 6.0
VMware vSphere ESXi 5.1
VMware ESXi 4.1.x Embedded
VMware vSphere ESXi 6.7

Cause

If persistent scratch space is not available, ESXi stores this temporary data on a ramdisk, which is constrained in space. This might be problematic in low-memory situations, but is not critical to the operation of ESXi. Information stored on a ramdisk does not persist across reboots, so troubleshooting information such as logs and core files could be lost. If a persistent scratch location on the host is not configured properly, you may experience intermittent issues due to lack of space for temporary files and the log files are not updated.

Scratch space is configured automatically during installation or first boot of an ESXi 6.x and earlier host, and is not required to be manually configured.

Resolution

Note: It is not supported to configure the scratch location on a vSAN datastore.

Caution: Do not share a scratch locker directory among multiple ESXi hosts.

Scratch space can be manually configured for an ESXi host using the vSphere Client, PowerCLI, Tech Support Mode, or during scripted installation. Select the method appropriate to your environment. Regardless of the method selected, the configuration is written to the /etc/vmware/locker.conf configuration file for use during the next boot.

Changing the configured scratch location requires a reboot of the ESXi host for the changes to take effect. Use vMotion to migrate or power off all virtual machines on the ESXi host prior to making the change. 

The configured scratch location must point to a directory on a filesystem accessible to the ESXi host, such as a VMFS volume. The location may be a directory on a shared or remote disk as long as the ESXi host is allocated its own separate directory.

Method to Configure a Persistent Scratch Location:

• Using the vSphere Client  for ESXi 6.x and later
• Using the VMware Host Client for ESXi 6.5
• Using PowerCLI 5.1 or later
• Using Tech Support Mode
• During scripted install using kickstart scripts

It may be necessary to do a reboot of the ESXi host before completing these procedures.

Configuring a persistent scratch location using the vSphere Web Client  for ESXi 6.x and greater. Follow similar steps when using HTML5 Client:

1. Log in to vCenter Server using the vSphere Web Client.
2. Click Hosts and Clusters, then select the specific host.
3. Click System
4. Click  Advanced System Settings.
5. Locate ScratchConfig.ConfiguredScratchLocation.
6. Click Edit and add the path to the scratch directory.
7. Reboot the host.

NOTE: To find datastore UUID go to the datastore--> Summary-->url 

Warning: Do not set scratch to the root of a directory. Refere this article Upgrading host from vSphere ESXi 6.x to 7.x datastores may become overwritten if incorrect scratch location is set, prior, or after upgrade

Configuring a persistent scratch location using the VMware Host Client for ESXi 6.5:

1. Connect to ESXi host using Host Client.
2. Click Storage.
3. Right-click a Datastore and click Browse.
4. Create a uniquely-named directory for this ESXi host.
   
   For example:
   
   .locker-ESXHostname
    
5. Close the Datastore Browser.
6. Click the Manage tab
7. Under System, click Advanced System Settings.
   
   The setting ScratchConfig.CurrentScratchLocation shows the current location of the scratch partition.
    
8. In the ScratchConfig.ConfiguredScratchLocation field, enter a directory path that is unique for this host.
   
   For example, /vmfs/volumes/DatastoreUUID/.locker-ESXHostname.
    
9. Reboot the host.

Configuring a persistent scratch location using the vSphere Client:

1. Connect to vCenter Server or the ESXi host using the vSphere Client.
2. Click the ESXi host in the inventory.
3. Click the Configuration tab > Storage.
4. Right-click a Datastore and click Browse.
5. Create a uniquely-named directory for this ESXi host.
   
   For example:
   
   .locker-ESXHostname
    
6. Close the Datastore Browser.
7. Click Software > Advanced Settings.
8. Select the ScratchConfig section.
9. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory.
   
   For example:
   
   /vmfs/volumes/DatastoreUUID/.locker-ESXHostname
   
   Note: To determine the Datastore UUID: 
   1. Navigate to Configuration > Storage.
   2. Select the Datastore.
   3. View location under Datastore Details.
      
      The Datastore UUID appears similar to /vmfs/volumes/xxxxxxxxxxxxxxxx where xxxxxxxxxxxxxxx is the UUID.
       
10. Click OK.
11. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect. 

Configuring a persistent scratch location using PowerCLI 5.1 or later:

Notes:

Before proceeding, ensure that /tmp/scratch exists. If it does not exist, use the command mkdir /tmp/scratch to create it.

1. Open a command prompt where the PowerCLI is installed.
2. Connect to the ESXi host using the command:
   
   connect-viserver esx_hostname_or_IP
    
3. Obtain a list of datastores reachable from this ESXi host using the command:
   
   Get-Datastore
    
4. Mount a datastore read/write as a PSDrive using the command:
   
   New-PSDrive -Name "mounteddatastore" -Root \ -PSProvider VimDatastore -Datastore (Get-Datastore "DatastoreName")
    
5. Access the new PSDrive using the command:
   
   Set-Location mounteddatastore:\
    
6. Create a uniquely-named directory for this ESXi host using the command:
   
   New-Item "DirectoryName" -ItemType directory
   
   For example:
   
   New-Item ".locker-ESXHostname" -ItemType directory
    
7. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   Get-VMhost | Get-AdvancedSetting -Name "ScratchConfig.ConfiguredScratchLocation"
    
8. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in step 6, using the command:
   
   Get-VMhost | Get-AdvancedSetting -Name "ScratchConfig.ConfiguredScratchLocation" | Set-AdvancedSetting -Value "/vmfs/volumes/DatastoreName/DirectoryName"
   
   For example:
   
   Get-VMhost | Get-AdvancedSetting -Name "ScratchConfig.ConfiguredScratchLocation" | Set-AdvancedSetting -Value "/vmfs/volumes/Datastore1/.locker-ESXHostname"
    
9. When prompted, select Yes to modify the advanced setting.
10. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect.

Configuring a persistent scratch location using Tech Support Mode

Notes:

• This can be done through the local console or SSH.
• The .filename is a hidden file. To view hidden files, use the ls -a command.

1. Open a console to the ESXi host. For more information, see Using Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.x.
2. Obtain a list of datastores reachable from this ESXi host using the command:
   
   ls /vmfs/volumes
    
3. Create a uniquely-named directory for this ESXi host using the command:
   
   mkdir /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   mkdir /vmfs/volumes/Datastore1/.locker-ESXHostname
    
4. Check the current value of the ScratchConfig.ConfiguredScratchLocation configuration option using the command:
   
   vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation
   
   For example:
   
   (vim.option.OptionValue) [
(vim.option.OptionValue) {
dynamicType = unset,
key = "ScratchConfig.ConfiguredScratchLocation",
value = "/path/to/location",
}
}
    
5. Change the ScratchConfig.ConfiguredScratchLocation configuration option, specifying the full path to the directory created in step 3, using the command:
   
   vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/DatastoreName/DirectoryName
   
   For example:
   
   /bin/vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string /vmfs/volumes/Datastore1/.locker-ESXHostname
    
6. Put the ESXi host into maintenance mode and reboot for the configuration change to take effect. 

 

Configuring a persistent scratch location during scripted install using kickstart scripts

You can configure persistent scratch space for ESXi during kickstart installation by adding commands to your kickstart scripts which create the directory and change the configuration option.

Append these commands to your custom kickstart script:

# Generate a new scratch directory for this host on a Datastore:
scratchdirectory=/vmfs/volumes/DatastoreName/.locker-$(hostname 2> /dev/null)-$(esxcfg-info -b 2> /dev/null)

# Create the scratch directory:
mkdir -p $scratchdirectory

# Change the advanced configuration option:
vim-cmd hostsvc/advopt/update ScratchConfig.ConfiguredScratchLocation string $scratchdirectory
 

Note: The configured scratch location is activated during startup of the ESXi host. A reboot is required after performing the preceding commands.

Additional Information

The alert/warning about persistent scratch location not set can be seen in VMware Skyline Health: VMware Skyline Health Diagnostics for vSphere - FAQ

ScratchConfig.ConfiguredScratchLocation" vs "Syslog.global.logDir"

1. Syslog.global.logDir is the setting of the log path, by default it points to "/scratch/log", but basically you can set it to anywhere.
2. "/scratch" is the scratch location defined by ScratchConfig.ConfiguredScratchLocation, in this path it could or could not contain logs depends on your Syslog.global.logDir setting. If you change Syslog.global.logDir to somewhere other than "/scratch/log", then logs won't go to the location defined by ScratchConfig.ConfiguredScratchLocation

If your "Syslog.global.logDir" is pointing to /scratch/log, then you just need to change "ScratchConfig.ConfiguredScratchLocation" to a persistent location.

Otherwise, change "Syslog.global.logDir" to a persistent location, it does not matter what "ScratchConfig.ConfiguredScratchLocation" is.
"ScratchConfig.ConfiguredScratchLocation" contains other things than merely logs, whether or not set up two different location would depend on:

1. If you want log to be in persistent storage while other things in ramdisk, then you can set up two different location as:

   Persistent storage for Syslog.global.logDir
   Ramdisk for "ScratchConfig.ConfiguredScratchLocation"

2. Otherwise, keep Syslog.global.logDir as default (/scratch/log), change ScratchConfig.ConfiguredScratchLocation to a persistent storage.

For more information on the scratch partition, see:

• The Set the Scratch Partition from the vSphere Web Client:  https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.esxi.install.doc/GUID-6A4FCA6C-498C-4080-BFE8-AB9911A4B033.html

If the host is running for more than 24 hours, this setting may not stay persistent after performing the preceding steps. In this case, perform these steps:

1. Place the host into Maintenance Mode and reboot the host.
2. Click the Configuration tab.
3. Click Software.
4. Click Advanced Settings.
5. Select the ScratchConfig section.
6. Change the ScratchConfig.ConfiguredScratchLocation configuration option and set it to the full path of the directory.
   
   For example:
   
   /vmfs/volumes/DatastoreName/.locker-ESXHostname
    
7. Click OK.
8. Close the window.
9. Click Advanced Settings to reopen the window.
10. Select the ScratchConfig section.
11. Verify that the ScratchConfig.ConfiguredScratchLocation field contains the scratch setting that you configured in step 6. If it does, reboot the host.
    
    The host is now using the configured scratch partition.

For more information on determining whether a disk device, see Identifying disks when working with VMware ESXi, and review the Is Local flag in the output of the esxcfg-scsidevs -l command:

• If Local is true, the storage device is not sharable among multiple ESXi hosts.
• If Local is false, the storage device is remote and potentially sharable among several ESXi hosts. Remote devices are not automatically used for scratch by ESXi.

ESXi selects one of these scratch locations during startup in this order of preference:

1. The location configured in the /etc/vmware/locker.conf configuration file, set by the ScratchConfig.ConfiguredScratchLocation configuration option, as in this article.
2. A Fat16 filesystem of at least 4 GB on the Local Boot device.
3. A Fat16 filesystem of at least 4 GB on a Local device.
4. A VMFS datastore on a Local device, in a .locker/ directory.
5. A ramdisk at /tmp/scratch/.

These are two examples where scratch space may not be automatically defined on persistent storage. In each case, the temporary scratch location is configured on a ramdisk:

• ESXi deployed on a Flash or SD device, including a USB key. Scratch partitions are not created on Flash or SD storage devices even if connected during install, due to the potentially limited read/write cycles available.
• ESXi deployed in a Boot from SAN configuration or to a SAS device. A Boot from SAN or SAS LUN is considered Remote, and could potentially be shared among multiple ESXi hosts. Remote devices are not used for scratch to avoid collisions between multiple ESXi hosts.

To remove the configuration or restore to ramdisk:

1. Place the host into MM and change only the ScratchConfig.ConfiguredScratchLocation in the specific host Advanced Settings to /tmp

2. Save and reboot the host.

3. Upon reboot you will see the ScratchConfig.CurrentScratchLocation updated to /tmp.

4. If the change does not reflect please reboot the host 1 additional time. You will then be able to remove the VMFS datastore.

Related Links:

• ESXi hosts without swap enabled cannot be added to a VMware High Availability Cluster
• Identifying disks when working with VMware ESXi/ESX
• Configuring syslog on ESXi
• Determining whether an ESXi host has persistent logging
• Configuring Scratch partition Manually in hosts with Large number of datastore setup (512 or 1k Volumes)