vmksummary logs missing on ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host log bundle may have one or more log files missing.

• The error on the log bundle in file error.log mentions that the resource or device being accessed is busy.

  error.log:77:Failed to add file <action cmd="copy" options="IGNORE_MISSING" path="/var/run/log/*">: [Errno 16] Device or resource busy: '/var/run/log/vmksummary.log'

• The vmksummary.log file cannot be seen under the /var/log directory. 

Environment

VMware vSphere ESXi

Cause

A lock held on the log file being collected by any other resources can cause the device/file to be identified as busy.
This can cause issues with stability of the ESXi host as multiple hosts could be trying to lock the same file due to an erroneous scratch configuration.

Resolution

Steps to Verify : 

• Check if the log file exists and if it is locked by a different host using the below commands :
  
  
  • Check if the symlink is present on the vmksummary.log file in the /var/log directory with command : ls /var/log/ | grep vmksummary
    The expected output is :

    ls -l  /var/log/ | grep vmksummary
    lrwxrwxrwx    1 root     root            27 Oct DD HH:MM vmksummary.log -> /scratch/log/vmksummary.log
    

• • Check if the file is present on the configured scratch directory using command : ls /var/run/log | grep vmksummary
    The expected output is :

    ls -l /var/run/log/ | grep vmksummary
    -rw-------    1 root     root          4917 Oct DD HH:MM vmksummary.log
    

  • Check if the file in the scratch directory is locked by another host using the command :
    vmfsfilelockinfo -p /var/run/log/vmksummary.log

 

• If the file exists in the scratch directory and is being locked by another host , check the scratch configuration parameters on the two hosts i.e the current host and the host that is locking the file, using the below commands :

• • vim-cmd hostsvc/advopt/view ScratchConfig.ConfiguredScratchLocation
  • vim-cmd hostsvc/advopt/view ScratchConfig.CurrentScratchLocation
  • vim-cmd hostsvc/advopt/view Syslog.global.logDirUnique

If the above command shows the same output for the 3 parameters , the scratch configuration on the ESXi host is problematic and needs to be rectified.

Steps to Fix : 

Follow the below KB article to fix the scratch configuration :
Creating a persistent scratch location for ESXi 8.x/7.x/6.x .

Additional Information

Article for configuring the scratch partition correctly : Creating a persistent scratch location for ESXi 8.x/7.x/6.x .