Error "Device or resource busy" when viewing log files on the ESXi host

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• ESXi host syslog is configured to write logs on a datastore
• When attempting to view log files on the ESXi host SSH session using cat or less command, You will see error similar to:

root@hostname: less var/run/log/hostd.log

hostd.log: Device or resource busy

Environment

VMware vSphere ESXi

Cause

The error "Device or resource busy" is caused when the log file is locked by another ESXi host in the environment. This occurs when multiple ESXi hosts are writing logs to the same location

Resolution

To resolve this issue identify the ESXi host holding the lock on the log file and correct the syslog configuration. 

Note: If a common datastore is used by all the ESXi hosts to save logs, each ESXi host should be configured with a unique directory on the datastore for syslog configuration.

Identify the ESXi host that is holding the lock on the log file:

• Open an SSH session to the ESXi host and login as the root user.
• Run this command:

vmkfstools -D <path of the log file>

Example:

root@hostname:vmkfstools -D /vmfs/volumes/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/logs/hostd.log
Lock [type 10c00001 offset 451526656 v 66, hb offset 3473408
gen 263, mode 1, owner xxxxxxxx-xxxxxxxx-xxxx-xxxxxxxxxxxx mtime 8934920

• From the output of the vmkfstools command, record the MAC address(highlighted section in the above example)
• Identify which ESXi host in the environment has this mac address by comparing the MAC address from the output of vmkfstools command with the MAC address of the vmnics of all other ESXi hosts

To check the MAC address of the vmnics on the ESXi hosts:

• On the vSphere Client, select the host and click the Configure tab > Networking >Physical adapters
• Or use the command esxcli network nic list 

Note: If there are many ESXi hosts in the environment we can use  vmfsfilelockinfo command to identify the ESXi host having a lock.

Example:

vmfsfilelockinfo -p <Logfilepath> -v <vCenter ip or hostname> -u <vCenter admin username>

vmfsfilelockinfo -p /vmfs/volumes/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/logs/hostd.log -v XX:XX:XX:XX -u [email protected]

Correcting the syslog configuration on the ESXi host:

After identifying the ESXi host holding the lock, review the syslog configuration of the host and change it to a unique directory

• Connect to the vCenter Server or the ESXi host using the vSphere Client.
• Select the ESXi host in the Inventory.
• Click the Configure tab
• Under the system heading, click Advanced system Settings.
• Select Edit
• Under the Key section, filter with "Syslog.global.logDir"
• Enter a valid unique path under "value"
• Select OK

Note: The unique log directory must exist before changing the global logging directory parameter.

 

Additional Information

For additional information related to vmfsfilelockinfo command, refer Investigating virtual machine file locks on ESXi hosts