The VMkernel log reports the log spew: owner xxxxx is not free on volume 'datastore'
search cancel

The VMkernel log reports the log spew: owner xxxxx is not free on volume 'datastore'

book

Article ID: 308985

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • The vmkernel.log file (located at /var/log/)contains a log spew similar to:

    vmkernel: 101:20:53:00.082 cpu14:4117)FS3: 2798: [Requested mode: 1] Checking liveness of lock holders [type 10c00001 offset 40669184 v 361, hb offset 3194880
    vmkernel: gen 1609, mode 1, owner 4be1feb6-3c92adf1-14ee-18a9055877ec mtime 59294]on volume 'datastore'.
    vmkernel: 101:20:53:04.084 cpu14:4117)FS3: 2890: [Requested mode: 1] Lock [type 10c00001 offset 40669184 v 361, hb offset 3194880
    vmkernel: gen 1609, mode 1, owner 4be1feb6-3c92adf1-14ee-18a9055877ec mtime 59294] is not free on volume 'datastore'
  • You see that not all hosts are affected.
  • Reviewing the /var/log/messages file indicates that the hosts that see the log spew also experience constant root login attempts from an IP address.
  • In the /var/log/messages, you see messages similar to:

    Aug 16 13:01:20 esx1 /usr/lib/vmware/bin/vmware-hostd[4476]: Accepted password for user root from 10.100.241.66
    Aug 16 13:02:19 esx1 /usr/lib/vmware/bin/vmware-hostd[4476]: Accepted password for user root from 10.100.241.66
    Aug 16 13:04:18 esx1 last message repeated 2 times
    Aug 16 13:05:19 esx1 /usr/lib/vmware/bin/vmware-hostd[4476]: Accepted password for user root from 10.100.241.66
    Aug 16 13:07:19 esx1 last message repeated 2 times
    Aug 16 13:08:18 esx1 /usr/lib/vmware/bin/vmware-hostd[4476]: Accepted password for user root from 10.100.241.66

    Note: In this sample output, the IP address that makes constant login attempts is 10.100.241.66 .

  • The vmkernel.log file contains a flood of lock holder messages.
  • Hosts in the environment that do not see this behavior also do not have any login attempts from the IP address:

    Aug 16 09:05:17 esx6 xinetd[4357]: EXIT: vmware-authd status=0 pid=29011 duration=0(sec)
    Aug 16 11:35:28 esx6 sshd[4501]: Connection from 10.100.10.148 port 61189
    Aug 16 11:35:37 esx6 sshd[4501]: Accepted keyboard-interactive/pam for root from 10.100.10.148 port 61189 ssh2
    Aug 16 13:08:14 esx6 /usr/lib/vmware/bin/vmware-hostd[4580]: Accepted password for user root from 127.0.0.1


Environment

VMware ESXi 4.1.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESXi 3.5.x Installable
VMware ESX Server 3.5.x
VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.0
VMware ESXi 4.0.x Installable
VMware ESX Server 3.0.x
VMware ESX 4.1.x
VMware ESXi 4.0.x Embedded
VMware ESXi 4.1.x Installable
VMware ESX 4.0.x
VMware vSphere ESXi 5.5

Resolution

This issue occurs when an external application (for example, EMC ECC) connects to the ESX hosts and pulls the virtual machine inventory information. This issue occurs because not all hosts are configured for the external application and the /var/log/vmkernel log for these hosts are flooded with lock holder messages.
To minimize these messages, check for the presence of third-party or partner processes and products, such as a backup product, that may frequently query the virtual machine inventory. Reducing the polling interval or stopping the monitoring reduces or stops the log spew. You need to restart the hostd service for the changes to take effect.
Note: With the additional logging in ESX 4.1, these messages are also reported by the NFC service (used by the Datastore browser). The NFC service needs to check the virtual machine inventory, virtual machine, and storage information when displaying information about a virtual machine. You can ignore these messages when they are reported by the NFC service.


Additional Information

VMkernel ログで [owner xxxxx is not free on volume 'datastore'] というログ出力が報告される

Powered by Wolken Software