NFS Datastore File Locking and Latency Issues on ESXi Hosts with Storage I/O Control Enabled

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms

Latency is observed on the datastore in ESXi hosts when multiple hosts attempt to access the datastore simultaneously.

Virtual machines residing on the datastore may experience delays or become unresponsive due to these locks.
Log entries in /var/log/vmkernel.log indicate that files are locked by a consumer, leading to delays or unresponsiveness in virtual machines residing on the datastore.

Environment

VMware ESXi 7.x , 8.x

Cause

The root cause of this issue is related to the interaction between Storage I/O Control and NFS datastore access.

File is being locked by a consumer -

When Storage I/O Control is enabled, each ESXi host checks the datastore’s writability status. If multiple hosts try to access the datastore at the same time, it can result in file locking and generate log events indicating that the file is locked by a consumer.
- In the /var/log/vmkernel.log events are observed related to the file access
  YYYY-MM-DDThh:mm:ss.250Z cpu74:12497368)WARNING: NFSLock: 4084: File is being locked by a consumer on host <host FQDN> with exclusive lock.
  YYYY-MM-DDThh:mm:ss.250Z cpu74:12497368)NFSLock: 5032: failed to get lock on file Virtual-Machine-flat.vmdk(.lck-f2c20#####000000) 0x430d2####890 on NFS-Datastore 464248####036916907-78128462####5098434 (127.0.0.1): Busy

Socket space full. rpc queued -

Additionally, if the NFS server’s RPC queue becomes full (due to insufficient socket buffer), the following log entry is observed: "Socket space full. rpc queued."
- - In /var/log/vmkernel.log
    YYYY-MM-DDThh:mm:ss.091Z cpu73:9417945)SunRPC: 4727: Socket space full. rpc queued. 0x8192ca0f

Storage I/O Control and Statistics Collection -

These issues occur when Storage I/O Control or Storage I/O statistics collection is enabled on NFS datastores.

Resolution

To address the underlying issue, engage the NFS vendor to validate the following parameters:

SunRPC.MaxConnPerIP
MaxQueueDepth

The vendor should provide recommended values for these parameters. VMware does not recommend making any changes to these parameters without guidance from the storage vendor.

For further information, refer to the Definition of the advanced NFS options and NFS connectivity issues on NetApp NFS filers on ESXi 6.x 7.x 8.x.

Workaround

These messages "File is being locked by a consumer" are expected in an environment with Storage I/O Control or Storage I/O Control statistics collection enabled for NFS datastores and can be safely ignored.

If Storage I/O Control is not required and the user wishes to prevent these messages, the following steps should be followed to disable both Storage I/O Control and Storage I/O Control statistics collection:

From the vSphere Web Client, navigate to Home -> Storage.
Select the NFS datastore and select to the Configure tab.
Select the Edit button next to Datastore Capabilities on the General link.
Deselect the Enable Storage I/O Control checkbox.
Select the checkbox for Disable Storage I/O statistics collection (applicable only if Storage I/O Control is disabled).
Click OK.
Perform this step for each NFS datastore in the environment.

Additional Information

For more information on Storage I/O control, Manage Storage I/O Resources with vSphere