ESXi Host Socket Tracker and NFS Lock Contention
search cancel

ESXi Host Socket Tracker and NFS Lock Contention

book

Article ID: 451638

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article addresses intermittent kernel memory allocation warnings (SetSockOpt SO_NETTRACKER failed with status Out of memory) on ESXi hosts observed during periods of high cluster workload or storage contention.
The root cause is identified as NFS datastore lock contention triggered by Storage I/O Control (SIOC).

Symptoms

ESXi hosts report persistent memory allocation warnings in /var/run/log/vmkwarning.log or /var/run/log/vmkernel.log:

  • WARNING: MigrateNet: <Session_ID> S: SetSockOpt SO_NETTRACKER of sock <Socket_Address> failed with status Out of memory
  • WARNING: MigrateNet: vm <World_ID>: 3303: SetSockOpt SO_NETTRACKER of sock <Socket_Address> status Out of memory

Associated symptoms include:

  • vMotion, vSphere Replication (HBR), or VM provisioning failures or timeouts.
  • Distributed Virtual Switch  port assignment failures with locking alerts:
    • WARNING: DVSDev: DVSDevPortAdd:526: failed to acquire lock <DVS_UUID>
  • Log indications of NFS file lock contention:
    • WARNING: NFSLock: 4156: File is being locked by a consumer on host <Host_FQDN> with exclusive lock.
    • NFSLock: 5141: failed to get lock on file slotsfile(.lck-<Lock_ID>) ... Busy.
  • Rapid, repeated vSphere HA (fdm.log) inventory reconciliation attempts.

Environment

VMware vSphere ESXi

Cause

The issue is caused by a resource exhaustion event triggered by NFS datastore lock contention:

The Storage I/O Control (SIOC) or statistics collection is enabled on shared NFS datastores, multiple hosts in a cluster concurrently attempt to acquire exclusive .lck file locks on the shared slotsfile.
This contention stalls low-level storage I/O worker threads in the VMkernel.
DVS Port Lockup this is a delayed I/O queue prevents dynamic virtual switch operations from acquiring the locks causing port attachment routines to hang.
Fault Domain Manager (FDM) retry loops with migration/replication socket tracking attempts, exhaust the VMkernel network heap (netPool), preventing SO_NETTRACKER socket options from allocating memory.

Resolution

 1: Disable SIOC and Statistics Collection on NFS Datastores

  1. Log in to the vSphere Client.
  2. Navigate to Storage in the inventory.
  3. Select the affected NFS datastore.
  4. Navigate to the Configure tab > General (under Datastore Capabilities).
  5. Click Edit.
  6. Deselect Enable Storage I/O Control.
  7. Select Disable Storage I/O statistics collection.
  8. Click OK to apply changes.

2: Reset Virtual Switch Ports and Replication Services

If active VMs are trapped in a locked DVS port state:

  1. Connect to the affected ESXi host via SSH.
  2. Locate the World ID of the affected virtual machine: esxcli vm process list | grep -A 1 -i "<VM_Name_Pattern>"
  3. Force-terminate the locked process: esxcli vm process kill --type=force --world-id=<World_ID>
  4. Restart local management services: /etc/init.d/hbrsrv restart /etc/init.d/hostd restart /etc/init.d/vpxa restart

3: Refresh Cluster Availability

  1. In vSphere Client, select the Cluster.
  2. Navigate to Configure > vSphere Availability.
  3. Click Edit, disable vSphere HA, and click OK.
  4. Wait 1–2 minutes for the fdm service to unbind across all hosts.
  5. Click Edit, re-enable vSphere HA, and click OK.

Additional Information

The vmkernel.log file shows message "failed to get lock on file slotsfile" on NFS datastores when Storage I/O Control (SIOC) statistics collection is enabled

Powered by Wolken Software