ESXi hosts experience APD and NetPort lock failures with NetApp NFS storage - VMware vSphere 8.0

Products

• VMware vSphere ESXi

Issue/Introduction

ESXi 8.0.x hosts connected to NetApp NFS storage experience the following symptoms:

• Intermittent All Paths Down (APD) events lasting approximately 30-60 seconds.

• Host management plane (vCenter connectivity) may become temporarily unresponsive.

• Virtual Machine I/O is disrupted during the APD window.

• Datastores intermittently show extremely high latency (e.g., in ARI Operations or vRealize Operations).

• Host logs show "NetPort lock failures" or "TCP ZeroWindow" events from the NetApp storage array.

• NFS.MaxQueueDepth is set to 128 on ESXi 8.0.

ESXi Logs (/var/run/log/vmkernel.log):

• NetPort: 708: Failed to acquire port non-exclusive lock 0x400000f[Failure]

• Admission failure in path: host/user/vm.XXXX:vmmanon.XXXX

• WARNING: NetAppNfs: ... Performance has deteriorated. I/O latency increased...

• Evidence of TCP Zero Window flow control in network traces or storage controller logs.

Environment

• Hypervisor: VMware ESXi 8.0.x

• Storage: NetApp ONTAP (NFS v3 or v4.1)

• Networking: High-concurrency environments (e.g., NSX-T, high VM density)

Cause

The issue is caused by storage controller saturation triggered by high concurrent I/O requests.

1. ESXi NFS Max Queue Depth (128) allows the host to overwhelm the storage controller.

2. When the controller reaches its limit, it triggers TCP Zero Window flow control to pause incoming traffic.

3. The resulting backlog on the ESXi host networking stack leads to NetPort lock contention (kernel deadlock precursor) and memory admission failures as the host attempts to manage the connection state during high stress.

4. This resource exhaustion results in an APD state.

Resolution

To resolve this issue, reduce the I/O concurrency on the storage controller and increase the available memory for the networking stack.

Step 1: Reduce NFS Queue Depth

Lower the maximum queue depth from 128 to 64. This prevents the host from saturating the storage controller ports.

Run the following commands on the affected ESXi host(s):

esxcfg-advcfg -s 64 /NFS/MaxQueueDepth
esxcfg-advcfg -s 64 /NFS41/MaxQueueDepth

Step 2: Increase TCP/IP Heap Size

Ensure the host has sufficient heap memory to handle networking transitions and reconnection storms.

Run the following commands:

esxcfg-advcfg -s 32 /Net/TcpipHeapSize
esxcfg-advcfg -s 1024 /Net/TcpipHeapMax

(Note: While TcpipHeapSize 32 is often sufficient, TcpipHeapMax should be increased to 1024 MB to provide ample headroom.)

Step 3: Reboot the Host

These are advanced kernel parameters. A host reboot is mandatory for these changes to take effect.

Additional Information

• Troubleshooting ESXi host lockup where console and network are unresponsive but hardware logs are clean.