NFSv3 datastore mount failure with Limit exceeded error

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

NFSv3 datastore mount operation may fail with Limit exceeded error. The below log will be visible in /var/run/log/vmkernel.log, on the ESXi hosts where the NFSv3 datastore mount failed,

"NFS: Limit for maximum number of connections (256) reached"
 

Environment

VMware vSphere ESXi 8.0.1
VMware vSphere ESXi 8.0

Cause

With the nConnect feature (added in 8.0U1), connection sharing using the advance config option /SunRPC/MaxConnPerIP could not be used due to technical limitations. Hence, each NFSv3 datastore uses the number of connections specified during mount without sharing with other NFSv3 datastores. Combined with the number of reserved ports available to use for NFS, a limit on the total number of RPC connections used by NFSv3 has to be imposed, so that NFSv3 does not exhaust all available reserved ports.

The total number of RPC connections that can be used by NFSv3, including NFS Protocol Endpoints (NFS-PE mounts), is limited to 256.

Resolution

Currently there is no resolution to the issue.

Workaround:

With the new limitations in place, the following table defines the new maximum number of NFSv3 datastores that can be mounted with the given number of connections.

Maximum NFSv3 Datastore Count	Connections Per Datastore	Total Connections in Use
256	1	256
128	2	256
64	4	256
32	8	256

Recommendation for number of connections is still 1 for 8.0U1 release so maximum datastores that can be mounted with connection 1 remains same to 256. Please refer Preliminary support for nConnect feature added in ESXi NFS client for details.

Note:

1. Each vVol NFS PE mount uses 1 connection from the global pool, the total number of connections available will  be reduced by the number of NFS PE mounts
2. The advance config option /NFS/MaxVolumes is also required to be increased accordingly
3. The number of connections can be configured at datastore level. Hence, the total may vary in a heterogeneous environment where each NFSv3 datastore is configured with a different number of connections
4. After the number of connections reach 256, further NFSv3 datastore mounts (including NFS PE mount) will fail with limit exceeded error. At any time the number of connections in use can be obtained using the following command:

[root@esx-host] vsish -e get /vmkModules/nfsclient/info
NFS Connections Info
{
mountedVolumes:64
totalConnections:256
}