Unable to mount NFS4.1 datastores with error "Permission denied"

Products

• VMware vSphere ESXi
• VMware vSphere ESXi 8.0

Issue/Introduction

• Mount NFS4.1 datastores to ESXi host fails with below error:
  
  An error occurered during host configuration: Operation failed. diagnostics report: Mount failed: Unable to complete Sysinfo operation. Please see the VMkernel.log file for more details: Permission denied.
  
  
• /var/run/log/vmkernel.log has below messages which also indicate lack of permission:
  
  WARNING: NFS41: NFS41FSGetRootFH:4480: Lookup <nfs fs name> failed for volume nfs41: Permission denied
WARNING: NFS41: NFS41FSCompleteMount:3976: NFS41FSGetRootFH failed: Permission denied
WARNING: NFS41: NFS41FSDoMount:4645: First attempt to mount the filesystem failed: Permission denied
WARNING: NFS41: NFS41_FSMount:4950: NFS41FSDoMount failed: Permission denied
  
  
• None of the vmkernel port ip address is in the same subnet as NFS server.
• It's intended to mount NFS4.1 datastore from a specific vmkernel port. Only the ip address of this vmkernel port is added to access control list and granted permission to access NFS4.1 datastore. 
• From the network trace captured when performing mount operation, it's observed that ESXi host reaches out to NFS4.1 datastore with management ip address but not the intended vmkernel port ip address. 

Environment

VMware vSphere ESXi 8.0.x

Cause

ESXi host reaches out to NFS4.1 datastore with management ip address instead of the intended NFS vmkernel port ip address. 

As permission is only granted to NFS vmkernel port ip address from storage, NFS server denies the mount request from ESXi host.

Resolution

If there is a vmkernel port ip address that's in the same subnet as NFS server, ESXi host will reach out to NFS server from this vmkernel port by default. So one way to workaround the issue is to configure a vmkernel port ip that's in the same subnet as NFS server and grant permission to this vmkernel port ip. 

If it's required to mount NFS4.1 datastore from a different subnet, either of the below configurations can be put in place to make it happen:

1. Vmkportbind feature can be leveraged to bind NFS traffic to the intended vmkernel port, procedure to follow: NFS 4.1 Datastores using Custom NFS TCP/IP Stack become inaccessible after upgrading to ESXi 8.0.2. Please be noted that vmkportbind feature for NFS4.1 is available on ESXi 8.0u3 and later. 
2. If the ESXi host is on version prior to ESXi 8.0u3 where vmkportbind option is not available, routing entry can be configured to direct NFS traffic to go through specific vmkernel port. For detailed steps to configure static routes please refer to: Configuring static routes for vmkernel ports on an ESXi host.

Additional Information

NFS server needs to be accessible from the intended vmkernel port ip, this can be confirmed with command:

vmkping -I <vmkX> <NFS server ip address>