NFS datastore mount fails with "Unable to connect to NFS server: VSI node (5001:)".

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

NOTE: The IPs used in this KB are dummy IPs and they do not belong to any environment. 

• Attempting to mount the NFS datastore from vCenter fails with error :
  "An error occurred during host configuration: Operation failed, diagnostics report: Mount failed: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Unable to connect to NFS server: VSI node (5001:)"
  
  
• The /var/run/log/vmkernel.log file on the affected host confirms that the mount failed since ESXi host failing to connect to NFS server.
  YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2098666 opID=4660f3ca)NFS: 349: Command: (mount) Server: (##.##.##.##) IP: (##.##.##.##) Connections: (1) Vmknic: (None)  Path: (########) Label: (######) Options: (None)
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2098666 opID=4660f3ca)StorageApdHandler: 965: APD Handle 0dd43dda-e53b3948 Created with lock[StorageApd-0x431a11f582b0]
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2098666 opID=4660f3ca)StorageApdHandler: 1051: Freeing APD handle 0x431a11f582b0 [0dd43dda-e53b3948]
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2098666 opID=4660f3ca)StorageApdHandler: 1135: APD Handle freed!
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu20:2098666 opID=4660f3ca)NFS: 3500: NFS mount failed for ##.##.##.##:/######/###### volume ######. Status: Unable to connect to NFS server.
  
  
• The /var/run/log/hostd.log file on the ESXi host also confirms that the host is unable to connect to NFS server.
  YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098666]: [Originator@6876 sub=Solo.Vmomi opID=####-####-####-####-##:####-##-##-### sid=##### user=vpxuser:###\######] Result:
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: --> (vim.fault.PlatformConfigFault) {
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->          key = "com.vmware.esx.hostctl.default",
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->          arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->                key = "reason",
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->                value = "Mount failed: Unable to complete Sysinfo operation.  Please see the VMkernel log file for more details.: Unable to connect to NFS server: VSI node (5001:) "
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->             }
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->          ],
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->       }
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->    ],
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->    text = "",
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: -->    msg = ""
YYYY-MM-DDTHH:MM.SSSZ In(166) Hostd[2098626]: --> }


• A ping from the VMkernel adapter to the target NFS IP fails with 100% packet loss
  vmkping -I vmk2 -d -s 1472 00.00.00.96
PING 00.00.00.96 (00.00.00.96): 1472 data bytes
--- 00.00.00.96 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The target NFS server's IP address and the ESXi host's VMkernel adapter IP address are on different IP subnets. Due to this ESXi host failed to establish a connection to the NFS storage.

Cause Validation:   


• Check the VMkernel adapter configuration and note down the IP and Netmask used:
  esxcfg-vmknic -l
Interface  Port Group/DVPort/Opaque Network        IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type                NetStack
vmk2       ##                                      IPv4      00.00.00.96                             255.255.255.224  ##.##.##.##     ##:##:##:##:##:## 1500    #####     true    STATIC              defaultTcpipStack


• Verify the routing table on the ESXi host from the command line to determine the valid IP range based on the configured "Netmask" and "Network".
  esxcli network ip route ipv4 list
Network        Netmask          Gateway        Interface  Source
-------------  ---------------  -------------  ---------  ------
00.00.00.64    255.255.255.224  0.0.0.0        vmk2       MANUAL

In the above example, the network is "00.00.00.64" with a netmask of "255.255.255.224". The usable IP range falls between "00.00.00.65" to "00.00.00.94". Since the target IP "00.00.00.96" which falls outside the subnet mask range, the NFS mount failed.

Resolution

• Use an NFS server IP address that falls within the valid range of the existing subnet mask.
  OR
• Update the vmkernel adapter subnet mask to accommodate the target NFS server's IP address.