NFS datastores are inaccessible post restoring the site from maintenance window.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• Post restoring the site from maintenance window, NFS datastores shows inaccessible in vcenter.
  
• Verify the mount status of the NFS datastores using the command "esxcli storage nfs list".
  esxcli storage nfs list
Volume Name             Host                        Share                    Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
----------------------  --------------------------  -----------------------  ----------  -------  ---------  -----  ---------------------
##############-07       ##########################  /############ -07            false    false      false  false  Unknown
##############-06       ##########################  /############ -06            false    false      false  false  Unknown
##############-05       ##########################  /############ -05            false    false      false  false  Unknown
##############-01       ##########################  /############ -01            false    false      false  false  Unknown
##############-02       ##########################  /############ -02            false    false      false  false  Unknown
##############-08       ##########################  /############ -08            false    false      false  false  Unknown
##############-03       ##########################  /############ -03            false    false      false  false  Unknown
##############-04       ##########################  /############ -04            false    false      false  false  Unknown
  
  From the above output, it is confirmed that the NFS datastores are unmounted from the ESXi host.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Datastores are unmounted during the site shut down. Mount restore is required to establish the connectivity.

Cause validation:

• Logs will not report any issues.
• Run the command "esxcfg-vswitch -l" to identify the vSwitch used for NFS traffic and check the MTU configured on it.
  esxcfg-vswitch -l
  

  DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
Switch-name      ####        8           512               1500    vmnic1,vmnic0

    DVPort ID                               In Use      Client
  ####                                    1           vmnic0
  ####                                    1           vmnic1
  ####                                    1           vmk0
  ####                                    1           vmk1
  ####                                    1           vmk2

  In the above example, it is confirmed that vmnic0 and vmnic1 are used for NFS communication. vSwicth is configured with 1500 MTU. 
  
  
• Run the command "esxcfg-vmknic -l" to verify the MTU set on the VMkernel adapter (vmk)
  esxcfg-vmknic -l
vmk2       266                                     IPv4      ###.###.###.##                           ###.###.###.###       ###.###.##.###     ##:##:##:##:##:##    #####     true    STATIC              defaultTcpipStack
  
  In the above example, it is confirmed that vmk2 is configured with MTU 1500.
• Run the command "esxcfg-nics -l" to confirm the MTU configured on the physical nics (vmnics).
  esxcfg-nics -l
Name    PCI          Driver      Link Speed      Duplex MAC Address       MTU    Description
vmnic0  ####:##:##.# nenic       Up   50000Mbps  Full   ##:##:##:##:##:## 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1  ####:##:##.# nenic       Up   50000Mbps  Full   ##:##:##:##:##:## 1500   Cisco Systems Inc Cisco VIC Ethernet NIC
  
  In the above example, it is confirmed that vmnics are configured with MTU 1500.
  
  
• Ping the NFS target from the host using 1500 MTU.
  

  Ping results for first target.
  
  vmkping -I vmk2 -d -s 1472 ###.###.###.#01
PING ###.###.###.#01 (###.###.###.#01): 1472 data bytes
1480 bytes from ###.###.###.#01: icmp_seq=0 ttl=63 time=1.025 ms
1480 bytes from ###.###.###.#01: icmp_seq=1 ttl=63 time=0.513 ms
1480 bytes from ###.###.###.#01: icmp_seq=2 ttl=63 time=0.734 ms

  --- ###.###.###.#01 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.513/0.757/1.025 ms

  Ping results for second target.
  
  vmkping -I vmk2 -d -s 1472 ###.###.###.#02
PING ###.###.###.#02 (###.###.###.#02): 1472 data bytes
1480 bytes from ###.###.###.#02: icmp_seq=0 ttl=63 time=0.718 ms
1480 bytes from ###.###.###.#02.136: icmp_seq=1 ttl=63 time=0.672 ms
1480 bytes from ###.###.###.#02: icmp_seq=2 ttl=63 time=0.574 ms

  --- ###.###.###.#02 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.574/0.655/0.718 ms

  Based on the results from the steps above, it is confirmed that the host is able to communicate with the NFS target
  
  
• Try to mount the datastore using the command "esxcli storage nfs add -H <hostname> -s <sharename> -v <volumename>". Mount fails with error "Volume with label " already exists.
  
  esxcli storage nfs add -H <Host> -s <Share> -v <Volume name>
Volume with label : ##############-01 already exists.

Resolution

• Restore the NFS mount using the following command "esxcfg-nas -r".
• NFS datastores will be mounted post restoring the NFS mount.
  esxcli storage nfs list
Volume Name             Host                        Share                    Accessible  Mounted  Read-Only   isPE  Hardware Acceleration
----------------------  --------------------------  -----------------------  ----------  -------  ---------  -----  ---------------------
##############-07       ##########################  /############ -07            true    true      false      false  Unknown
##############-06       ##########################  /############ -06            true    true      false      false  Unknown
##############-05       ##########################  /############ -05            true    true      false      false  Unknown
##############-01       ##########################  /############ -01            true    true      false      false  Unknown
##############-02       ##########################  /############ -02            true    true      false      false  Unknown
##############-08       ##########################  /############ -08            true    true      false      false  Unknown
##############-03       ##########################  /############ -03            true    true      false      false  Unknown
##############-04       ##########################  /############ -04            true    true      false      false  Unknown