A virtual machine may hang or its guest operating system's file system may become corrupted when an NFS datastore loses connection due to a network interruption

Products

• VMware vCenter Server

Issue/Introduction

• When an NFS datastore loses its connection due to a network interruption, a virtual machine that was previously running on that NFS datastore will likely become unresponsive.
• In the vmkernel.log file, which is located at /var/log/, you will see the error:
  <YYYY-MM-DD> <Time> cpu11:2098577)WARNING: NFS: 338: Lost connection to the server 10.##.##.### mount point NAME1, mounted as 08d934a6-########-####-############ ("DATA_STORE_NAME")
• The file system of the virtual machine may become corrupted.

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x

Cause

This issue typically occurs when a virtual machine performs read/write I/O operations on an NFS datastore, and the NFS datastore's network connection is interrupted.

Resolution

To resolve this issue, it is essential to engage both the network team and the storage team to ensure that the network and storage infrastructure are stable.

You can verify whether the network and storage have returned to normal by using the following methods. Please validate from multiple ESXi servers to confirm that the issue is indeed related to the network or storage:

1. Ping the NFS server's IP address from the ESXi host

ping <NFS server IP>

2. change to the NFS datastore, and create or copy file to see if there is any error

cd <DATA_STORE_NAME>

touch <NEW_FILE_NAME> 

cp <OLD_FILE_NAME> <NEW_FILE_NAME>

Once the network and storage have returned to normal, the following actions can be taken to recover the virtual machines:

• The affected virtual machines require restarting to recover from the hung status
• If the ESXi host is unresponsive, restarting the ESXi host is required
• If a virtual machine experiences file system corruption, you must engage with the guest operating system's support team to recover from the corruption