Virtual machines on an NFS 4.1 datastore fail after the NFS 4.1 share recovers from an APD state with the error: The lock protecting VM.vmdk has been lost

Products

• VMware vSphere ESXi

Issue/Introduction

When the Network File System(NFS) 4.1 storage enters an All Paths Down (APD) state and then exits it after a grace period, you experience these symptoms:
Note: The grace period depends on the array vendor.

• powered on virtual machines that run on the NFS 4.1 datastore fail.
  
  
• After the NFS 4.1 share recovers from the APD condition, you see this message on the virtual machine summary page for virutal machine VM in the vSphere Web Client:
  
  The lock protecting VM.vmdk has been lost, possibly due to underlying storage issues. If this virtual machine is configured to be highly available, ensure that the virtual machine is running on some other host before clicking OK.
  
  
• After you click OK, crash files will be generated and the virtual machine will power off

Environment

VMware vSphere ESXi 6.x

Cause

This issue occurs because NFSv3 and v4 are different protocols with different behaviors. After the grace period (grace period differs depending on vendor), the NFS server flushes the client state.

Resolution

This is expected behavior in NFSv4 servers. Currently, there is no resolution or work around.