Power on Virtual machine on NFSv3 fails with -"An error occurred while opening configuration file "/vmf/volumes/datastore/VM/VM.vmx": Insufficient permission to access the file"

Products

• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• Power on virtual machine fails with below errors
   State: "Failed-An error occurred while opening configuration file "/vmf/volumes/datastore/VM/VM.vmx":Insufficient permission to access the file
  Errors 

               An error occurred while opening configuration file "/vmf/volumes/datastore/VM/VM.vmx"
                   Insufficient permission to access the file

             Cannot open the configuration file "/vmf/volumes/datastore/VM/VM.vmx"
             Failed to start the virtual machine (error -18).

 

• During VM snapshot task ,the  VM on NFSv3 datastore powers off the VM . The vmware.log of VM shows similar entries as below

yyyy-mm-ddThh:mm:ss No(00) vcpu-0 - ConfigDB: Setting scsi0:0.fileName = "VM-000001.vmdk"
yyyy-mm-ddThh:mm:ssWa(03) vcpu-0 - FILE: FileIO_Lock on '/vmfs/volumes/datastore/VM/VM.vmx' failed: Permission denied
yyyy-mm-ddThh:mm:ss Wa(03) vcpu-0 - FILE: FileIO_Lock on '/vmfs/volumes/datastore/VM/VM.vmx' failed: Permission denied
yyyy-mm-ddThh:mm:ss In(05) vcpu-0 - Msg_Reset:
yyyy-mm-ddThh:mm:ss In(05) vcpu-0 - [msg.configdb.open] An error occurred while opening configuration file "/vmfs/volumes/datastore/VM/VM.vmx": Insufficient permission to access the file.
yyyy-mm-ddThh:mm:ss In(05) vcpu-0 - [msg.configdb.open] An error occurred while opening configuration file "/vmfs/volumes/datastore/VM/VM.vmx": Insufficient permission to access the file.
yyyy-mm-ddThh:mm:ss In(05) vcpu-0 - ----------------------------------------
yyyy-mm-ddThh:mm:ssIn(05) vcpu-0 - SNAPSHOT: SnapshotConfigWrite: Failed to write config file '/vmfs/volumes/datastore/VM/VM.vmx'.
yyyy-mm-ddThh:mm:ss Cr(01) vcpu-0 - PANIC: Critical snapshot error.
yyyy-mm-ddThh:mm:ss In(05) vcpu-0 - Backtrace:
yyyy-mm-ddThh:mm:ss verbose hostd[2099613] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/VM/VM.vmx <opID> user=vpxuser Commit Vigor batch operation failed: vim.fault.GenericVmConfigFault
yyyy-mm-ddThh:mm:ssverbose hostd[2099613] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/VM/VM.vmx <opID> user=vpxuser Commit Vigor batch operation message: Unable to write VMX file: /vmfs/volumes/datastore/VM/VM.vmx.
--> Insufficient permission to access the file
[....]
yyyy-mm-ddThh:mm:ss info hostd[2099613] [Originator@6876 sub=Libs <opID> user=vpxuser SNAPSHOT: SnapshotDeleteVMInt failed: Insufficient permission to access the file (3)
yyyy-mm-ddThh:mm:ss verbose hostd[2099613] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/VM/VM.vmx 
<opID>= Delete VM translated error to vim.fault.CannotAccessFile
yyyy-mm-ddThh:mm:ss verbose hostd[2099613] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/datastore/VM/VM.vmx <opID> user=vpxuser Delete VM message: Failed to delete the virtual machine, '/vmfs/volumes/datastore/VM/VM.vmx'.


•  vMotion  of VM fails and vmware.log will show similar entries as below

yyyy-mm-ddThh:mm:ss Wa(03) vmx - FILE: FileIO_Lock on '/vmfs/volumes/datastore/VM/VM.vmx ' failed: Permission denied
yyyy-mm-ddThh:mm:ss Wa(03) vmx - FILE: FileIO_Lock on '/vmfs/volumes/datastore/VM/VM.vmx ' failed: Permission denied
yyyy-mm-ddThh:mm:ss In(05) vmx - Msg_Reset:
yyyy-mm-ddThh:mm:ss In(05) vmx - [msg.configdb.open] An error occurred while opening configuration file "/vmfs/volumes/datastore/VM/VM.vmx ": Insufficient permission to access the file.
yyyy-mm-ddThh:mm:ss In(05) vmx - [msg.configdb.open] An error occurred while opening configuration file "/vmfs/volumes/datastore/VM/VM.vmx ": Insufficient permission to access the file.
yyyy-mm-ddThh:mm:ss In(05) vmx - ----------------------------------------
yyyy-mm-ddThh:mm:ss Wa(03) vmx - Migrate: Failed to write out config file.
yyyy-mm-ddThh:mm:ssIn(05) vmx - Migrate: Caching migration error message list:


•      Performing network packet capture shows "NFS3ERR_ACCES" as the response from NFS server

     Packet capture command - tcpdump-uw -i vmk0 -vv -n host <NFS aarray IP> -w /test.pcap

Cause

This is caused when Fpolicy is enabled on the Netapp NFS Server.

Resolution

This is not due to ESXi NFS client but from the NFS server array. Contact Netapp Support for further assistance.