Virtual machines restored from Tintri VMstore to an NFS 4.1 datastore are unable to power ON

Products

• VMware vSphere ESXi

Issue/Introduction

Virtual machines restored from Tintri VMstore do not power ON. 

Power On virtual machine
VMNameisTest
File /vmfs/volumes/d76#####-bf88####-0000-000000000000/VMNameisTest/VMNameisTest.vmx was not found Cannot open file "/vmfs/volumes/d76#####-bf88####-0000-000000000000/VMNameisTest/VMNameisTest.vmx ": 116 (Stale file handle). Unable to load configuration file "/vmfs/volumes/d76#####-bf88####-0000-000000000000/VMNameisTest/VMNameisTest.vmx ": Unable to load configuration file "/vmfs/volumes/d76#####-bf88####-0000-000000000000/VMNameisTest/VMNameisTest.vmx".
Powering on the new Virtual Machine
[email protected]
5 ms
07/29/2024, 3:38:14 PM
07, 24, PM


VMkernel.log:

2024-07-29T10:08:14.120Z In(182) vmkernel: cpu1:2098519 opID=281cf40a)NFS41: NFS41FileOpenFile:3008: Open of 0x4308e7e321a0 name VMNameisTest.vmx with openFlags 0x1 failed: Stale file handle
2024-07-29T10:08:14.120Z Wa(180) vmkwarning: cpu1:2098519 opID=281cf40a)WARNING: NFS41: NFS41FileOpOpenFile:3743: Open of obj 0x4308e7e321a0 name VMNameisTest.vmx (fhID 0x31515af) failed: Stale file handle
2024-07-29T10:08:14.121Z In(182) vmkernel: cpu1:2098519 opID=281cf40a)NFS41: NFS41FileOpenFile:3008: Open of 0x4308e7e321a0 name VMNameisTest.vmx with openFlags 0x1 failed: Stale file handle
2024-07-29T10:08:14.121Z Wa(180) vmkwarning: cpu1:2098519 opID=281cf40a)WARNING: NFS41: NFS41FileOpOpenFile:3743: Open of obj 0x4308e7e321a0 name VMNameisTest.vmx (fhID 0x31615af) failed: Stale file handle
2024-07-29T10:08:14.121Z In(182) vmkernel: cpu1:2098519 opID=281cf40a)NFS41: NFS41FileOpenFile:3008: Open of 0x4308e7e321a0 name VMNameisTest.vmx with openFlags 0x1 failed: Stale file handle
2024-07-29T10:08:14.121Z Wa(180) vmkwarning: cpu1:2098519 opID=281cf40a)WARNING: NFS41: NFS41FileOpOpenFile:3743: Open of obj 0x4308e7e321a0 name VMNameisTest.vmx (fhID 0x31715af) failed: Stale file handle
2024-07-29T10:08:14.122Z In(182) vmkernel: cpu5:2098519 opID=281cf40a)NFS41: NFS41FileOpenFile:3008: Open of 0x4308e7e321a0 name VMNameisTest.vmx with openFlags 0x1 failed: Stale file handle

 

Environment

VMware ESXi 8.0.3

Cause

Tintri VMstore:

During a full VM restore with Tintri, the .vmx1 file is replaced by .vmx2, which also initiates the VM's power ON. This process is managed by Tintri, so ESXi lacks information about it.
 
ESXi Host:

When a VM powers ON, ESXi accesses the .vmx file. The NFS41 DNLC cache still has a valid .vmx1 entry, allowing the lookup to be served from the cache, which contains the 'file handle' for .vmx1. However, when ESXi uses this file handle to open the file, it encounters a STALE FILE HANDLE error because Tintri has replaced .vmx1 with .vmx2.
 
The solution to drop the DNLC cache upon receiving a stale file handle from NFS and retrying from the upper layer will cause NFS41 to send a LOOKUP request, retrieving the latest file handle.

Resolution

This issue will be fixed in a future release of VMware ESXi 8.0.3 Patch 6 & ESXi 9.0.1