Virtual machine file cannot be found on recovered NFS datastore - dr.storageProvider.fault.ResolvedVmFileNotFound

Products

• VMware Live Recovery

Issue/Introduction

To assist in troubleshooting and resolving Test Failover or Planned Migration issues using NFS datastores with Site Recovery Manager.

Symptoms:

1. SRM Test Recovery fails to recover VMs on NFS datastores

2. Target NFS datastore is mounted successfully to all hosts.

3. Recovery plan fails on 'Create writable storage snapshot' step.

vmware-dr.log : 

-->       "</fault>",
-->       "            </Step>",
-->       "          </Children>",
-->       "        </Step>",
-->       "        <Step elapsedTime="00:00:00" endTime="2023-07-13T14:37:04Z" objectId="protected-vm-217316818" startTime="2023-07-13T14:37:04Z" status="error">",
-->       "          <Key>RecoveryStepRecoverVm.name</Key>",
-->       "          <Name>VirtualMachine</Name>",
-->       "          <fault>",
-->       "              <_type>dr.fault.MultipleFault</_type>",
-->       "              <faults>",
-->       "                  <_length>1</_length>",
-->       "                  <_type>vmodl.MethodFault[]</_type>",
-->       "                  <e id="0">",
-->       "                      <_type>dr.storageProvider.fault.ResolvedVmFileNotFound</_type>",
-->       "                      <filePath>[vmname] VirtualMachine/VirtualMachine.vmx</filePath>",
-->       "                      <msg/>",
-->       "    </e>",
-->       "  </faults>",
-->       "              <msg>Virtual machine file &apos;[vmname] VirtualMachine/VirtualMachine.vmx&apos; cannot be found on recovered datastore.</msg>",
-->       "</fault>",
-->       "          <Children>",
-->       "            <Step elapsedTime="00:00:00" endTime="2023-07-13T14:37:04Z" objectId="protected-vm-217316818" startTime="2023-07-13T14:37:04Z" status="error">",
-->       "              <Key>RecoveryStepConfigStorageOp.name</Key>",
-->       "              <Name>Configure storage</Name>",
-->       "              <fault>",
-->       "                  <_type>dr.storageProvider.fault.ResolvedVmFileNotFound</_type>",
-->       "                  <filePath>[vmname] VirtualMachine/VirtualMachine.vmx</filePath>",
-->       "                  <msg>Virtual machine file &apos;[vmname] VirtualMachine/VirtualMachine.vmx&apos; cannot be found on recovered datastore.</msg>",

Environment

VMware Site Recovery Manager 8.x

Cause

This error manifests when SRM receives "testFailoverStart responded" from SRA, it starts searching for VMX files in the newly mounted NFS datastores, but if all the VM snapshot copies aren't ready in the datastore, it will fail with this error. This happens because there is still some work in progress at the storage level. When the storage array completes all of its tasks, the host is able to access all the VMX files. 

Resolution

Go to SRM UI > Advanced Settings > Storage Provider 

• Configure "storageProvider.fetchDatastoreDelaySec" setting to 60 or 180 on both the vCenters (sites)

 

Change Storage Provider Settings

Now, you should be able to run recovery plans successfully. 

NOTE:

StorageProvider.hostRescanDelaySec is for FC & iSCSI datastores 
StorageProvider.fetchDatastoreDelaySec is for NFS datastores (Option was introduced in SRM 8.2)

Additional Information

Impact/Risks:

A Test failover or Planned Migration & Disaster Recovery task in Site Recovery Manager (SRM) fails.