SRM Placeholder creation fails with error "Unable to access file Datastore path"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

1. Virtual Machines in Protection group appears with warning on placeholder VMs,
   
   
   
   Error : Placeholder Create Error : Unable to access file Datastore path.
   
   
2. Recreating placeholder fails with below error,
   
   
   
   Error : Unable to protect VM due to inventory conflicts. The resource pool and folder belongs to separate datacenters.

Environment

VMware Liver Recovery 9.x

Cause

The cause of this issue is that the ESXi hosts at the recovery site have lost read/write access to the designated placeholder datastore, causing SRM's attempts to create the Placeholder VM configuration files to time out and fail.

From the logs /opt/vmware/support/logs/srm/vmware-dr.log, VM placeholder creation timeout.

####-##-##T##:##:09.892Z verbose vmware-dr[01182] [SRM@6876 sub=ProductionVmManagerImpl opID=a267cfda] [UpdatePlaceholderVmInfo] Updating PlaceholderVmInfo for protected VM 'protected-vm-#######':
--> (dr.replication.ProtectedVm.PlaceholderVmInfo) {
-->    vm = <unset>,
-->    folder = <unset>,
-->    folderAutoConfigured = true,
-->    computeResource = <unset>,
-->    resourcePool = <unset>,
-->    rpAutoConfigured = true,
-->    host = <unset>,
-->    datacenter = <unset>,
-->    placeholderCreationFault = (dr.fault.Timedout) {
-->       faultCause = (vmodl.MethodFault) null,
-->       faultMessage = <unset>,
-->       timeout = 480
-->       msg = "Operation timed out: 480 seconds"
-->    },
-->    repairNeeded = true
--> }

####-##-##T##:##:09.893Z verbose vmware-dr[01216] [SRM@6876 sub=ProductionVmManagerImpl opID=d4e41147] [UpdatePlaceholderVmInfo] Updating PlaceholderVmInfo for protected VM 'protected-vm-####':
--> (dr.replication.ProtectedVm.PlaceholderVmInfo) {
-->    vm = 'vim.VirtualMachine:########-####-####-####-############:vm-#####',
-->    folder = 'vim.Folder:########-####-####-####-############:group-v#####',
-->    folderAutoConfigured = true,
-->    computeResource = 'vim.ClusterComputeResource:########-####-####-####-############:domain-######',
-->    resourcePool = 'vim.ResourcePool:########-####-####-####-############:resgroup-#####',
-->    rpAutoConfigured = true,
-->    host = 'vim.HostSystem:########-####-####-####-############:host-#####',
-->    datacenter = 'vim.Datacenter:########-####-####-####-############:datacenter-#####',
-->    placeholderCreationFault = (vmodl.MethodFault) null,
-->    repairNeeded = false
--> }
####-##-##T##:##:09.893Z verbose vmware-dr[01216] [SRM@6876 sub=ProductionVmManagerImpl opID=d4e41147] [UpdatePlaceholderVmInfo] Detected no changes in PlaceholderVmInfo for protected VM 'protected-vm-####'

####-##-##T##:##:09.894Z verbose vmware-dr[01056] [SRM@6876 sub=ProductionVmManagerImpl opID=cc9a8985] [UpdatePlaceholderVmInfo] Updating PlaceholderVmInfo for protected VM 'protected-vm-#######':
--> (dr.replication.ProtectedVm.PlaceholderVmInfo) {
-->    vm = <unset>,
-->    folder = <unset>,
-->    folderAutoConfigured = true,
-->    computeResource = <unset>,
-->    resourcePool = <unset>,
-->    rpAutoConfigured = true,
-->    host = <unset>,
-->    datacenter = <unset>,
-->    placeholderCreationFault = (vim.fault.CannotAccessFile) {
-->       faultCause = (vmodl.MethodFault) null,
-->       faultMessage = <unset>,
-->       file = "ds:///vmfs/volumes/########-########-####-############/##############-########-phVm"
-->       msg = "Unable to access file ds:///vmfs/volumes/########-########-####-############/##############-########-phVm"
-->    },
-->    repairNeeded = true
--> }
####-##-##T##:##:09.894Z verbose vmware-dr[01056] [SRM@6876 sub=ProductionVmManagerImpl opID=cc9a8985] [UpdatePlaceholderVmInfo] Detected no changes in PlaceholderVmInfo for protected VM 'protected-vm-#######'

• The above log events indicates, SRM is trying to create the Placeholder VM files, but the ESXi host at the recovery site cannot access the placeholder datastore. Because the host cannot write the placeholder configuration file (.vmx) to that specific datastore, the creation fails, and the VM is flagged with repairNeeded = true. The timeout in log was due to the vCenter server waiting for the storage to respond before finally giving up.

Resolution

1. Identify the Datastore: Log into the recovery site vCenter and identify which datastore configured for placeholder datastore and corresponds to the UUID from log/Error. (This is the datastore configured in SRM for placeholders).

2. Check Storage Health engage storage vendor if necessary:

   1. Is the datastore currently disconnected, unmounted, or in an APD (All Paths Down) state on the recovery hosts?

   2. Is the datastore 100% full?

   3. Are there underlying SAN/NAS connectivity issues between the recovery ESXi hosts and this specific LUN/Volume?

3. Repair the Protection Group: Once datastore access is restored and the hosts can read and write to it again, please return to the Site Recovery Manager (SRM) interface. From there, click Recreate Placeholder for the affected virtual machines, or alternatively, repair the entire Protection Group.

4.