Error: "Unable to reverse replication... Error while searching (hms.DatastorePath)" during SRM reprotect task
search cancel

Error: "Unable to reverse replication... Error while searching (hms.DatastorePath)" during SRM reprotect task

book

Article ID: 447637

calendar_today

Updated On:

Products

VMware Site Recovery Manager VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • A virtual machine reprotect task hangs at 25 percent progress and fails to complete after a successful Planned Migration.

  • The vSphere Replication Management Server (VRMS) user interface displays the following error:

"Unable to reverse replication for the virtual machine '########'. A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error while searching (hms.DatastorePath) dynamicType = null, dynamicProperty = null, datastore = MoRef: type = Datastore, value = datastore-########, serverGuid = null, path = ########, fileName = ########.vmdk '."

  • Verifying the Host-Based Replication (HBR) network status on the source ESXi host, the vmkernel.log file indicates that the local HBR address is unavailable:

2026-07-09T13:38:41.345Z In(182) vmkernel: cpu20:5526668)Hbr: 377: Failed to connect to 127.0.0.2 (groupID=GID-31f9ec63-3d10-3b91-8117-#########): Address not available
2026-07-09T13:38:41.345Z In(182) vmkernel: cpu20:5526668)Hbr: 5355: Failed to establish connection to [127.0.0.2]:32032 (groupID=GID-31f9ec63-3d10-3b91-8117-#########): Address not available

  • Validating the remote replication configuration, the hms.log file reports a generic error where the datastore path is unable to be found:

2026-07-10 10:36:25.514 ERROR com.vmware.hms.job [hms-main-thread-1058] (..job.impl.PrimaryVc2VcConfigureReplicationWorkflow) [operationID=e0fb9847-2fe3-4b5b-92b8-b6aa40c85417-reprotect:e63f:248d:71d2-HMS-255992,sessionID=46741167, operationID=e0fb9847-2fe3-4b5b-92b8-b6aa40c85417-reprotect:e63f:248d:71d2-HMS-255992,sessionID=46741167, task=HTID-9067e005-694a-46bf-b6a5-68fc890771c2, task=HTID-8ea48fcd-ca1d-4f11-bb64-81216c01a12a] | Failed remote replication configuration
com.vmware.vim.binding.hms.fault.HmsMethodFault: A generic error occurred in the vSphere Replication Management Server. Exception details: 'Error while searching (hms.DatastorePath) {
   dynamicType = null,
   dynamicProperty = null,
   datastore = MoRef: type = Datastore, value = datastore-####, serverGuid = null,
   path = datastore_name,
   fileName = vm_name.vmdk
}'.

  • Verifying the replication appliance health monitor, the hms.log shows one ESXi host hbrsrvuw service is disconnected but healthy:

2026-07-10 10:06:00.931 DEBUG com.vmware.hms.vlsi.step.InvocationStep [tcweb-26] (..vlsi.step.InvocationStep) [] | vmomiOp-finish [method: HmsTask.getInfo; target: HTID-128f725b-7e61-4f9c-afc9-677476065dad; user: SRM-remote-5adb6503-1540-498f-a8e3-########@#####.LOCAL; client: 10.#.#.##:42216; operationID=e0fb9847-2fe3-4b5b-92b8-b6aa40c85417-reprotect:e63f:248d:71d2-HMS-255992; sessionID=142FC0B0]; time: 1 ms
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 0 in-maintenance hbrsrvuws:
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 1 disconnected hbrsrvuws: 10.#.#.##(37383638-3330-5a43-3338-########),
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 0 removed hbrsrvuws:
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 0 user decommissioned:
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 0 disallowed hbrsrvuws:
2026-07-10 10:06:01.046 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | 3 of 4 healthy hbrsrvuws
2026-07-10 10:06:01.056 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | replications health: checking 4 replications...
2026-07-10 10:06:01.056 DEBUG com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | replications health: fixable:0; unfixable:0; healthy:4; total:4
2026-07-10 10:06:01.056 INFO  com.vmware.hms.hbrsrvuw.HbrsrvuwMonitor [hms-main-scheduled-thread-24] (..hms.hbrsrvuw.HbrsrvuwMonitor) [operationID=#315683] | rebalance not needed

  • Validating the connection status to the problematic ESXi host, an unexpected 503 HTTP status code is returned during the connectivity check:

2026-07-10 10:05:55.090 ERROR com.vmware.hms.net.hbr.ping.svr.37383638-3330-5a43-3338-######### [hms-ping-scheduled-thread-7] (..net.impl.VmomiPingConnectionHandler) [operationID=a883eeb7-ac82-47ef-9ffa-1faffff3c274-HMS-PING] | Ping for server 10.#.#.##:443/hbr for session: N/A failed:
com.vmware.vim.vmomi.client.common.UnexpectedStatusCodeException: Unexpected status code: 503

 

Environment

  • vSphere Replication 9.x
  • VMware Live Recovery 9.x
  • VMware ESXi 8.x
  • VMware ESXi 9.x

Cause

  • Host-based replication (HBR) communication is interrupted on a specific ESXi host due to underlying storage connectivity failures. This storage disconnection prevents the datastore path from being resolved, which causes the reprotect configuration task to fail and hang at 25 percent.
  • The underlying storage failure is validated by reviewing the vmkernel.log file on the disconnected ESXi host. Verifying the storage multipathing and device I/O status, the logs confirm that the storage device is inaccessible. The storage paths are observed failing, resulting in an inability to open the target volume:

2026-07-10T05:42:08.356Z In(182) vmkernel: cpu45:2098173)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x1a (0x45d9c07578c0, 0) to dev "naa.############" on path "vmhba1:C0:T0:L2" Failed:
2026-07-10T05:42:08.356Z In(182) vmkernel: cpu45:2098173)NMP: nmp_ThrottleLogForDevice:3898: H:0x8 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4308831722b0 CmdSN 0x13f20f
2026-07-10T05:42:08.357Z Wa(180) vmkwarning: cpu45:2098173)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.############" state in doubt; requested fast path state update...
2026-07-10T05:42:08.357Z In(182) vmkernel: cpu45:2098173)ScsiDeviceIO: 4686: Cmd(0x45d9c07578c0) 0x1a, CmdSN 0x13f20f from world 0 to dev "naa.############" failed H:0x8 D:0x0 P:0x0
2026-07-10T05:42:08.556Z In(182) vmkernel: cpu2:4245256)qlnativefc: vmhba1(af:0.0): qlnativefcEhAbort:2959:C0:T0:L2: Abort command succeeded -- 1
2026-07-10T05:42:08.557Z Wa(180) vmkwarning: cpu44:2099461)WARNING: ScsiDeviceIO: 13044: READ CAPACITY on device "naa.############" from Plugin "NMP" failed. I/O error
2026-07-10T05:42:08.557Z In(182) vmkernel: cpu44:2099461)LVM: 6450: Could not open device naa.############:1, vol [6a03170c-57ff255e-7c34-############, 6a03170c-57ff255e-7c34-############, 1]: No such target on adapter

  • Verifying further Fibre Channel communication errors, command timeouts and aborts are registered on the storage adapter:

2026-07-10T05:42:48.780Z In(182) vmkernel: cpu45:2099293)qlnativefc: vmhba1(af:0.0): qlnativefcAsyncEvent:1277:Discard RND Frame -- ffff 0001 0000.
2026-07-10T05:42:58.780Z In(182) vmkernel: cpu45:2098159)qlnativefc: vmhba1(af:0.0): qlnativefcAsyncEvent:1277:Discard RND Frame -- ffff 0001 0000.
2026-07-10T05:42:59.240Z In(182) vmkernel: cpu44:3738801)qlnativefc: vmhba1(af:0.0): qlnativefcTaskMgmt:2424:Task Mgmt abort on serial num 0
2026-07-10T05:42:59.240Z In(182) vmkernel: cpu44:3738801)qlnativefc: vmhba1(af:0.0): qlnativefcEhAbort:2913:SCSI command timeout counter incremented to 111363
2026-07-10T05:42:59.240Z In(182) vmkernel: cpu44:3738801)qlnativefc: vmhba1(af:0.0): qlnativefcEhAbort:2916:qlnativefcEhAbort: aborting sp 0x45b977000b80 handle 447 from RISC. serialNumber=0, Command timeout=10 sec.

Resolution

To resolve this issue and allow the reprotect task to complete:

  1. Identify the problematic ESXi host that is generating the 503 HTTP status code in the hms.log file.

  2. Place the affected ESXi host into Maintenance Mode in the vSphere Client.

  3. Execute the reprotect task again from the vSphere Replication Management interface.

  4. Engage storage vendor or internal storage administration team to permanently resolve the underlying physical storage connectivity and multipathing issues on the isolated ESXi host.

Powered by Wolken Software