Multiple VMs show “RPO Violation” with error “No host can be used to access datastore path <target_datastore>”
search cancel

Multiple VMs show “RPO Violation” with error “No host can be used to access datastore path <target_datastore>”

book

Article ID: 420703

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:

  • Multiple VMs will be in “RPO Violation” state with the error message: “No host can be used to access datastore path <target_datastore>”

  • Protection Groups gets into "Not configured" state leading protected VM into invalid state.

    Invalid: Virtual Machine 'VM_Name'  is no longer protected. The Session is not authenticated.

Environment

  • vSphere Replication 9.x

Cause

  • When the vSphere Replication Server service (hbrsrv) starts, it tries to communicate with all ESXi hosts registered under vCenter.

  • The issue occurs when communication between the vSphere Replication Appliance and one or more ESXi hosts fails. Because of this, the hbrsrv service continuously attempts to reconnect to the unreachable host.

  • These repeated failed communication attempts cause hbrsrv to refuse connections from the hms service, which leads to RPO violations on the replicated VMs.

  • The /opt/vmware/support/logs/hbrsrv.log contains HbrError entries indicating: “Connection reset by peer”

### ## ##:##:## vr_hostname hbrsrv[1191]: ####-##-##T##:##:##.496Z error hbrsrv[01207] [Originator@6876 sub=Main] HbrError stack:
### ## ##:##:## vr_hostname hbrsrv[1191]: ####-##-##T##:##:##.496Z error hbrsrv[01207] [Originator@6876 sub=Main] [0] ClientConnection (client=[10.###.###.101]:57278) request callback failed: Connection reset by peer: The connection is terminated by the remote end with a reset packet. Usually, this is a sign of a network problem,  timeout, or service overload.
### ## ##:##:## vr_hostname hbrsrv[1191]: ####-##-##T##:##:##.496Z error hbrsrv[01207] [Originator@6876 sub=Main]    [1] Dropping error encountered from network
### ## ##:##:## vr_hostname hbrsrv[1191]: ####-##-##T##:##:##.496Z info hbrsrv[01207] [Originator@6876 sub=StatsLog] HbrEvent: {"eventID":"lwdConnectionReset","groupID":"GID-c2b310a3-d6b2-4b9d-9632-9ddb8b129667","clientAddress":"[10.###.###.101]:57278","serverID":"52bdff74-8d03-b24c-9df3-9df1bfac2516","hbrEvent":1}

  • From the /opt/vmware/hms/logs/hms.log on the Embedded Appliance, it will be observed that the hms service is refusing the connection with the hbr service on port 8123.

####-##-## ##:##:##.224 WARN  com.vmware.hms.monitor.hbr.52bdff74-8d03-b24c-9df3-9df1bfac2516.PCFilterManager [hms-vlsi-client-thread-1333] (..jvsl.pcm.PCFilterManager) [operationID=0901e397-4a7f-4deb-87ba-317c7dad1ebb-HMSINT-5261376] | 'WaitForUpdates' for 'Stub: moRef = (ManagedObjectReference: type = PropertyCollector, value = session[5242ec1f-8e04-c960-a6ef-82cd0a790acb]52015b5b-bd3a-a0a4-ca10-c6e2779d3c89, serverGuid = null), binding = https://vr_hostname:8123/' failed.
com.vmware.vim.vmomi.client.exception.ConnectionException: https://vr_hostname:8123/ invocation failed with "org.apache.http.conn.HttpHostConnectException: Connect to vr_hostname:8123 [vr_hostname/127.0.0.1, vr_hostname/0:0:0:0:0:0:0:1] failed: Connection refused"

Also, The below log confirms that vSphere Replication Appliance (HMS) has lost its network path to the ESXi host (##.###.##.##) on port 443.

####-##-## ##:##:##.916 ERROR com.vmware.hms.net.hbr.ping.svr.77ee749b-59f0-4a7c-b114-a0578833a0a2 [hms-ping-scheduled-thread-8] (..net.impl.VmomiPingConnectionHandler) [operationID=fc3fba6f-a53e-436a-b3c4-2026bbee3d6f-HMSINT-2, operationID=faad5ab5-37ab-417e-8083-df46e2d18212-HMS-PING] | Ping for server ##.###.##.##:443/hbr for session: N/A failed:com.vmware.vim.vmomi.client.exception.ConnectionException: https://##.###.##.##:443/hbr invocation failed with "java.net.NoRouteToHostException: No route to host"
          com.vmware.jvsl.sessions.net.impl.PersistentConnection$ReconnectingPing.run(PersistentConnection.java:375) ~[jvsl-sessions-9.0.2.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$RunnableWithDiagnosticContext.run(LoggerOpIdConfigurator.java:132) ~[hms.jar:?]
        at com.vmware.hms.util.executor.LoggerOpIdConfigurator$2.run(LoggerOpIdConfigurator.java:99) ~[hms.jar:?]
        at com.vmware.jvsl.sessions.net.impl.TlsPreservingWrapper$2.run(TlsPreservingWrapper.java:47) ~[jvsl-sessions-9.0.2.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
        at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.NoRouteToHostException: No route to host
        at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]

Followed by, logs that vSphere Replication Appliance (HMS) attempts to ping the host , the host's management service (hostd) is responding with a null pointer. This usually means the host-side replication service is in a zombie state—it's alive enough to respond to the connection, but too broken to process the request.

####-##-## ##:##:##.582 ERROR com.vmware.hms.net.hbr.ping.svr.b0356e98-aefd-4227-9644-9125ea2dc3c6 [hms-ping-scheduled-thread-5] (..net.impl.VmomiPingConnectionHandler) [operationID=fc3fba6f-a53e-436a-b3c4-2026bbee3d6f-HMSINT-2, operationID=23111c57-14cf-4ff7-a01b-9983b3704361-HMS-PING] | Ping for server ##.###.##.##:443/hbr for session: N/A failed:
com.vmware.vim.binding.vmodl.fault.SystemError: null
        at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source) ~[?:?]
        at java.lang.reflect.ReflectAccess.newInstance(Unknown Source) ~[?:?]
        at jdk.internal.reflect.ReflectionFactory.newInstance(Unknown Source) ~[?:?]
        at java.lang.Class.newInstance(Unknown Source) ~[?:?]

  • Validation of the logs (/var/run/log/vmkernel.log) on the ESXi host further confirms that the host is failing to receive data from the VR Appliance due to network issues:

####-##-##T##:##:##.829Z cpu13:12228180)WARNING: Hbr: 893: Failed to receive from 10.###.##.7 (groupID=GID-########-####-####-####-############): Timeout
####-##-##T##:##:##.499Z cpu0:12263467)WARNING: Hbr: 893: Failed to receive from 10.###.##.6 (groupID=GID-########-####-####-####-############): Timeout

(OR)

####-##-##T##:##:##.468Z Wa(180) vmkwarning: cpu106:3811467)WARNING: Hbr: 788: Failed to receive from 127.0.0.1 (groupID=GID-########-####-####-####-############): Broken pipe
####-##-##T##:##:##.509Z Wa(180) vmkwarning: cpu83:3819621)WARNING: Hbr: 788: Failed to receive from 127.0.0.1 (groupID=GID-########-####-####-####-############): Broken pipe

Resolution

  • Work with the network team to fix the connectivity issue between the vSphere Replication appliance and the ESXi hosts.
  • Ensure all required ports and firewall rules are correctly configured.
  • Refer to the official VMware Ports and Protocols documentation for detailed requirements.

As a workaround if you want to exclude hosts or a cluster of hosts under a given cluster or datacenter from registration in vSphere Replication, you can tag them with the com.vmware.vr.disallowedHost tag. This is valid for the incoming replications on the target site. 

Exclude hosts from registration in vSphere Replication

Powered by Wolken Software