Unable to configure VM - "No host can be used to access datastore path.vmdk"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• While configuring VM for replication, an error observed as:"No host can be used to access datastore path.vmdk"

•  You may see connectivity issues with respect to the required ports not being opened.

 

Environment

vSphere Replication 8.x
vSphere Replication 9.x

Cause

The issue occurs due to a connection timeout on port 31031 between vSphere Replication (VR) and the ESXi host. The port 31031 is essential for communication between VR and the host. If it is blocked or experiencing timeouts, VR fails to establish a connection, leading to replication errors. 

From ESXi, /var/log/vmkernel.log,an error triggered "Failed to establish connection"

2017-06-07T16:31:07.042Z cpu12:42983549)WARNING: Hbr: 2997: Command INIT_SESSION failed (result=Failed) (isFatal=FALSE) (Id=0) (GroupID=GID-#######-####-####-####-##########d0)
2017-06-07T16:31:07.042Z cpu12:42983549)WARNING: Hbr: 4521: Failed to establish connection to [x.x.x.x]:31031(groupID=GID-#######-####-####-####-##########d0): Failure
2017-06-07T16:32:37.061Z cpu12:42983549)Hbr: 2196: Wire compression supported by server x.x.x.x: FastLZ
2017-06-07T16:32:37.074Z cpu16:42983549)Hbr: 2988: Command: INIT_SESSION: error result=Failed gen=-1: Error for (datastoreUUID: "########-#########"), (diskId: "RDID-######-####-####-####-#########be"), (flags: on-disk-open): No accessible host for da$
2017-06-07T16:32:37.074Z cpu16:42983549)WARNING: Hbr: 2997: Command INIT_SESSION failed (result=Failed) (isFatal=FALSE) (Id=0) (GroupID=GID-#######-####-####-####-##########d0)

From VR, /var/log/vmware/hbrsrv.log, an error triggered "storage not accessible"

2017-06-07T16:50:05.161Z verbose hbrsrv[7F96E7009700] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: lastError, Hbr.Replica.Host.host-372. Applied change to temp map.
2017-06-07T16:50:05.161Z error hbrsrv[7F96E444A700] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:00007f96d8002b50, TCP:x.x.x.x:80>; cnx: (null), error: N7Vmacore17CanceledExceptionE(Operation was canceled)
2017-06-07T16:50:05.162Z verbose hbrsrv[7F96E710D700] [Originator@6876 sub=PropertyProvider] RecordOp ADD: expectedDatastoresList["Hbr.Replica.Datastore.53558063-########-####-441ea147e1b4"], HbrStorageManager. Applied change to temp map.
2017-06-07T16:50:05.161Z warning hbrsrv[7F96E7150760] [Originator@6876 sub=Default] Failed to connect socket; <io_obj p:0x00007f96eb0937a0, h:-1, <TCP '0.0.0.0:0'>, <TCP 'x.x.x.x:80'>>, e: system:125(Operation canceled)
2017-06-07T16:50:05.162Z warning hbrsrv[7F96E7150760] [Originator@6876 sub=Default] Operation cancelled
2017-06-07T16:50:05.162Z error hbrsrv[7F96E7150760] [Originator@6876 sub=HttpConnectionPool-000045] [ConnectComplete] Connect failed to <cs p:00007f96d8002b50, TCP:x.x.x.x:80>; cnx: (null), error: N7Vmacore17CanceledExceptionE(Operation was canceled)
2017-06-07T16:50:05.171Z verbose hbrsrv[7F96E4409700] [Originator@6876 sub=SessionManager] hbr.replica.Task.GetInfo: authorized
2017-06-07T16:50:05.179Z verbose hbrsrv[7F96E710D700] [Originator@6876 sub=DiskMove] Disk Move spec: (nfc.CopySpec) []
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] HbrError for (datastoreUUID: "4eaede45-########-####-#########b6") stack:
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [0] No accessible host for datastore 4eaede45-#######-####-#########b6
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [1] Code set to: Storage was not accessible.
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [2] Failed to find host to remove file
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [3] Couldn't do cleanup for file '/vmfs/volumes/4eaede45-########-####-#########b6/Main Content Svr/hbrgrp.GID-cab02c86-####-####-####-##########d0.txt' (key=4).
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [4] Will retry later.
2017-06-07T16:50:05.195Z info hbrsrv[7F96E710D700] [Originator@6876 sub=Main] [5] Ignored error.
2017-06-07T16:50:05.195Z verbose hbrsrv[7F96E710D700] [Originator@6876 sub=StorageMap] Datastore 4eaede45-########-####-#########b6 removed from storage map

From DR VRMS we are unable to reach the Hosts in Destination.

If you are trying to perform curl test from DR VRMS to DR hosts, we are getting the below details confirming that the port is being blocked.

root [ /var/log/vmware ]# curl -v telnet://1##.#.#.#:80
Trying 1##.#.#.##:80... *
connect to 1##.#.#.## port 80 failed: No route to host * Failed to connect to 1##.#.#.## port 80 after 3077 ms: Couldn't connect to server * Closing connection 0 curl: (7) Failed to connect to 1##.#.#.## port 80 after 3077 ms: Couldn't connect to server

Port 80 requirement is only in vSphere replication 8.x version.

Resolution

• Ensure port 31031 is open from source ESXi(Protected site) to vSphere Replication appliance at (Recovery site) - Port numbers that must be open for vSphere Replication 8.x

• Also, ensure no firewall blockages.
  
  
• Involve the networking team to verify connectivity between VR appliance and the ESXi host.