"Invalid or inaccessible datastore." when creating new replications/Migrations in vCloud Availability 4.x

Products

• VMware Cloud Director

Issue/Introduction

Symptoms:

• When creating a new replication, the task fails and you see a similar error:

Invalid or inaccessible datastore.
Timed Out

• In /var/log/vmkernel.log on the source ESXi host, you see entries similar to:

yyyy-mm-ddThh:mm:ss,sssZ cpu34:#######)Hbr: 2998: Command: INIT_SESSION: error result=Failed gen=-1: Error for (datastoreUUID: "########-####-####-####-##########"), (diskId: "H4D-########-####-####-####-##########"), (flags: on-disk-open): $
yyyy-mm-ddThh:mm:ss,sssZ cpu34::#######))WARNING: Hbr: 3007: Command INIT_SESSION failed (result=Failed) (isFatal=FALSE) (Id=0) (GroupID=H4-########-####-####-####-##########)
yyyy-mm-ddThh:mm:ss,sssZ cpu34::#######))WARNING: Hbr: 4570: Failed to establish connection to [##.##.###.##]:44046(groupID=H4-########-####-####-####-##########): Failure

• In /opt/vmware/h4/replicator/log/replicator.log on the destination Replicator, you see an entry similar to:
  
  com.vmware.vim.binding.hbr.replica.fault.StorageNotAccessible: Error for (datastoreUUID: "########-####-####-####-##########"), (diskId: "H4-########-####-####-####-##########"), (flags: on-disk-open): No accessible host for datastore '########-####-####-####-##########'; Code set to: Storage was not accessible.; Failed to find host to get disk type; While getting host capabilities for disk. (native snapshot hint =  false); Failed to open disk, couldn't get disk type/capabilities; Set error flag: on-disk-open;  Failed to open replica (/vmfs/volumes/########-####-####-####-##########/C4-########-####-####-####-##########/VM_NAME.vmdk); Failed to open active Disk (GroupID=H4-########-####-####-####-##########) (DiskID=########-####-####-####-##########); Can't create replica state (GroupID=H4-########-####-####-####-##########) (DiskID=########-####-####-####-##########); Cannot activate group. Loading disks from database (GroupID=H4-########-####-####-####-##########) ;

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue can occur when a dedicated VMkernel interface has been created for replication traffic on the same subnet as the management VMkernel interface on the ESXi hosts.

 

Resolution

• Ensure that the replication VMkernel interface is on a separate subnet to the management VMkernel interface as per the VMware Cloud Availability deployment requirements. For more information, refer: Isolating the Network Traffic of vSphere Replication

• Test connectivity between replicators and ESXi hosts to ensure proper communication.
• From the Replicator SSH , run the following command -

 #curl -kv telnet://Host_IP_or_FQDN:80

 #curl -kv telnet://Host_IP_or_FQDN:902

• If there is no output indicating communication between the replicator and the ESXi host.
• Ensure that port 80 and 902 are open on the ESXi host firewall to allow proper communication for replication.

Additional Information

For information about the network ports required for the correct operation of VMware Cloud Director Availability, see VMware Cloud Director Availability - VMware Ports and Protocols.