Error: "Invalid or inaccessible datastore" when creating a new incoming replication in VMware Cloud Director Availability

Products

• VMware

Issue/Introduction

• A new incoming replication from On-prem to Cloud fails with the error:
  
  "Invalid or inaccessible datastore" 
  
  
• From /opt/vmware/h4/cloud/log/cloud.log you see entry similar the one reported below:
  
  

  DATE TIME DEBUG - [UI-########-####-####-####-########ed51] [job-55] c.vmware.h4.cloud.job.VmReplicationJob   : H4 manager task result: ReplicationInfo{id='H4-########-####-####-####-########269', vmName='{NAME_OF_THE_VM}', owner='Example@CloudSite', settings=ReplicationSettings{vcId='########-####-####-####-########878e', vmId='{vm-ID}', description='Protected via VCDA UI', externalKey='C4-########-####-####-####-########8650', excludedDiskKeys=[], rpo=1440, dataConnectionType=ENCRYPTED, quiesced=false, sourceLocation=ReplicationLocation{site='{site_name}', replicatorId=########-####-####-####-########2248},destinationLocation=ReplicationLocation{site='ExampleCloudSite', replicatorId=########-####-####-####-########7dce}, replicationPlacement=ReplicationPlacement{datastoreId='{datastore-moref}', datastoreFolder='C4-########-####-####-####-########8650', vcId='########-####-####-####-########878e', disks=[DiskPlacement{diskKey=2000, datastoreId='{datastore-moref}', diskPath='C4-####.####.####.####/example.vm.vmdk', isSeed=false, storageProfileId='########-####-####-####-########4fc9', contentId='null'}], validateSeedUuids=true, storageProfileId='########-####-####-####-########4fc9'}, retentionPolicy=RetentionPolicy{rules=[]}, autopinInstances=false, targetDiskType=THIN, archiveSettings=null, startPaused=false, initialSyncTime=0, isMigration=false, isReversed=false}, destinationState=DestinationStateInfo{currentRpoViolation=-1, latestInstance=null, state=passive, recoveryInfo=RecoveryInfo{recoveryState=NOT_STARTED, vcId='null', vmId='null', vmName='null', optimizeUntil=null, isMigration=null}, lastError=null, replicatedDisks=[ReplicatedDiskInfo{diskKey=2000, uuid=########-####-####-####-########c479, diskType='thin', baseName='example.vm.vmdk', capacityBytes=107374182400, isSeed=false, isReplicated=true, spaceRequirement=-1}], spaceRequirement=10485760, isMovingReplica=false, stateAge=11}, sourceState=SourceStateInfo{state=syncing, progress=ProgressInfo{progress=0, bytesTransferred=0, bytesToTransfer=0, checksumTotalBytes=107374182400, checksumComparedBytes=0}, stateAge=2}, isPaused=false, lastUpdated=1725955899131, dataConnectionState=INITIAL_SYNCING, overallHealth=GREEN, sourceVmEncrypted=false, initialSyncAt=null}


• On the /var/log/vmware/hbrsrv.log on the replicator, you can see there is a connectivity issue with the chosen host as per below entry:
  
  

  2024-09-10T08:14:11.627Z verbose hbrsrv[870081] [Originator@6876 sub=IO.Connection opID=UI-########-####-####-####-########-as-Ix-31] Attempting connection; <resolver p:0x00007fefb4086f60, '##.##.##.##:80', next:<TCP '##.##.##.## : 80'>>, last e: 0(Success)
2024-09-10T08:14:11.627Z warning hbrsrv[01342] [Originator@6876 sub=IO.Connection opID=UI-####.####.####.####1-as-Ix-31] Failed to connect; <io_obj p:0x00007fef9817dd10, h:48, <TCP '##.##.##.## : 44728'>, <TCP '##.##.##.## : 80'>>, e: 111(Connection refused), duration: 0msec
2024-09-10T08:14:11.627Z verbose hbrsrv[01342] [Originator@6876 sub=IO.Connection opID=UI-########-####-####-####-########-as-Ix-31] Attempting connection; <resolver p:0x00007fefb4086f60, '##.##.##.##:80', next:(null)>, last e: 111(Connection refused)
2024-09-10T08:14:11.627Z warning hbrsrv[01342] [Originator@6876 sub=HttpConnectionPool-052687 opID=UI-########-####-####-####-########-as-Ix-31] Failed to get pooled connection; <cs p:00007fef94008a00, TCP:##.##.##.##:80>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
2024-09-10T08:14:11.627Z info hbrsrv[01342] [Originator@6876 sub=IO.Http opID=UI-########-####-####-####-########-as-Ix-31] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
2024-09-10T08:14:11.627Z error hbrsrv[01342] [Originator@6876 sub=IO.Http opID=UI-########-####-####-####-########-as-Ix-31] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
2024-09-10T08:14:11.627Z error hbrsrv[01342] [Originator@6876 sub=vmomi.soapStub[52949] opID=UI-########-####-####-####-########-as-Ix-31] Resetting stub adapter: service state request failed; a: <<cs p:00007fef94008a00, TCP:##.##.##.##:80>, /sdk>, pa: <cs p:00007fef94008a00, TCP:##.##.##.##:80>, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)

• The host selected during the replication has no datastore access as per environment configuration. 

Environment

VMware Cloud Director Availability 4.7
VMware Cloud Director 10.5

Cause

The host selected during the replication has no datastore access as per environment configuration.
The selected datastores is part of one of the storage policies on the destination Organization VDC on VMware Cloud Director site.

Resolution

Review Storage Policy Properties in the VMware Cloud Director Tenant Portal
If the storage policies contain datastores that are not supposed to be part of the policy the datastores should be removed from the storage policy and try again the VM replication.
You can refer to the documentation Managing the VM Storage Policies on a Provider Virtual Data Center in Your VMware Cloud Director and  Creating and Managing vSphere VM Storage Policies