HCX RAV/Bulk Migrations stalls at "Storage not accessible error observed"
search cancel

HCX RAV/Bulk Migrations stalls at "Storage not accessible error observed"

book

Article ID: 447075

calendar_today

Updated On:

Products

VMware HCX

Issue/Introduction

  • HCX Bulk or RAV migration fails with the following error in the HCX migration UI: "Storage not accessible error observed"



  • The following events are observed in Target HCX /common/logs/admin/app.log indicating that IX cannot connect to any ESXi datastore:

    <timestamps> UTC [ReplicationTransferService_SvcThread-265, Ent: HybridityAdmin, , TxId: TxId: ####-####-####-fa5eaee6bae2] ERROR c.v.h.s.r.j.ReplicationTransferMonitor- No accessible host for datastore is found for syncing, please check communication between HBR and Host
<timestamps> UTC [ReplicationTransferService_SvcThread-265, Ent: HybridityAdmin, , TxId: TxId: ####-####-####-fa5eaee6bae2] INFO  c.v.h.s.r.j.ReplicationTransferMonitor- Syncing storageNotAccessibe event for the transfer '####-####-d7f89f95eeea': {"reason":"No accessible host for datastore is found for syncing, please check communication between HBR and Host"}

     

  • The target IX appliance HBR logs, located in /var/log/vmware/hbrsrv.log, display the following events indicating IX cannot reach ESXi port 443 :
    <timestamps>Z info hbrsrv[07418] [Originator@6876 sub=Main opID=TxId: ####-####-19a5435389b8:hs-21b8] HbrError for (datastoreUUID: "vsan:####-54fdc44645939452"), (diskId: "RDID-####-####-8194b388b92c"), (pathname: "<vm-name>/<vm-name>.vmdk") stack:
<timestamps>Z info hbrsrv[07418] [Originator@6876 sub=Main opID=TxId: ####-####-19a5435389b8:hs-21b8]    [0] No accessible host for datastore 'vsan:####-54fdc44645939452'
<timestamps>Z info hbrsrv[07418] [Originator@6876 sub=Main opID=TxId: ####-####-19a5435389b8:hs-21b8]    [1] Code set to: Storage was not accessible.

<timestamps>Z warning hbrsrv[07761] [Originator@6876 sub=IO.Connection opID=####-####-89a8af1d881f] Failed to connect; <io_obj p:0x00007f177402f1e0, h:84, <TCP '<IX_IP> : 59944'>, <TCP '<ESXi_IP> : 443'>>, e: 110(Connection timed out), duration: 64926msec
<timestamps>Z warning hbrsrv[07415] [Originator@6876 sub=IO.Connection opID=####-####-89a8af1d881f] Failed to connect; <io_obj p:0x00007f1744040e70, h:86, <TCP '<IX_IP> : 33882'>, <TCP '<ESXi_IP> : 443'>>, e: 110(Connection timed out), duration: 65510msec

     

  • OpenSSL connection test does not return certificate chain.
    root@IX [ /var/log/vmware ]# openssl s_client -connect <host-ip>:443
    CONNECTED(00000003)

    Note: An OpenSSL connection test might fail on a specific VMkernel interface with the replication service enabled (e.g., vmk1), while succeeding on others (e.g., vmk0). This indicates that traffic is being blocked exclusively on the replication network, rather than globally across all networks.

Environment

VMware HCX
VMware ESXi

Cause

The migration stalls because TCP port 443 communication between the HCX Interconnect (IX) appliance and the target ESXi host is blocked, preventing the Host-Based Replication (HBR) server from successfully accessing the datastore.

Resolution

  • Verify the network firewall rules and routing between the HCX Interconnect (IX) appliance and the target ESXi hosts.
  • Ensure that TCP port 443 is allowed from HCX-IX to the ESXi.
  • Once the port is unblocked, retry or resume the HCX RAV or Bulk migration.
  • As a temporary measure, the replication service can be moved to the other management network if that path is unrestricted. A restart of HCX services is required after this change to ensure the IX appliance refreshes its connection path.

Additional Information

Powered by Wolken Software