Reverse Migration fails with error "Storage not accessible error observed"

Products

• VMware HCX

Issue/Introduction

When performing a reverse migration with VMware HCX, the operation may fail after Base Sync is initiated with the error message: "Storage not accessible error observed."

This error prevents the virtual machine migration from completing successfully.

Common Logs

In common/logs/admin/app.log:

No accessible host for datastore; Code set to: Storage was not accessible.; Failed to find host to get disk type; While getting host capabilities for disk. (native snapshot hint = false); Failed to open disk, couldn't get disk type/capabilities; Set error flag: on-disk-open;

IX Logs

In the IX logs at /var/log/vmware/hbrsrv.log:

Code set to: Storage was not accessible.

HbrError for (datastoreUUID: "#####"), (pathname: "#####") stack:HbrError for (datastoreUUID: "#####"), (pathname: "#####") stack:

• These logs indicate an issue with the Host-based Replication (Hbr) service accessing the datastore.
• No correlating Hbr errors appear on the ESXi host at the same time.
• To diagnose this issue, run Data-Plane Diagnostics to identify any failed probes according to this article - Run Data-Plane Diagnostics

Environment

VMware HCX

VMware ESXi

Cause

Connectivity over port 902 is required for the IX appliances to communicate to the hosts to see storage and if this is down it may appear as storage inaccessible on the host during migration.

Resolution

1. If the IX appliance was working previously, redeploy the IX appliances to re-establish the connection on the source side. If there is not an issue with a firewall, this will typically restore the connection.
2. If redeploying the IX appliances does not fix the connectivity issue:
   • Review the firewall configuration between the source IX appliances and ESXi hosts that the reverse migration would be sending the VM back to
   • Find the allow rule which the traffic is passing over
   • If you are unable to see the traffic for port 902 between the IX appliance IP address and ESXi hosts:
     • Create a firewall rule that explicitly opens up all traffic between the IX appliance and ESXi hosts
     • Verify that the port 902 traffic is communicating as expected
     • After verification, restrict the rule further according to your security preferences

Many times this storage issue occurs when port 902 traffic accidentally hits a drop rule instead of an expected allow rule in the firewall configuration.

If the error persists after following these steps, contact Broadcom Support for further assistance.

Please provide the below information when opening a support request with Broadcom for this issue

• Migration details (VM Name, Migration id, Screenshot of error as seen via UI)
• Source and Target HCX log bundles with IX appliance selected.