In this scenario:
/var/log/vmkernel.log
shows:"Failed to establish connection to VR (target) over port 31031: Failure."
Further investigation reveals the following:
Example log entries from /var/log/vmkernel.log
:
2025-04-29T12:39:53.301Z cpu26:15399511)Hbr: 3410: Command: INIT_SESSION: error result=Failed gen=-1: Error for (datastoreUUID: "vsan:Datastore
"), (diskId: "RDID-47cf15df-####-####-####-######927fbc"), (hostId: "host-#"),
2025-04-29T12:39:53.301Z cpu26:15399511)WARNING: Hbr: 3438: Command INIT_SESSION failed (result=Failed) (isFatal=FALSE) (Id=0) (GroupID=GID-43b914a1-####-####-####-######39a58e)
2025-04-29T12:39:53.301Z cpu26:15399511)WARNING: Hbr: 5093: Failed to establish connection to [192.###.##.####]:31031 (groupID=GID-
43b914a1-####-####-####-######39a58e
): Failure
This can happen for all datastore types
vSphere Replication (All Versions
The issue is caused by the .vmdk file being locked on the source ESXi host, preventing it from being accessed by the target ESXi host for replication. Below is a breakdown of the related errors:
NFC File Locked Error:
Example log entries from /var/log/vmware/hbrsrv.log:
2025-05-02T18:32:45.723+05:30 info hbrsrv[15516] [Originator@6876 sub=StorageManager groupID=GID-
GID-
opID=hsl-########] Destroying NFC connection to host-#.43b914a1-####-####-####-######39a58e
2025-05-02T18:32:45.723+05:30 error hbrsrv[15516] [Originator@6876 sub=Main groupID=
GID-
opID=hsl-########] HbrError for (datastoreUUID: "vsan:Datastore...),43b914a1-####-####-####-######39a58e
2025-05-02T18:32:45.723+05:30 error hbrsrv[15516] [Originator@6876 sub=Main groupID=
GID-
opID=hsl-########] NFC error: NFC_FILE_LOCKED43b914a1-####-####-####-######39a58e
"Can't open remote disk" messages in hostd.log:
Example log entries from /var/log/hostd.log:
2025-05-05T12:22:46.477Z warning hostd[2106636] [Originator@6876 sub=Libs opID=hsl-########-000000cedf24a0a0-TicketID:5267e6f7-####-####-####-######ad52e6 ExpirationDate:5018#####0054] [NFC ERROR] Failed to pass the Get() operation to the chain layer: The called function cannot be performed on partial chains.
2025-05-05T12:22:46.480Z info hostd[2106636] [Originator@6876 sub=DiskLib opID=hsl-########-000000cedf24a0a0] Failed to open '/vmfs/volumes/vsan:Datastore...': Could not find the file.
2025-05-05T12:22:46.480Z info hostd[2106636] [Originator@6876 sub=Libs opID=hsl-########-000000cedf24a0a0-TicketID:5267e6f7-####-####-####-######ad52e6 ExpirationDate:5018#####0054 RemainingUseCount:0] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/vsan:Datastore/28936b5e-####-####-####-######533154/VM.vmdk' : Could not find the file (393218).
Remove and Re-add VMs to Replication:
Outcome:
If there are still issues after this process, open a case with Broadcom support.