Storage for Datastore Path ###.vmdk is Locked During Replication
search cancel

Storage for Datastore Path ###.vmdk is Locked During Replication

book

Article ID: 396628

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms

  • While performing replication, the following error is observed:
    “Error (RPO violation): Storage for datastore path ###.vmdk is locked.”

In this scenario:

  • The source ESXi host can successfully ping the target vSphere Replication server, but the round-trip time (RTT) exceeds 50 ms.
  • Ports 31031, 443, and 902 are open and not blocked.
  • Despite this, replication fails, and an NFC error indicating that the storage is locked occurs.
  • The /var/log/vmkernel.log shows:
    "Failed to establish connection to VR (target) over port 31031: Failure."

Further investigation reveals the following:

  • The .vmdk file is unavailable in the destination folder.
  • The target ESXi host is unable to open the parent disk.
  • The source ESXi host encounters errors related to the INIT_SESSION command.

Example log entries from /var/log/vmkernel.log:

2025-04-29T12:39:53.301Z cpu26:15399511)Hbr: 3410: Command: INIT_SESSION: error result=Failed gen=-1: Error for (datastoreUUID: "vsan:Datastore"), (diskId: "RDID-47cf15df-####-####-####-######927fbc"), (hostId: "host-#"),
2025-04-29T12:39:53.301Z cpu26:15399511)WARNING: Hbr: 3438: Command INIT_SESSION failed (result=Failed) (isFatal=FALSE) (Id=0) (GroupID=GID-43b914a1-####-####-####-######39a58e)
2025-04-29T12:39:53.301Z cpu26:15399511)WARNING: Hbr: 5093: Failed to establish connection to [192.###.##.####]:31031 (groupID=GID-43b914a1-####-####-####-######39a58e): Failure

This can happen for all datastore types

Environment

vSphere Replication (All Versions

Cause

The issue is caused by the .vmdk file being locked on the source ESXi host, preventing it from being accessed by the target ESXi host for replication. Below is a breakdown of the related errors:

  • NFC File Locked Error:

    • The error message in the log suggests that the replica disk could not be opened for replication due to the disk being locked.

Example log entries from /var/log/vmware/hbrsrv.log:

2025-05-02T18:32:45.723+05:30 info hbrsrv[15516] [Originator@6876 sub=StorageManager groupID=GID-GID-43b914a1-####-####-####-######39a58e opID=hsl-########] Destroying NFC connection to host-#.
2025-05-02T18:32:45.723+05:30 error hbrsrv[15516] [Originator@6876 sub=Main groupID=GID-43b914a1-####-####-####-######39a58e opID=hsl-########] HbrError for (datastoreUUID: "vsan:Datastore...),
2025-05-02T18:32:45.723+05:30 error hbrsrv[15516] [Originator@6876 sub=Main groupID=GID-43b914a1-####-####-####-######39a58e opID=hsl-########] NFC error: NFC_FILE_LOCKED

  • "Can't open remote disk" messages in hostd.log:

    • The error “Can't open remote disk” and the following log entries indicate failure in accessing the replica disk, confirming that the disk is locked.

Example log entries from /var/log/hostd.log:

2025-05-05T12:22:46.477Z warning hostd[2106636] [Originator@6876 sub=Libs opID=hsl-########-000000cedf24a0a0-TicketID:5267e6f7-####-####-####-######ad52e6 ExpirationDate:5018#####0054] [NFC ERROR] Failed to pass the Get() operation to the chain layer: The called function cannot be performed on partial chains.
2025-05-05T12:22:46.480Z info hostd[2106636] [Originator@6876 sub=DiskLib opID=hsl-########-000000cedf24a0a0] Failed to open '/vmfs/volumes/vsan:Datastore...': Could not find the file.
2025-05-05T12:22:46.480Z info hostd[2106636] [Originator@6876 sub=Libs opID=hsl-########-000000cedf24a0a0-TicketID:5267e6f7-####-####-####-######ad52e6 ExpirationDate:5018#####0054 RemainingUseCount:0] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/vsan:Datastore/28936b5e-####-####-####-######533154/VM.vmdk' : Could not find the file (393218).

    • The logs indicate that the replication is failing because the .vmdk file is locked on the source and cannot be opened on the target host.

Resolution

Remove and Re-add VMs to Replication:

    • Remove and re-add the VMs to the replication configuration to start fresh and eliminate any inconsistencies.

Outcome:

  • After the initial sync, the replication status returned to OK, confirming successful synchronization.

 

If there are still issues after this process, open a case with Broadcom support.