VM replication fails with an error 'NFC_NETWORK_ERROR'

Products

• VMware Live Recovery

Issue/Introduction

Symptoms :

• Replication of the Virtual Machines failing with RPO Violation error.
• Last error shows 'NFC_NETWORK_ERROR'
  
  A replication error occurred at the vSphere Replication Server for replication 'VMName'. Details: 'Error for (datastoreUUID: "########-########-####-###########"), (diskId: "RDID-########-####-####-####-###########"), (hostId: "host-##"), (pathname: "VMName/hbrdisk.RDID-########-####-####-####-###########.#######.#############.vmdk"), (flags: nfc-error, retriable): Class: NFC Code: 3; NFC error: NFC_NETWORK_ERROR; Code set to: Error connecting to host.; Set error flag: retriable; Set error flag: nfc-error; Can't get block allocation information on remote disk'.



Environment

VMware vSphere Replication 8.x

VMware vSphere Replication 9.x

 

Cause

This issue is caused due to network connectivity issue between the vSphere replication servers and the ESXi hosts at the target sites.

/var/log/vmware/hbrsrv.log on the target VR reports 'Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR).'

2025-04-01T06:29:51.080Z info hbrsrv[01842] [Originator@6876 sub=Main]    [0] Class: NFC Code: 3
2025-04-01T06:29:51.080Z info hbrsrv[01842] [Originator@6876 sub=Main]    [1] NFC error: NFC_NETWORK_ERROR
2025-04-01T06:29:51.080Z info hbrsrv[01842] [Originator@6876 sub=Main]    [2] Code set to: Error connecting to host.
2025-04-01T06:29:51.080Z info hbrsrv[01842] [Originator@6876 sub=Main]    [3] Set error flag: retriable
2025-04-01T06:29:51.080Z info hbrsrv[01842] [Originator@6876 sub=Main]    [4] Set error flag: nfc-error
2025-04-01T06:29:51.081Z warning hbrsrv[01842] [Originator@6876 sub=Libs] [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
2025-04-01T06:29:51.081Z warning hbrsrv[01842] [Originator@6876 sub=Libs] [NFC ERROR]NfcFssrvr_Close: Failed to send close message: The operation experienced a network error (NFC_NETWORK_ERROR)

/var/log/hostd.log on the target ESXi host indicates network-related issues resulting in connection timeouts from the target site VR appliance to the target ESXi host.

2025-03-31T15:46:52.118Z info hostd[2100627] [Originator@6876 sub=Libs opID=0000005e8d9ab2f0] SSL: syscall error 32: Broken pipe
2025-03-31T15:46:52.118Z warning hostd[2100627] [Originator@6876 sub=Libs opID=0000005e8d9ab2f0] [NFC ERROR]NfcNetTcpSetError: Broken pipe
2025-03-31T15:46:52.118Z warning hostd[2100627] [Originator@6876 sub=Libs opID=0000005e8d9ab2f0] [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
2025-03-31T15:46:52.118Z warning hostd[2100627] [Originator@6876 sub=Libs opID=0000005e8d9ab2f0] [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
2025-03-31T15:46:52.396Z info hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] SSL: syscall error 110: Connection timed out
2025-03-31T15:46:52.396Z warning hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC ERROR]NfcNetTcpSetError: Connection timed out
2025-03-31T15:46:52.396Z warning hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC ERROR]NfcNetTcpRead: bRead: -1. errno: 110.
2025-03-31T15:46:52.396Z warning hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC ERROR]NfcNet_Recv: requested 264, recevied only 0 bytes
2025-03-31T15:46:52.396Z warning hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC ERROR]NfcGetMessage: NfcNet_Recv failed: NFC_NETWORK_ERROR
2025-03-31T15:46:52.396Z warning hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC ERROR]NfcServerProcessClientMsg: Failed to get message from client: NFC_NETWORK_ERROR (The operation experienced a network error)
2025-03-31T15:46:52.396Z error hostd[2100641] [Originator@6876 sub=Nfcsvc opID=0000005e8e6ad170] Read error from the nfcLib: NFC_NETWORK_ERROR (done = yep)
2025-03-31T15:46:52.396Z info hostd[2100641] [Originator@6876 sub=Libs opID=0000005e8e6ad170] [NFC INFO]NfcServer_SessionCleanup: Cleaning up server session with currentState: NFC_FSSRVR_READY
2025-03-31T15:46:52.398Z info hostd[2100641] [Originator@6876 sub=Nfcsvc opID=0000005e8e6ad170] Session is closing
2025-03-31T15:46:52.398Z info hostd[2100641] [Originator@6876 sub=Nfcsvc opID=0000005e8e6ad170] Cleaning up NFC session with ticketId: TicketID: ExpirationDate:0 RemainingUseCount:0 from Client: 10.##.##.##

* where Client IP: 10.##.##.## is the target site VR IP address.

Resolution

Engage the physical network team to investigate the connectivity issue and ensure stable connectivity with sufficient bandwidth between source and target sites.

Following commands can be used to capture packets from ESXi hosts and can be analyzed by the physical switch vendor for potential issues on the network..

1. To capture packets on the uplink vmnic of source ESXi host where the VM is running:  

    # pktcap-uw --uplink vmnic --dir 2 -o /vmfs/volumes/Datastore_name/vmnic.pcap

1. To capture packets on the VMkernel interface used for replication traffic: 

    # pktcap-uw --vmk vmk0 --dir 2 -o /vmfs/volumes/Datastore_name/vmk2.pcap

1. To capture packets on the network adapter of the DR site replication appliance:

    # tcpdump -i eth0 -w /tmp/eth0.pcap 

For more details regarding the packet capture from ESXi hosts, follow this article: Using the pktcap-uw tool in ESXi 5.5 and later (341568)