VM backup fails with NFC_NETWORK_ERROR due to Asymmetric Routing.

Products

• VMware vSphere ESXi

Issue/Introduction

Backups for virtual machines residing on a standalone ESXi host fail during the data transfer phase using Commvault backup (NBD mode). While virtual machine snapshots are created successfully, the Network File Copy (NFC) session is abruptly terminated shortly after initialization.

Symptoms:

• Commvault backup jobs fail after the snapshot is completed.
• The ESXi hostd.log records the following error sequence exactly 30 seconds after the NFC connection is established:

  [NFC ERROR] NfcNetTcpSetError: Connection reset by peer
  [NFC ERROR] NfcNet_Recv failed: NFC_NETWORK_ERROR
  [NFC ERROR] NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
  [NFC ERROR] detail="The operation experienced a network error -- Failed to send complete message: Broken pipe"

   

Environment

VMware ESXi Server 8.x.

Cause

• The issue is caused by asymmetric routing between the Backup Proxy VM and the ESXi host management/backup interface.

  In this scenario, the outbound packet from the Backup Proxy to the ESXi host follows one path, while the return packet from the ESXi host follows a different path. Stateful firewalls or security appliances along these paths fail to see the complete TCP handshake or session state, causing them to sever the connection. This is reflected in the logs as a Connection reset by peer.

Resolution

To resolve this issue, network symmetry must be established between the backup proxy VM and the ESXi hosts. Customers should engage their network team for further investigation and to correct the routing configuration