Backup of virtual machines fails with following error in vCenter : "An error occurred while quiescing the virtual machine"

Products

• VMware vCenter Server

Issue/Introduction

• Following error encountered when backup of virtual machine fails when initiated from Commvault backup server:
  
• No ping drops observed when pinging from the virtual machine to the backup server.
• Telnet connection over port 902 TCP from backup server to ESXi host is successful.
• Backup is successful when using NBD/SSL protocol but fails when using NBD protocol to perform backup. (Commvault utilizes the NBD (Network Block Device) protocol by default when performing backup of virtual machines (VMs) using its Virtual Server Agent (VSA).) 
• You may observe logs indicating that NFC connection timed out under /var/log/hostd.log

  2025-06-10T09:30:49.470Z Wa(164) Hostd[2099720]: [Originator@6876 sub=Libs opID=nbdmode-00000000ce9ac9d0] [NFC ERROR]NfcAioSendData: Failed to send data: NFC_NETWORK_ERROR
  2025-06-10T09:30:49.470Z Wa(164) Hostd[2099720]: [Originator@6876 sub=Libs opID=nbdmode-00000000ce9ac9d0] [NFC ERROR]NfcAioLogFatalSessionErrorLocked: A fatal session error occurred. The error was: 'NFC_NETWORK_ERROR' (3)
  
  2025-06-10T09:32:18.059Z Wa(164) Hostd[2099704]: [Originator@6876 sub=Libs opID=nbdmode-00000000d333b900] [NFC ERROR]NfcNetTcpSetError: Connection timed out
  
  2025-06-10T09:32:18.059Z Wa(164) Hostd[2099704]: [Originator@6876 sub=Libs opID=nbdmode-00000000d333b900] [NFC ERROR]NfcAioRecvData: Failed to receive data: NFC_NETWORK_ERROR
  2025-06-10T09:32:18.059Z Wa(164) Hostd[2099704]: [Originator@6876 sub=Libs opID=nbdmode-00000000d333b900] [NFC ERROR]NfcAioLogFatalSessionErrorLocked: A fatal session error occurred. The error was: 'NFC_NETWORK_ERROR' (3)

• The log snippets above indicate a network connectivity issue; therefore, performing end-to-end packet captures on the ESXi host can help identify where packets might be getting dropped.
  
  

Environment

VMware vSphere ESXi 7.x
VMware vCenter Server 7.x
VMware vSphere ESXi 8.x
VMware vCenter Server 8.x

Cause

• If a packet sent from the ESXi host does not receive an acknowledgment, it will be retransmitted repeatedly. After multiple failed attempts, the packet will be dropped, leading to a connection timeout between the ESXi host and the backup server:
  
• A firewall rule may be preventing communication between the ESXi host and the backup server.

 

Resolution

Verify the firewall configuration between the ESXi host and the backup server to determine whether any rules are preventing traffic flow.

Additional Information

Following article can be referred to perform a packet capture on ESXi host: Packet capture on ESXi