Error: VDDK async operation error: 16000 during Veeam NBD backup

Products

• VMware vSphere ESXi
• VMware Cloud Foundation

Issue/Introduction

• Veeam replication or backup jobs using Network (NBD) transport mode hang at the snapshot stage. Manual intervention is required to stop the tasks. The following errors appear in Veeam logs:
   Error (3) VDDK async operation error: 16000. Value: 0x0000000000003e80 Error (3) in c++: Disk read operation has failed.

• NFC error in /var/log/hostd.log:

Hostd[2108015]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf8d96240] [NFC ERROR]NfcCheckAndReserveMem: Cannot allocate any more memory as NFC is already using 49308080 and allocating 1048576 will make it more than the maximum allocated: 50331648. Please close some sessions and try again

 Hostd[2108015]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf8d96240] [NFC ERROR]Couldn't reserve memory for size 1048576

Hostd[2108015]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf8d96240] [NFC ERROR]NfcAioGetBuffer: Cannot allocate AIO buffer, malloc failed

• The above event indicates ESXi host running out of memory for its Network File Copy (NFC) service.

In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]NfcServer_SessionCleanup: Cleaning up server session with currentState: NFC_IDLE
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]Nfc_CloseSessionEx: session=4CF8C526E0
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC ERROR]NfcNetTcpSetError: Broken pipe
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC ERROR]NfcNetTcpWrite: bWritten: -1. Errno: 32.
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC ERROR]NfcSendMessage: NfcNet_Send failed: NFC_NETWORK_ERROR
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]NfcUtil_DestroyTimer: Removed periodic timer(4CF8C44120) for session=4CF8C526E0, OpID: nbdmode.
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]NfcSessionStats: session=4CF8C526E0, type=server, clientName='vddk', streamMode=0, fssrvrMode=0, aioMode=1, version=11, remoteVersion=11, currState=NFC_IDLE, prevState=NFC_AIO_SESSION, returnCode=NFC_NETWORK_ERROR (3), detail="The operation experienced a network error -- Failed to send complete message: Broken pipe", opID='nbdmode', sessionDurationUs=30440350, sessionIdleTimeUs=2373 (0.0%), fileIoIdleTimeUs=30408502 (99.9%), attemptedFileTransfers=0, successfulFileTransfers=0, totalBytesTransferred=2097152, totalFilesSize=0, fileLockFailureCount=0
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]readIoLatencyStats: count 4 min/max/avg 6/30573/16327 usec
In(166) Hostd[2099479]: [Originator@6876 sub=Libs opID=nbdmode-0000004cf80a95c0] [NFC INFO]netSendLatencyStats: count 49 min/max/avg 4/2696/79 usec

• Because the pipe was broken, the ESXi host failed to send its message back to the backup proxy.

Environment

VMware vSphere ESXi: 7.0, 8.0, 8.0U3e

VCF: 5.2

Veeam Backup

NBD (Network) Transport Mode

Cause

The ESXi host experiences Network File Copy (NFC) memory exhaustion. VMware hard-codes the NFC memory limit to approximately 48–50 MB. Each active NBD connection consumes a portion of this buffer. When concurrent tasks exceed the host's capacity, the NFC service cannot allocate additional memory, leading to broken pipes and hung tasks.

 

Resolution

Reduce the load on the ESXi host's network stack by limiting concurrent backup tasks.

1. Open the Veeam Backup & Replication Console.
2. Navigate to Backup Infrastructure > Backup Proxies.
3. Right-click the impacted VMware Proxy and select Properties.
4. Lower the Max concurrent tasks limit. It is recommended to set this to 7 or fewer for NBD mode.
5. Alternatively, stagger backup job start times to prevent multiple jobs from hitting the same host simultaneously.