VDDK fails with error 14009 while backup application tries to read/write after session is disconnected

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:
When a backup application tries to do a read/write operations and there is already nbd failure reported by vddk, you see this symptom:

In the VDDK logs, you see entries similar to:

05/22/2019 22:07:00.357 : VixDiskLib: Detected DiskLib error 2338 (NBD_ERR_NETWORK_CONNECT).
05/22/2019 22:07:00.357 : VixDiskLib: VixDiskLib_Read: Read 8192 sectors at 1493057536 failed. Error 14009 (The server refused connection) (DiskLib error 2338: NBD_ERR_NETWORK_CONNECT) at 5814.
..
..
..
06/04/2019 22:49:08.901 : [NFC ERROR] NFC_NETWORK_ERROR: The session is in a faulted state: NfcAio_AIO
06/04/2019 22:49:08.901 : [NFC ERROR] NFC_NETWORK_ERROR: The session is in a faulted state: NfcAio_Wait
06/04/2019 22:49:08.901 : DISKLIB-LIB : no answer. numBytes didn't reach 0. Dumping iov.
06/04/2019 22:49:08.901 : ###### dumping content of iov ######
06/04/2019 22:49:08.904 : READ
06/04/2019 22:49:08.904 : startSector = 34209792
06/04/2019 22:49:08.904 : numSectors = 8192
06/04/2019 22:49:08.904 : numBytes = 4194304
06/04/2019 22:49:08.904 : numEntries = 1
06/04/2019 22:49:08.904 : entries[0] = D6392EE000 / 4194304
06/04/2019 22:49:08.904 : PANIC: VERIFY d:/build/ob/bora-9272949/bora/lib/disklib/diskLib.c:4086 bugNr=24092

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Cause

This issue occurs because in VDDK 6.7 and its first two update releases, VDDK may crash due to I/O retry after an NBD asynchronous I/O (AIO) error. If the backup application ignore the errors and continues reading or writing, it then results in an NBD network-connect error and subsequent program crash.

Resolution

This is a known issue affecting VMware VDDK 6.7.x and later versions.

• VDDK 6.7.0 Build 8173251 
• VDDK 6.7.1 Build 10362358 
• VDDK 6.7.2 Build 13015725

This issue is fixed in VMware VDDK 6.7 Update 3, available at VMware Download.