Snapshot consolidation is failing in NBD mode during NFC timeout

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When backing up a virtual machine in NBD mode, the backup completes, but deleting the snapshot fails with a consolidation error.
• While the snapshot is removed, the virtual machine is left in a state where it requires consolidation. Subsequent consolidation attempts also fail due to locked files.
• In the hostd.log file, you see the error:
  
  TaskCreated : haTask-14-vim.vm.Snapshot.remove-158582742
  
  [67380B90 info 'DiskLib' opID=ecacd868-b5] DISKLIB-LIB: Cannot remove extent `/vmfs/volumes/50f6af3f-5b1
  Failed to consolidate disks in Foundry: Error: (15) The file is already in use

Cause

The issue occurs only when the backup application opens all disks of a virtual machine simultaneously, then proceeds to back them up one at a time. If there is large data to back up, it causes long idle times on the other open disks and these connections can be closed by an external application due to inactivity. This results in the disks not being closed correctly and the redo log files remain locked.

Resolution

This issue is resolved in vSphere Virtual Disk Development Kit 5.5, available at VMware Downloads.

To workaround the issue if you do not want to upgrade, back up one disk at a time ensuring that there is only one virtual machine disk open at one time per job.