Consolidating disks associated with a backup snapshot fails with the error: The file is already in use

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• When consolidating the disks associated with a backup snapshot, you receive the error:
  
  The file is already in use
  
  
• After enabling increased logging, in the hostd.log file you see messages similar to:
  
  
  • [2C581B90 info 'BlklistsvcPlugin' opID=9e67f392-59] Block list built for [] /vmfs/volumes/50194e8a-########-####-#######95c/example_vm/example_vm.vmdk; lease is 528d29b7-c357-####-####-########dcc
    
    
  • [4B6B3B90 verbose 'vm:/vmfs/volumes/50648696-74a####-####-########727a/example_vm/example_vm.vmx'] Done disk consolidation.
    [4B6B3B90 info 'vm:/vmfs/volumes/50648696-74a9####-####-########27a/example_vm/example_vm.vmx'] Failed to consolidate disks in Foundry: Error: (15) The file is already in use
    
    
  • [4B6B3B90 verbose 'vm:/vmfs/volumes/50648696-74a9####-####-########27a/example_vm/example_vm.vmx'] Done disk consolidation.
    [4B6B3B90 info 'vm:/vmfs/volumes/50648696-74a9####-####-########27a/example_vm/example_vm.vmx'] Failed to consolidate disks in Foundry: Error: (15) The file is already in use

Environment

Virtual Disk Development Kit 5.1
Virtual Disk Development Kit 5.0
VMware Virtual Disk Development 1.2

Cause

This issue occurs because the consolidation attempt is made prior to the backup proxy lease being cleaned up.

The series of events should match:

1. Backup starts
2. Snapshot created for backup
3. Lease established
4. Failure of SAN mode, then fallback to NBD mode
5. Backup finishes
6. Consolidation attempt (fails)
7. Lease released

Note: The backup proxy must be a separate physical machine to trigger the issue; virtual machine based proxies cannot access SAN mode backups.

Resolution

This is a known issue affecting Virtual Disk Development Kit (VDDK) 5.0 and 5.1. The issue is fixed in VDDK 5.1.2, 5.5 and in 5.0 Update 3.

After the lease is released, normal consolidation is possible.
To work around this issue, configure the backup software to attempt NBD directly instead of attempting SAN-based backup first, or ensure the correct block devices are present on the backup proxy server prior to attempting a SAN-based backup.

Note: Auto mode attempts SAN-based backup first then falls back to NBD if that fails. This situation will also trigger the issue.