Virtual machine backups via PowerProtect Data Manager (PPDM) fail at the snapshot consolidation step due to locked .vmdk files
search cancel

Virtual machine backups via PowerProtect Data Manager (PPDM) fail at the snapshot consolidation step due to locked .vmdk files

book

Article ID: 430403

calendar_today

Updated On:

Products

VMware vSphere ESXi 8.0 VMware vCenter Server 8.0

Issue/Introduction

  • Multiple virtual machine backup operations fail in PPDM with the following error: 

message#1 : Unable to collect list of allocated blocks for 'Hard disk 1'.  Unable to query allocated blocks of virtual disk '[Datastore_name] VM_name/VM_name.vmdk'.  Unable to open source VMDK '[Datastore_name] VM_name/VM_name.vmdk'.  VDDK Error: 16052: The specified key is not found in the disk database.
message#2  : CBT data not available: Synthesizing used block list to cover entire logical disk.

  • Following error message is observed in vCenter UI under the VM tasks indicating consolidation failure: Unable to access file since it is locked filePath: /vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk host: ESXi_Host_Name,ESXi_Host_IP mac: ['##:##:##:##:##:##'] id: ####### worldName: vmx lockMode: Read-Only

  • Manual virtual machine disk consolidation may complete successfully or need ESXi host management agents to be restarted to facilitate the consolidation.

  • The following error messages are observed within the virtual machine and ESXi host log files:

/vmfs/volumes/Datastore_name/VM_name/vmware.log

YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - AIOGNRC: Failed to open '/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk' : Failed to lock the file (40003) (0x42003).
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - DISKLIB-LIB_MISC   : DiskLibGatherFileLockInfo: Lock conflict on "/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk", FSS openFlags 0x2, possible local worldID: ####### and worldName vmx-vcpu-#:VM_Name, possible VMFS on-disk MAC address ########-########-####-############ and lock mode 0x2
YYYY-MM-DDTHH:MM:SS In(05) vcpu-0 - [msg.disklib.lockinfo.local] Locking conflict for file "/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk". Kernel open flags are 0x2. Owner process on this host is world ID ####### with world name vmx-vcpu-#:VM_Name.

/var/run/log/hostd.log:

YYYY-MM-DDTHH:MM:SSZ Db(167) Hostd[#######]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/Datastore_name/VM_name/VM_name.vmx] Consolidate Disks translated error to vim.fault.FileLocked
YYYY-MM-DDTHH:MM:SS0Z Db(167) Hostd[#######]: [Originator@6876 sub=Vigor.Vmsvc.vm::/vmfs/volumes/Datastore_name/VM_name/VM_name.vmx] Consolidate Disks message: An error occurred while consolidating disks: Failed to lock the file.
YYYY-MM-DDTHH:MM:SSZ Db(167) Hostd[#######]: --> Locking conflict for file "/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk". VMFS on-disk lock owner is host with MAC address ########-########-####-############ and VMFS lock mode 0x2.
YYYY-MM-DDTHH:MM:SSZ Db(167) Hostd[#######]: --> Locking conflict for file "/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk". Kernel open flags are 0x2. Owner process on this host is world ID ####### with world name vmx-vcpu-#:VM_Name.
YYYY-MM-DDTHH:MM:SS0Z Db(167) Hostd[#######]: --> Consolidation failed for disk node 'scsi0:#': Failed to lock the file.
YYYY-MM-DDTHH:MM:SSZ Db(167) Hostd[#######]: -->
YYYY-MM-DDTHH:MM:SSZ In(166) Hostd[#######]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 9403 : Virtual machine VM_Name disks consolidation failed on ESXI_hostname in cluster . in ha-datacenter.

  • Rebooting the ESXi host to release the file lock does not provide a permanent resolution, as the issue recurs during subsequent operations.

  • Analysis of the hostd trace logs confirms that the Network File Copy (NFC) service successfully establishes a file handle to the target .vmdk for backup data ingestion. PPDM fails to close this active NFC session before issuing the snapshot consolidation request. 

    • The below log snippets from /var/run/log/hostd.log (captured with Trivia logging enabled) demonstrate the opening operation for the data ingestion session:

YYYY-MM-DDTHH:MM:SSZ Db(###) Hostd[#######]: [Originator@6876 sub=DiskLib opID=nbdmode-################] DISKLIB-VMFS  : "/vmfs/volumes/Datastore_name/VM_name/VM_name-flat.vmdk" : open successful (##) size = ############, hd = #######. Type 3

    • Below snippet is observed in /var/run/log/hostd.log when the consolidation request is received:

YYYY-MM-DDTHH:MM:SSZ  Db(###) Hostd[2105618]: [Originator@6876 sub=Libs opID=########-##-#### sid=######## user=vpxuser:DOMAIN\user_name] VigorTransport_ClientSendRequest: opID=########-##-####  seq=####: Sending Snapshot.Delete request.

Environment

  • VMware vSphere 8.x

Cause

  • This issue occurs because the backup software, PPDM, fails to close the active NFC session established for data ingestion prior to initiating a virtual machine snapshot consolidation request.

  • Analysis of the ESXi host trace logs (/var/run/log/hostd.log with Trivia logging enabled) under operation ID nbdmode-################ confirms that the NFC service successfully establishes a file handle to the target .vmdk for backup data ingestion.

  • The ESXi host never receives the file close request for this ingestion session before the snapshot consolidation operation.

  • Consequently, an active file lock remains on the .vmdk during the consolidation attempt, causing the operation to fail.

 

Resolution

Engage Dell Technical Support for further investigation regarding PPDM failing to close the NFC session prior to the consolidation request.

Powered by Wolken Software