Deleting files or folder on datastore fails with error "Deletion of file or directory [DatastoreX]ServerX from "DatastoreX" was initiated from VMware"XXX" and completed with status 'Failure'"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Deleting file from folder /var/run/log/Datastorex/vm folder/TestVM may report "Input/output error"
• You cannot touch a file under VM path
• Running "vmkfstools -V" may not help.
• Unable to touch file on the Folder where we have files to be deleted.
• Rebooting the host may not fix the issue

 

Validation:

1. You may see that the file could not be accessed or opened with error " Device or resource busy" and "Input/output error" under /var/run/log.vmkernel.log

 

<timestamp> In(166) Hostd[2099331]: [Originator@6876 sub=Libs opID=esxui-####-#### sid=52bfe141 user=root] DictionaryLoad: Cannot open file "/vmfs/volumes/########-########e-####-###############/***VM/***VM.vmx": Device or resource busy.

<timestamp> In(166) Hostd[2099331]: [Originator@6876 sub=Libs opID=esxui-####-#### sid=52bfe141 user=root] FILE:open error on /vmfs/volumes/########-########e-####-###############/***VM/***VM.vmdk: Input/output error

<timestamp> In(166) Hostd[2099331]: [Originator@6876 sub=Libs opID=esxui-####-#### sid=52bfe141 user=root] OBJLIB-FILEBE : FileBEOpen: can't open '/vmfs/volumes/########-########e-####-###############/***VM/***VM-flat.vmdk' : Input/output error (327684).

 

2.Deleting file from folder or accessing under /var/run/log/Datastorex/vm folder/TestVM may report "Input/output error" and ": Device or resource busy"

[root@Hostname:/vmfs/volumes//########-########e-####-###############/***VM/***VM/temp] rm -rf ***VM.vmx.lck

rm: can't remove '***VM.vmx.lck': Input/output error

 

[root@Hostname/vmfs/volumes//########-########e-####-###############/***mu] less *.vmx|grep vmdk

****VM.vmx: Device or resource busy

 

 

3. Other ESXi host in the cluster may hold the locks. Rebooting the host where locks are help may not fix the issue.

 

[root@Hostname:/] vmfsfilelockinfo -p /vmfs/volumes########-########e-####-###############/VMname/VMname-flat.vmdk

vmfsfilelockinfo Version 2.0

Looking for lock owners on "****VM-flat.vmdk"

"*****VM-flat.vmdk" is locked in Exclusive mode by host having mac address [##:##:##:##:##:##:##']

Trying to use information from VMFS Heartbeat

 

Host owning the lock on file is ##:###:##.##, lockMode : Exclusive

Total time taken : 4.371340672019869 seconds.


Environment

VMware vSphere ESXi 7.0.x

VMware vSphere ESXi 8.0.x

Cause

This issue is caused due to the file lock. There could be an outdated lock action which is leading to the file or folder deletion task failure.

These files can be locked because the virtual machines are still powered on or because one particular host has a stale lock on the file and/or directory

 

1. /var/run/log/vmkernel.log reports Exclusive locks

 

<timestamp> In(182) vmkernel: cpu23:2099451 opID=###)DLX: 2670: vol 'LM Class 2 Prod', lock at 171589632: Lock type: 10C00001. Exclusive Lock(s) held on a file on volume ########-########-####-###############. numHolders:0 gblNumHolders:0, volume state 10,$

 

 

2. You may observe "Lost previously held disk lock messages" under /var/run/log/vmkernel.log.

 

<timestamp> Wa(180) vmkwarning: cpu24:2099880 opID=###)WARNING: HBX: 5596: Replay of journal <type 6 addr 25165831> on vol 'Datastore#' failed: Lost previously held disk lock

<timestamp> Wa(180) vmkwarning: cpu46:2099271 opID=###)WARNING: HBX: 5596: Replay of journal <type 6 addr 25165831> on vol 'Datastore#' failed: Lost previously held disk lock

 

Resolution

1.Migrate the virtual machines residing on the datastore were files or folders need to be deleted using storage vmotion. Make sure the datastore is free with no VM or VM templates.

2.You may run VOMA using below commands after unmounting the datastores to release the lock.

voma -m vmfs -d /vmfs/devices/disks/naa.00000000000000000000000000:1 -s /tmp/analysis.txt

voma -m vmfs -f fix -d /vmfs/devices/disks/naa.naa.00000000000000000000000000:1 -s /tmp/analysis_fix.txt

Please refer to Using vSphere On-disk Metadata Analyzer (VOMA) to check VMFS metadata consistency

OR

Delete the datastore and recreate with new one. This option is feasible when workload is less and there are less number of Virtual machines to unregister for unmounting the datastore for running the VOMA.  See Create a VMFS Datastore