Virtual Machine File Lock most common scenarios:
1-Proxy Back Up Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files.
2-An ESXi Host is locking the Virtual Machine files abnormally.
First Scenario
- The most common case of virtual machine file lock is when the Backup Proxy Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files. This issue can be resolved using the below steps:
- Right-click the backup proxy virtual machine.
- Click Edit Settings.
- Expand all the Hard Disk(s).
- Select the Hard Disk(s) which belong to the virtual machine that has the problem.
- Click on the X beside the Hard Disk to unmount the Hard Disk from the VM.
Caution: Do NOT select Delete files from the datastore.
- Click OK.
- Consolidate/Delete the snapshot on the VM.
" Power on the VM if the issue was unable to power on"
Second Scenario
- If the first section is not the case, then we need to identify which process is locking the virtual machine files to release.
- Connect to the ESXi host the virtual machine is on with an SSH session
- Navigate to the VM directory and take a note of where the VM is currently writing.
To know the VM directory, follow the below steps:
1- Run the below command
#
vim-cmd vmsvc/getallvms | grep -i "VM_Name"![](https://api-broadcomcms-software.wolkenservicedesk.com/attachment/get_attachment_content?uniqueFileId=1512265701527)
2- Run the below command:
#
cd /vmfs/volumes/datastore_name/virtual_machine_name/ --->
From step 1![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmUAAAAkCAYAAADIFE3HAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAAzdSURBVHhe7Z0Pdts2DMaX9DT1epI4PUfsXCNWjtHYt2jcw2kE/wkgQVKyJNtZv733e5tJCgQoTfgCyvI/5p8eAAAAAADcHLURAAAAAABcF7Xx9my7/s+fP47TXh8D7oLN/tSf9hu17x64d//W5tbxb/ZHO/+D0ge+PpvdsT/u/+0fHvR+AMAEnPA59fuN0nkrNvv+xHzadn/6bpuMuQI0bymZUR+t3TrJbtt3XpCun0z9XBcLXzq+67dq3zzCGgcuuwaW949ETvDplmJnHHr8peuXxyZx/z/q/bX7h5n/bOb3CZsEWn58sPGQHHsbdsez4p+hexbCY3s4m/XLxQiJlLM/RuufxfbQn8/cr65/flh23Sj+szkfr/8+jBDST/3Bnt8xYwEATW6VVPancqLM/rLfdneV/Gxi6rar+5Wtw2qYxHmhKFvNRxLmC1RI11zD652fy9F8pLbS9cvHb7uzEcIu4e9Pg/CiMUIgGztnc660pJxWyfhnYf94XFSU7Y8HKxS0vjHs0uM3u/5o1myKwFqzgrQ9mPUaJZouY0fnY4R9VMkAWJj0pkzsT+GvvI29AYe/yLiICmO0vmq/rYLxPge/yW+7kADor3zXL/xMbSjJW8xvEkpM8OxYO2fcJpVJJxyrrU+kKsqC72Xx2YInyKmIKpOJnSfVdP267aWijGKU8fF5re9xfX0b/dv4Etcm9g920gqZJfGveH4FE/1jNlrXN6Gdn3DthOtZVKTCujfin3V9C/L4BdXrl3w/R9HEoRgzUUaChY1xmPlZlSylZJ/iP/JqkIktTfp7Xs3y8dsx6bEemmeKgAmiTBMdYythRcFCAi+Lb5qAbIkyV+0Kc+TVNKryxf7Ta/9q1pDbC6KMRPRQmXN2hjlRJQNgcYo35ZAsws3efA43YkoI4jgay5JCq9+NKSeLIMoooQXhxO0JgWGwiZDZT5OlTYx8fpv0vA2fAEWS8WhJV5D4JaGE6G5kWZy0HnxtrQ/5uOb8BWy8VAnxn+36RNFJfvH1834m52cMJf+ydoo3+MPX266Di5uOEeeAxhV8Su1n59dzkX+GMdcvUbMvYjHHx3Ej4p99fXtK/kW4Xwo1UeaSdMDEoAiT1rNkJftp1cwKAxNfEDepXRIYFD8XP/MrZYPoKwmvVpWo1J9WoazIs/6P97cmysh34TNteR4H+1aQsa1YO/9ZbldKUZaKseE4VMkAWJjiTbl0w6akwhJYIFa3Wv3+8xhRFpOTktRkUiC4vSCItD5PtCP94sxNajqDKBqSm+5Dc34VY19J0BHNZ1qL2jEqFEfpHMq+rEoX5jK+BDFEsY4VZaPO7xz/Rly/ROn8ZLGk128tfuoXsWkxzo3f07h+J1XKsmvYzF+pkhGqfRO/VumyMUbRsu0PYgzvcyy1fVkTHi1RovanVTIWw5Rnw4qijOwf8m3WYbxZOybQSsRK2+m1MBZVMgBWoXhTLt2wW0lrZFIbI8pEgooJ2tzspwoISn7pMdTmbYsEwygl3Ugjqc2lOb9KY300n2mdJ65py7eYuGmd+fXA52J9WaKf4hPZScbO8m/E9UuU5shi4WvejH+Z67sVv4X7pTBalCltrSoZodsPooG3NaBKkImfH7P4M2UKF4mykaKoRSrKaK6DWcuHBUVZ15nryotFVMkAuBLFm3Llhp1ur6SJpNWfjaGkwv6qbyUU2q6p9Wfzs+RHkP1hPuObEWaavWZiqyY1Z5fHNZXm/AXy9XG+xKSf+FTa/iqT28gxY4zN7FwsIMpa53eWf4a8zY3lY4jS+RHtFAe/vkbEP/f6Hhe/oXr9kh9nRTQp54rmO3Of6LOZv5GwS/bjtxqT9kD2pQASZSZ+KcqcCIn9tB4ThNB6oozFN0PQFEWZ+e/8IX0pxGh+6ZfpP5tz8WyO8W3Rhv22J54lA+Bq5DdluqH76hQjS5hz+glKJLE/SSD2+KGNnvFJjxcPOic2an2UUEK7tcn8cGuhx89tWBGT9mdJO9hJYptAKemPIV0DsX5i7Q3mL2Lra0EEpYz1y641Fwv8uqB27wfZCueF/MzPnx/v7dTOL3Gxf4Hq9du+PuTxRqzsfZxmncfETzYuvb6JVvyt6zf4MmDs+2Sd9zn4fK0qGfXL4wf7AfEgfxzjREWtL2KFWKW/An+ezJ6/VOAYATP0B8L2Y6tfmyPvL2JFEj9uwIoqGmO3SKX92OeJ25NKv+tzz5jZ58/8mNOrE3KokgGwImOS1y0QN38tcf4FtJLrbSBRklaS7ol7929tbh2/md9WzcaLIPCVoCpZWokDACyGEz5/cxK7R4ZqzL2KZgAAAAAsjtoIAAAAAACui9oIAAAAAACui9oIAAAAAACui9oIAAAAAACui9oIAAAAAACui9p4e/i7tEa+P+urcp+vvhi4d/8AAOD/ymb3Yd8L94j3wv0d3OUrMezLNwef6GWX4uWnV6L2ZvXwAs75YoVef9Fd/ILZGulLQi9bw+X94++gg9gDAIAST/3h89A/P7r3wpFA018eTC/7fbyLd8ftPj7Fi4kj78/9o4kjjHt6+zRi84cRm/Kdht9NjJ8+xtNr3j+Lp7f+9yd/sfJ7//Pbsuv28uu3e/nyj28FIf3Uv/12a0TxfWNrYrlVUqz99mVWmdnWfw7m2lhR4d/IPtev1apQJGwXqDCuWSVb0zYAAHx10ioZ//x0+OwPz0ZQmP/efXz0+wVF2e7j0G+9ENT6W+w+3szxzJ/Nrv84bIUoa0Hi7EMRbUvw9ObXa6Xq48uvX/2+KMoc319+2fhGibLwMy7UZwWIVZRSRMmfeskFVrFf+Qkbgldxhh9/LrxENbWhiA8x/8kk/zCGHWvnjNukQ2VuiLlRyamKsuB7rcqUV6F4dcva5tu41Eb/NvFE27F/sJNWyCzJGhXXRzDRP2ajdX0QmigLax+uB1GRDOeuEf+s60Pwd/6SBADgHpBVsrSfizLRRwKI/8zW6TUTQ6KaZe9/fgwdKypJjndtngpBlGlbr2MrYUVR9j3xUYmvRUuUvXxQtSvEn1fTnt5+xxho/tfTUYiwIMr+3f0axiV2JokyS0h2IVmZzyFRUkITx9HYJCHX+t0YPVETQZRRQg7CidtLf5DZJnJmP032NrHz+W3S9jZ8AueiMKCJBkHil6Qtykr2s3Zav/BTU9xfavf26RgRA41L1jzQXB/PRf4Zxpx/omZfxMLXeUT8s68PD41L2wAAYA1S8dF6lqwkytKqmd3yPA7CJbVLds6JsJlfKRtEnxNe+RgSXc4PXVCVRJmN78cQtxV5Jr5M3FSoiTISZEfyOdijLc/j3th/tJ+tIOt+xvm+v5DIlNuVUpR16hbpRaJM7aOkqPwWZaxutfr95zGiLCZXJSk75cnh9oYKW97niXakX5w0eWeU1mgUeRWq1CdEBvkdBASJEr/WXJRk4zJGrM8c/0acf6K0vlks6fmvxR/PawqPZXz8sg0AANZBVsTqVTJCFWVplSzyzmyRbf6D9LzPsdT2ZU1YXiTK0ioZi2HKs2FFUfb9Rd1mDeMfH576t4+96XcCrYR7rsz4ZcTuN+OXNuZLijKRYKPAMMmyKDYKUPJOj6E2b1sIAMaaoqxlO4oMJjwsLVGijWuhrM8s/0acf6I0RxYLX+dm/MtcH634AQBgLVpVMkKvlBnBZUTDQ5roazwd+s8jCY3hmMWfKVO4rFIWRNGE+BRSUUZzHbZGdG2WE2Vd996fC1UyYjlRZki3h9JE2OrPxlBSZNWKVkKk7aZafzY/S94E2R/mo4qIbq+ZmCtrFOxOr0IF3JplsSwgylrrM8s/Q97mxvIxRGl9RTvFwc/PiPjnXh/j4gcAgDVoV8mI0vYltdcEXfalABJlh2chfsQY6jfiouUPZz1RRgJJ/9bmFIqizHx+8dujw/pJIUbbl3b+KKbctym7n/n25aP9tufs7UtKSL46xcgS/px+ghJh7E8SoD1+aKNnlNLjxYPaiY1aHyXv0G5tMj/cWujxcxv2GaS0PxMdwU4Sm6EkRlKsr1ws8HWldu872QpxUUx5/H68t1NbH+Ji/wLV899eX3m8EVB7H2dnRHBor8RPNi69Poix8QMAwFIE0dOqktlnxMRrMbTtx/S1FMOYWl+cxwqxsE2o9FeQ9ulZKy5IjIARW6cBmoPGtfq1OVz/qO3L7JUYA+9BVNEWZjIm9nk78osAsp+/EmN7GL4QkL7+Ynql7MaEJGvREv+XhURJXl26H+7dv7X52+MHANwOqpIdZSUL/C8pijInfJCEAAAAAADWxW13UqVNfXls8gEAAAAAANwGtREAAAAAwFF4seyAfO7r6hRflxFY/ieVVkJtBAAAAAAA10VtBAAAAAAA10VtBAAAAAAA10VtBAAAAAAA10VtBAAAAAAAV+Of/j/neM8y0agw2wAAAABJRU5ErkJggg==)
3-Run the below command:
# grep -i vmdk *vmx
![](https://api-broadcomcms-software.wolkenservicedesk.com/attachment/get_attachment_content?uniqueFileId=1627795412716)
*Now the VM is writing on Test_File_Lock-000002.vmdk in the example
- Check which ESXi Host is locking the VM file(s) and clear the lock by running the below script in the VM directory:
# ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; done
Note: The script will not work if the virtual machine files has spaces. In this case, you will need to run the command vmfsfilelockinfo -p on each and every file listed under the virtual machine directory.
Example: vmfsfilelockinfo -p Test_File_Lock-flat.vmdk
There will be two probabilities: (after running the script)
- The VM files are locked by same ESXi Host where the VM is running (owning Host).
Notes:
1-If the VM is powered on so it should be normally locked by the ESXi Host it is running on (referred by the MAC address in the script output)
2-If the VM is powered off then there shouldn't be an output from the script as VM files shouldn't be normally locked by any process (from any ESXi Host)
To know what is locking the VM files and release the lock , follow the below :
#
lsof | grep -i "VM_Name"a- It could be vpxa-worker ----> this can be fixed by restart
vpxa agent using the below commands:
# /etc/init.d/vpxa stop# /etc/init.d/vpxa startb- It could be
hostd or
hostd-profiler ---> this can be fixed by restarting
hostd agent using the below commands:
# /etc/init.d/hostd stop# /etc/init.d/hostd startc- The VM files might be stored in a temp directory
/dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
#cd /dev/deltadisks![](https://api-broadcomcms-software.wolkenservicedesk.com/attachment/get_attachment_content?uniqueFileId=1609676673055)
#
rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like
"8857142-Test_File_Lock-000002-delta.vmdk"Warning: Make sure that the directory is
/dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.
Notes:
- If the VM is powered off then, this script should show no output as no process should be locking the VM files. So if the script shows and output and the VM is powered off, then there is a stale process that will only be released by rebooting the ESXi Host.
- If you got the error "device or resource busy" while trying to run the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/detadisks and perform the remove again.
- The VM files are locked by another ESXi Host
All the files are locked by Host that has the same MAC address except one file "Test_File_Lock-flat.vmdk" that is locked by two ESXi Hosts (two MAC addresses)
As the VM is powered on so there should be normal lock from the owing ESXi Host but there is another abnormal lock coming from another Host that has the MAC address "00:50:##:##:##:16" in the example
To release the lock from the other ESXi:
- Identify which ESXi Host locking the the file from the vCenter GUI
- Expand the Cluster and select any ESXi Host
- Click on Configure
- Click on Physical adapters and look for the MAC address "Output from the previous step"
Note: You can identify which ESXi Host has a NIC with the MAC address resulted from the script using the below command: (only if vSphere High Availability is enabled)
# esxcli network ip neighbor list
Output should be like the below:
Neighbor Mac Address Vmknic Expiry State Type
-------------- ----------------- ------ -------- ----- -------
192.168.0.10 00:50:##:##:##:09 vmk0 1190 sec Unknown
192.168.0.1 00:50:##:##:##:01 vmk0 989 sec Unknown
192.168.0.82 00:50:##:##:##:16 vmk0 572 sec Unknown ----> 192.168.0.82 is the IP address of the locking ESXi Host
192.168.0.51 00:50:##:##:##:0a vmk0 662 sec Unknown
192.168.0.83 00:50:##:##:##:1c vmk0 733 sec Unknown
192.168.10.120 00:50:##:##:##:09 vmk1 122 sec Unknown
192.168.20.82 00:50:##:##:##:c6 vmk2 833 sec Unknown
- Connect to the ESXi Host where the virtual machine is on with an SSH session
- To know what is locking the VM files and release the lock , follow the below :
# lsof | grep -i "VM_Name"
a- It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:
# /etc/init.d/vpxa stop
# /etc/init.d/vpxa start
b- It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:
# /etc/init.d/hostd stop
# /etc/init.d/hostd start
c- The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files
#cd /dev/deltadisks
#rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"
Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.
Notes:
- If the VM has no clear process locking the files (no output from the command lsof | grep -i "VM_Name") then this is a stale process and the locking ESXi Host has to be rebooted to release the lock.
- If you got the error "device or resource busy" while trying to run the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/detadisks and perform the remove again.