Understanding the NFS .lck lock file to understand the ESX host and NFS filename it refers to

Products

• VMware vSphere ESXi

Issue/Introduction

The purpose of this article is to understand which ESX host has the lock, or, which NFS file the .lck refers to.

Symptoms:

When an ESX host connects to a supported NFSv3 server, ESX does not use the traditional Network Lock Manager (NLM) protocol to handle NFS locking. Instead, a .lck file is created for each associated file and populate said file with the hostname in plain text, and also indirectly points to the inode of the file on the NFS export.

Environment

VMware ESXi

Cause

There can be times when it is unclear which lock file belongs to which file. A good example of this would be a shared swap directory which may contain multiple swap files from multiple VMs.

Resolution

Understanding the hostname:

1) SSH into the ESX host

2) Navigate to the appropriate directory:

# cd /vmfs/volumes/NFSExport/VM_Name/

3) Use the hexdump command to interrogate the .lck file for the hostname:

# hexdump -C .lck-e003090001000000

00000010 01 00 00 00 64 68 69 6e 67 2d 65 73 78 2e 76 6d |..........example.com|

00000020 77 61 72 65 2e 63 6f 6d 00 00 00 00 00 00 00 00 |example.com........|

 

In the above example, we can see this lock was being held for example.com

 

Understanding the filename this lock refers to:

1) SSH into the ESX host

2) Navigate to the appropriate directory

 

# cd /vmfs/volumes/NFSExport/VM_Name/

 

3) Observe the filename of the .lck file:

 

.lck-e003090001000000

 

4) Reverse the endianness of the .lck filename:

 

.lck-e003090001000000 becomes .lck-00000001000903E0

 

5) Convert hex to decimal:

 

0x00000001000903E0 = 4295558112

 

6) Use grep to find which inode this refers to:

 

# stat * | grep -B2 4295558112 | grep File

 

File: VM_NAME.vmdk

 

In the above example, we can see that the .lck file .lck-e003090001000000 belongs to the file VM_NAME.vmdk

 

leverage the below command to do the same thing automatically on a live ESX host:

 

# stat * | grep -B2 `v2=$(v1=.lck-e003090001000000;echo ${v1:13:2}${v1:11:2}${v1:9:

2}${v1:7:2}${v1:5:2});printf "%d\n" 0x$v2` | grep File

 

File: VM_NAME.vmdk

 

NOTE: modify the v2 variable to reflect the correct .lck file (Including the leading period (.))

Additional Information

Impact/Risks:
Manually deleting .lck files should never be done because it can cause random data corruption which may not be immediately visible, or, under the express instruction of a VMware Technical Support Engineer.