"Virtual machine Consolidation Needed Status" "Unable to access file since it is locked" "Failed to lock the file" VM Consolidation fails or getting alert message

Products

• VMware vSphere ESXi
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• Post Virtual Machine Backup, Multiple VM's consolidation pending.
• Multiple VM Disk consolidation pending alert "Virtual machine Consolidation Needed Status"

• Manually consolidating the VM fails with error: Unable to access file since it is locked An error occurred while consolidating disks: Failed to lock the file. Consolidation failed for disk node 'scsi0:O': Failed to lock the file.

• VM Consolidation fails error: Disk consolidation for VM <vm_name> has failed: Unable to access file since its locked.

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware vCenter Server 7.x
VMware vCenter Server 8..x

Cause

Backup workflow
- Backup software copies data, it will temporarily access the disk, leading to a read lock during the copy.

If backup still has active read on VM Disks and a consolidation request comes in, then consolidations will fail.

hostd.log 

YYYY-MM-DDTHH:MM:SS.672Z info hostd[2726954] [Originator@6876 sub=Libs opID=nbdmode-0000008 *** ] Setting NFC log level to 0

YYYY-MM-DDTHH:MM:SS.672Z trivia hostd[2726954] [Originator@6876 sub=PropertyCollector opID=nbdmode-0000008 ***]

TriggerProcessGUReqs: Session ########-####-####-####-############

YYYY-MM-DDTHH:MM:SS.672Z trivia hostd[2726954] [Originator@6876 sub=PropertyJournal opID=nbdmode-0000008 ***] Notified a listener

on ha-host for path "dataTransportConnectionInfo"

YYYY-MM-DDTHH:MM:SS.672Z trivia hostd[2726954] [Originator@6876 sub=PropertyProvider opID=nbdmode-0000008 ***] RecordOpInt called

for dataTransportConnectionInfo.

YYYY-MM-DDTHH:MM:SS.672Z trivia hostd[2726954] [Originator@6876 sub=PropertyProvider opID=nbdmode-0000008 ***]

[RecordAndNotifyChange] Removing stored value for property dataTransportConnectionInfo

YYYY-MM-DDTHH:MM:SS.672Z verbose hostd[2726954] [Originator@6876 sub=PropertyProvider opID=nbdmode-0000008 ***] Recordop ASSIGN:

dataTransportConnectionInfo, ha-host. Sent notification immediately.

YYYY-MM-DDTHH:MM:SS.672Z trivia hostd[2727088] [Originator@6876 sub=PropertyCollector opID=nbdmode-0000008 ***] ProcessGUReqs

Start: Session ########-####-####-####-############

YYYY-MM-DDTHH:MM:SS.673Z trivia hostd[2727088] [Originator@6876 sub=PropertyCollector opID=nbdmode-0000008 ***] ProcessGUReqs End:

Session ########-####-####-####-############ (1 filter updates, 1 GUReqs)

YYYY-MM-DDTHH:MM:SS.673Z trivia hostd[2727048] [Originator@6876 sub=DiskLib opID=nbdmode-0000008 ***] DISKLIB-LIB : Removing

obsolete flag OPEN_BUFFERED.

YYYY-MM-DDTHH:MM:SS.676Z trivia hostd[2727048] [Originator@6876 sub=DiskLib opID=nbdmode-0000008 ***] DISKLIB-VMFS :

"/vmfs/volumes/datastore/vmname-flat.vmdk" : open successful (14) size = 536870912000, hd = 1713111519. Type 3

YYYY-MM-DDTHH:MM:SS.676Z trivia hostd[2727048] [Originator@6876 sub=DiskLib opID=nbdmode-0000008 ***] DISKLIB-LIB_BLOCKTRACK :

Resuming from change tracking info file /vmfs/volumes/datastore/vmname/vmname-ctk.vmdk.

There is an operation with opID nbdmode-0000008 *** that opened the disk at YYYY-MM-DDTHH:MM:SS.676Z. The "(14)" is the disk open flags includes the flags for opening with a read only lock. The operation closes the disk at YYYY-MM-DDTHH:MM:SS.779Z. During this time, any consolidates will fail because of the read lock.

These logs had the session close because of a network error. Based on the need to restart hostd to workaround the issue, the session was never closed.

Resolution

Increase Logging Level:

1. SSH into the host.
2. Increase hostd main log level and history:

$ vim-cmd hostsvc/advopt/update Config.HostAgent.log.level string trivia
$ esxcli system syslog config logger set --id=hostd --rotate=20 --size=50000

3. Increase NFCLIB logging. From https://knowledge.broadcom.com/external/article?legacyId=82227

3a. Dump hostd internal config to a file:
$ /bin/configstorecli config current get -c esx -g services -k hostd -outfile tmp.json

3b. Open file in vi:
$ vi tmp.json

3c: Find the section titled "nfcsvc". Within it, find the line "log_level": "INFO". Change INFO to DEBUG and save.
3d: Update the hostd internal config with the edited file:
$ /bin/configstorecli config current set -c esx -g services -k hostd -infile tmp.json

4: Restart hostd:
$ /etc/init.d/hostd restart

5. Verify all the above commands worked by running this:
$ cat /var/log/hostd.log | grep -i "NFC connection accept"

If the above steps worked, there will be an output similar to below:

[root@s**-10-**-1**-150:~] cat /var/log/hostd.log | grep -i "NFC connection accept"
YYYY-MM-DDTHH:MM:SS.649Z info hostd[134631] [Originator@6876 sub=Libs] [NFC VERBOSE]NFC connection accept timeout: 180000 milliseconds