ESXi virtual machine power-on fails with ".vmdk locking conflict" due to hostd NFC memory exhaustion

Products

• VMware vSphere ESXi

Issue/Introduction

An administrator cannot power on a virtual machine (VM) in a vSphere cluster environment. The operation fails during initialization, indicating a file locking conflict on the virtual machine's flat disk file (-flat.vmdk), despite no active external lock being detected on the file system level.

Symptoms include:

• The virtual machine fails to power on with an exclusive file lease error.

• In the /var/log/hostd.log file, you observe log entries reporting a locking conflict for the virtual disk file:

--> Locking conflict for file "/vmfs/volumes/<Volume_UUID>/<VM_Name>/<VM_Name>_2-flat.vmdk". Kernel open flags are 0x4008. Owner process on this host is world ID <World_ID> with world name vmx-vcpu-0:<VM_Name>.

• In the /var/log/vmkernel.log file, you see buffered vs unbuffered open conflicts:

FSS: 6835: Conflict between buffered and unbuffered open (file '<VM_Name>_2-flat.vmdk'):flags 0x4008, requested flags 0x8

• Intermittent or recurring Network File Copy (NFC) allocation failures are recorded in the /var/log/hostd.log:

[NFC ERROR]NfcCheckAndReserveMem: Cannot allocate any more memory as NFC is already using 50082448 and allocating 720896 will make it more than the maximum allocated: 50331648. Please close some sessions and try again
[NFC ERROR]Couldn't reserve memory for size 720896
[NFC ERROR]NfcAioGetBuffer: Cannot allocate AIO buffer, malloc failed

• Running lsof reveals that orphaned or leaked threads inside the ESXi host daemon (hostd) maintain active file descriptors targeting the VM's disk infrastructure even when the VM is shut down.

Environment

VMware vSphere ESXi 7.0.x / 8.0.x

Cause

The root cause is memory exhaustion within the Network File Copy (NFC) service subsystem managed by the ESXi host daemon (hostd).

When large-scale concurrent storage tasks (such as network-block-device (NBD) backups, cloning, vMotion, or massive provisioning workflows) saturate the host, the NFC memory allocation pool (maxMemory) reaches its maximum predefined threshold. When memory runs out, hostd fails to allocate asynchronous I/O (AIO) heap buffers (malloc failed) and drops ongoing backup or management sessions abruptly. This improper termination leaves stale, orphaned NFC worker tasks active in the memory of hostd. These zombie processes retain internal kernel open flags (0x4008) and un-released file descriptors targeting the VM's disk files. Consequently, subsequent attempts by vCenter to open or power on the VM trigger a false-positive file locking conflict against these stale, inside-memory handles.

Resolution

To resolve the file lock blockade and prevent recurrence under high storage concurrency, restart the management agents to clear orphaned locks, then expand the hostd NFC buffer parameters:

• Clear Immediate Phantom Locks (Workaround): Evacuate any active virtual machines running on the affected ESXi host using vMotion, then restart the local management daemons via the command-line interface to flush the leaked process handles:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

• Increase the Hostd NFC Memory Limit (Permanent Fix): If the environment frequently scales concurrent backup or replication jobs, adjust the maxMemory limit inside the ESXi configuration database (configstorecli) as follows:

1. Import the configuration to a temporary JSON file:

  $ /bin/configstorecli config current get -c esx -g services -k hostd -outfile tmp.json

 

2. Edit the file:

  $ vi tmp.json

 

3. By default, maxMemory parameter is set to value 100663296. Set it to 150663296 (set the value as per the environmental requirement) :

 

<nfcsvc>

<path>libnfcsvc.so</path>

<enabled>true</enabled>

<maxMemory>100663296</maxMemory>           <================= increased this to 150663296 for example

<maxStreamMemory>35651584</maxStreamMemory>

  </nfcsvc>

 

4. Save the changes : Hit Esc -> :wq!

 

5. Apply the file to the database:

  $ /bin/configstorecli config current set -c esx -g services -k hostd -infile tmp.json

 

6. Restart hostd service:

  $ /etc/init.d/hostd restart

Additional Information

If the problem persists after the adjustment, consult your third-party backup software provider to optimize the concurrency scheduling of NBD backup sessions, or engage your storage vendor to review backend array latency anomalies during peak processing windows.