VM replication fails during VM reconfiguration task due to ConfigStore "database is locked" error on ESXi Host

Products

• VMware vSphere ESXi

Issue/Introduction

• VM replication is initiated by third party software. 
• The reconfigure starts normally and enters the reconfiguring state, but fails when attempting to add the new VNIC details.
• When hostd attempts to write the new NIC configuration, the ConfigStore repeatedly fails to start a transaction due to the database being locked:

var/run/log/hostd.log

YYYY-MM-DDTHH:MM:SS Hostd[2101348]: Task Created :haTask-4-vim.VirtualMachine.reconfigure-#####
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: State Transition (VM_STATE_OFF ->VM_STATE_RECONFIGURING)
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: VmReconfigure: adding opaquenetwork vnic
..
..
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: error [ConfigStore] Exec failed:'database is locked'
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: error [ConfigStore] Failed tostart transaction
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: Add VNIC info to config store failed: Failed to start a transaction. (retry 0)

• After exhausting retries, the reconfigure operation aborts and the VM exits the reconfiguring state without applying the NIC changes

YYYY-MM-DDTHH:MM:SS Hostd[2101340]: Add VNIC info to config store failed: Failed to start a transaction. (retry 4)
..
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: Unexpected exception during reconfigure
YYYY-MM-DDTHH:MM:SS Hostd[2101340]: State Transition(VM_STATE_RECONFIGURING -> VM_STATE_OFF)

/var/log/vmware/vpxd/vpxd.log

YYYY-MM-DDTHH:MM:SS error vpxd[3836563]:vim.VirtualMachine.reconfigure failed with vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS error vpxd[3836563]: -- ERROR task-16923208 --vm-#### -- vim.VirtualMachine.reconfigure

Environment

ESXi version 8.0.3 

Cause

A vNIC memory leak within the nsxt_vnic_cache is causing unbounded growth of the vm_vif_state object. This unchecked growth consumes a disproportionate amount of the ConfigStore ramdisk and results in a vNIC leak issue.

Resolution

This issue is resolved in VCF 9.0.

There is currently no functional workaround on ESXi 8.0.3 version.