ESXi host face maintenance mode issue and virtual machines have file lock and network issues

Products

• VMware vSphere ESXi
• VMware vCenter Server

Issue/Introduction

From vCenter Server, below issues are reported:

• ESXi host does not enter maintenance mode and stuck at 75% for more than a day.
• The virtual machines on the esxi host face network issue or vm file lock issues.
• The virtual machines also face consolidation issues.

From vmkernel.log, below errors are seen:

vmkernel.log:
YYYY-MM-DD:HH:MM.SS info hostd[#######] [Originator@#### sub=Hostsvc.ModeMgr opID=#####-##-###### user=vpxuser:domain-name.username] Host mode transition blocked by: VSAN op started by task 'haTask-ha-host-vim.HostSystem.enterMaintenanceMode'
YYYY-MM-DD:HH:MM.SS info hostd[#######] [Originator@#### sub=Hostsvc.ModeMgr opID=#####-##-###### user=vpxuser:domain-name.username] Host mode transition blocked by: NFC server session '################'
YYYY-MM-DD:HH:MM.SS info hostd[#######] [Originator@#### sub=Hostsvc.ModeMgr opID=#####-##-####### user=vpxuser:domain-name.username] Host mode transition blocked by: NFC server session '################'

 

Below command shows multiple read-only locked ctk files in vm folder:
 # for file in *; do echo ${file}; vmfsfilelockinfo -p ${file} |grep -i mode; done

"vm-name-ctk.vmdk" is locked in Read-Only mode by host having mac address ['##:##:##:##:##:##']
Host owning the lock on file is host-name, lockMode : Read-Only
"vm-name_1-ctk.vmdk" is locked in Read-Only mode by host having mac address ['##:##:##:##:##:##']
Host owning the lock on file is host-name, lockMode : Read-Only
"vmware.log" is locked in Exclusive mode by host having mac address ['##:##:##:##:##:##']
Host owning the lock on file is host-name, lockMode : Exclusive

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

VMware ESXi Host 7.x

VMware ESXi Host 8.x

Cause

The third party tool is holding NFC lock on the host which is causing disk related task failures:

[root@host-name]  esxcli network ip connection list | grep 902
tcp         0       0  host-ip:port                  <third-party-tool-ip>:port  ESTABLISHED    newreno  busybox
tcp         0       0  host-ip:port                  <third-party-tool-ip>:port  ESTABLISHED    newreno  busybox
tcp         0       0  [::]:902                           [::]:0                LISTEN        newreno  busybox
tcp         0       0  0.0.0.0:902                        0.0.0.0:0             LISTEN        newreno  busybox

Resolution

1. Note down the "third-party-tool-ip" from the above command output.

2. Engage or request customer to engage the respective third party vendor to release the NFC lock on port 902.

3. Once locks are released, from vCenter you notice that all the hung tasks on ESXi host and virtual machines will resume back and completes successfully.

4. The virtual machine consolidation also completes without any issues.