Disk on Microsoft Windows Cluster went offline and triggered failover of active node

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Disk on active node of Windows Server Failover Clustering (WSFC) went offline triggering failover.
  
  
• Though the behaviour shows disk as offline, in real the network adapter of the Active WSFC node is down.
  
  
• Following observed in Event log in the OS (Windows) of Active node:
  
  "Cluster network interface '<VM-name>- Heartbeat' for cluster node '<VM-name>' on network 'Heartbeat' failed. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges." 

Environment

vSphere ESXi 8.x

Cause

• There was a silent installation performed of VMware Tools on these node of WSFC
  
  
• As a part of installation of VMware tools, it was found vmxnet driver has latest driver available to install.
  During the process of installation, the existing tools was un-installed which disconnected the network card (vmxnet3) and it got connected post the installation.
  
  
  
• vmware.log of the VM EFTGAS12 confirming VMware Tools installation:
  
  2025-06-26T16:31:12.011Z In(05) vcpu-14 - Guest: Command line: /S /v "/qn REBOOT=R"
2025-06-26T16:31:12.011Z In(05) vcpu-14 - Guest: Installing VMware Tools 12.5.2.52591 (build-24697584)
  
  
• C:\Windows\Temp\vminst.log -- shows request for reboot to update the vmxnet3 driver
  
  2025-06-27T00:32:29.843+08:00| tools-build-24697584| I1: PnpDriverManager::InstallDriver: Determining whether to install the vmxnet3 driver
2025-06-27T00:32:29.843+08:00| tools-build-24697584| I2: DInfo_GetDriverInfoFromInf: Got catalog file for inf: vmxnet3.cat
2025-06-27T00:32:30.499+08:00| tools-build-24697584| I1: GetDriverForDeviceCallback: original inf is "vmxnet3.inf" with cat "vmxnet3.cat"
2025-06-27T00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3ndis6
2025-06-27T00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3
2025-06-27T00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3
2025-06-27T00:32:38.171+08:00| tools-build-24697584| W1: PnpDriverManager::InstallPnPDevices: Reboot required to update vmxnet3 driver for device PCI\VEN_15AD&DEV_07B0&SUBSYS_07B015AD
2025-06-27T00:32:38.171+08:00| tools-build-24697584| I2: BaseDriverManager::WriteStartType: WriteStartType of vmxnet3 of 4->4 from 4 skipped
2025-06-27T00:32:38.171+08:00| tools-build-24697584| W1: Util_SetRebootW: Reboot requested by 'vmxnet3'

Resolution

• It is an expected behaviour wherein VMware Tools installation updates vmxnet3 driver and this requires a reboot.
  
  
• Triggering the reboot results in failover of the WSFC node.