Disk on Microsoft Windows Cluster went offline and triggered failover of active node
search cancel

Disk on Microsoft Windows Cluster went offline and triggered failover of active node

book

Article ID: 403553

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • The disk on the active node of a Windows Server Failover Cluster (WSFC) unexpectedly goes offline, triggering a cluster failover. While the behavior presents as an offline disk, the underlying issue is that the network adapter of the active WSFC node has gone down.



  • The following error is observed in the Windows Event Log on the active node:

    "Cluster network interface '<VM-name>- Heartbeat' for cluster node '<VM-name>' on network 'Heartbeat' failed. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges."

Environment

VMware vSphere ESXi 8.x

Cause

An automated or silent upgrade of VMware Tools was executed on the WSFC nodes. This process unloaded the VMXNET3 network driver (upgrading it from version 1.9.19.0 to 1.9.20.0), causing a temporary loss of network connectivity. This interrupted WSFC network heartbeats and prompted the cluster to trigger an immediate node failover to maintain database availability.

  • The /VMFS/VOLUMES/<Datastore UUID>/<VM Name>/vmware.log file of the virtual machine confirms the silent VMware Tools installation and subsequent driver unload:

YYYY-MM-DDT08:49:08.169Z In(05) vcpu-0 - Guest: Command line: /S /v"/qn REBOOT=ReallySuppress"
YYYY-MM-DDT08:49:08.169Z In(05) vcpu-0 - Guest: Installing VMware Tools 13.0.10.0 (build-25056151)
YYYY-MM-DDT08:49:48.843Z In(05) vcpu-0 - VMXNET3 user: Quiesce device 0.
YYYY-MM-DDT08:49:48.925Z In(05) vcpu-0 - Guest: Driver=vmxnet3, Version=1.9.19.0 - unloaded
YYYY-MM-DDT08:49:48.984Z In(05) vcpu-0 - Guest: Driver=vmxnet3, Version=1.9.20.0
YYYY-MM-DDT08:50:07.082Z In(05) vmx -    Guest: toolbox: Version: 13.0.10.0 (build-25056151)

  • The guest OS log located at C:\Windows\Temp\vminst.log shows the registry deletion and reboot request required to complete the VMXNET3 driver update:

YYYY-MM-DDT00:32:29.843+08:00| tools-build-24697584| I1: PnpDriverManager::InstallDriver: Determining whether to install the vmxnet3 driver
YYYY-MM-DDT00:32:29.843+08:00| tools-build-24697584| I2: DInfo_GetDriverInfoFromInf: Got catalog file for inf: vmxnet3.cat
YYYY-MM-DDT00:32:30.499+08:00| tools-build-24697584| I1: GetDriverForDeviceCallback: original inf is "vmxnet3.inf" with cat "vmxnet3.cat"
YYYY-MM-DDT00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3ndis6
YYYY-MM-DDT00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3
YYYY-MM-DDT00:32:32.546+08:00| tools-build-24697584| I1: DriverBaseInfo::DeleteFromRegistry: Deleted info for driver vmxnet3
YYYY-MM-DDT00:32:38.171+08:00| tools-build-24697584| W1: PnpDriverManager::InstallPnPDevices: Reboot required to update vmxnet3 driver for device PCI\VEN_15AD&DEV_07B0&SUBSYS_07B015AD
YYYY-MM-DDT00:32:38.171+08:00| tools-build-24697584| I2: BaseDriverManager::WriteStartType: WriteStartType of vmxnet3 of 4->4 from 4 skipped
YYYY-MM-DDT00:32:38.171+08:00| tools-build-24697584| W1: Util_SetRebootW: Reboot requested by 'vmxnet3'

Resolution

This is an expected behavior. The VMware Tools installation updates the VMXNET3 driver, which temporarily unbinds the network adapter and requires a guest OS reboot to finalize. Triggering this sequence results in a drop in heartbeats and the failover of the WSFC node.

To prevent unplanned database failovers:

  1. Identify the patch management system or orchestration script that initiated the silent VMware Tools installation.

  2. Exclude all WSFC nodes from automated VMware Tools or VMXNET3 driver updates during active production hours.

  3. Treat VMware Tools upgrades as disruptive network maintenance events.

  4. Manually fail over the active database role to a secondary node prior to upgrading VMware Tools during a designated maintenance window.

Powered by Wolken Software