vDS sync issues and NSX configuration failures following network outage on ESXi hosts
Article ID: 448782
Updated On:
Products
Issue/Introduction
Following a network outage or transient physical link failure, administrators may observe:
- ESXi hosts showing as "Out of Sync" for the Virtual Distributed Switch (vDS) in vCenter.
- NSX-T/NSX Manager reporting a "Failed" configuration status for transport nodes in the cluster.
- Virtual Machines (VMs) on the affected hosts become unreachable.
- SSH access to the host reveals network interfaces (vmnics) are up but not passing traffic.
- Log entries in
/var/run/log/vmkernel.logorvmkwarning.logsimilar to:WARNING: netschedHClk: NetSchedHClkWatchdogSysWorld: vmnicX : scheduler lock up [stopped=1]WARNING: Uplink: Queue 0 of device vmnicX stuck, resetting the device
Environment
- VMware NSX
- VMware vCenter Server
- VMware vSphere ESXi
Cause
This issue occurs when a physical network outage causes the NIC firmware to enter an unresponsive state. While the physical link may appear "Up" after the outage, the firmware level I/O locks and TX hangs prevent the ESXi network scheduler from synchronizing configuration data between the host and management planes (vCenter/NSX).
Resolution
To recover from the unresponsive firmware state and restore synchronization, perform a rolling reboot of the affected ESXi cluster:
- For each host in the cluster, migrate if possible or power off resident Virtual Machines.
- Place the host in Maintenance Mode.
- Reboot the ESXi host to fully reset the physical NIC hardware and firmware state.
- Exit Maintenance Mode and proceed to the next host in the cluster.
Note: If the issue persists after a full host reboot, engage the hardware vendor to perform a diagnostic stress test on the physical NICs. Also, ensure firmware/drivers are aligned with the VMware Compatibility Guide.
Additional Information
Feedback
