Transient ~90 Second Outage Across All Traffic Types Following esxcli nic down on LACP Uplink

Products

VMware vSphere ESXi VMware vCenter Server VMware vSAN VMware NSX

Issue/Introduction

When an administrator executes esxcli network nic down -n <vmnic#> from a terminal session, SSH, or automated script to facilitate maintenance or failover testing, virtual machines (VMs), VMkernel adapters (vmknics), and storage transport endpoints using a Link Aggregation Group (LAG) on a vSphere Distributed Switch (vDS) experience transient network degradation or complete isolation for approximately 90 seconds before self-healing.

Observed symptoms and service events include:

Full or partial network connectivity loss across the vDS, lasting approximately 90 seconds.
Heartbeat loss events were recorded across one or more of the following logs: vSphere HA (/var/log/fdm.log), vSAN (/var/log/clomd.log and /var/log/vsansystem.log), and VMFS storage (/var/log/vobd.log).
The ESXi host briefly transitions to a "Not Responding" state in vCenter Server if the management VMkernel interface is mapped to the impacted vDS.
Terminal command esxcfg-nics -l reports the operational link status of the disabled physical NIC as down.
Performing an administrative switch port shutdown (shutdown) on the physical switch does not impact the network and results in immediate failover.

Log Entries in /var/log/ confirm the sequence of events and the subsequent 90-second self-healing behavior:

shell.log: Indicates that an administrator manually disabled and subsequently re-enabled the physical network adapter vmnic# using esxcli commands.

YYYY-MM-DDT16:03:33.213Z In(14) shell[12113029]: [root]: esxcli network nic down -n <vmnic#>
.
YYYY-MM-DDT16:07:24.836Z In(14) shell[12113029]: [root]: esxcli network nic up -n <vmnic#>

hostd.log: Records the local administrator toggling the physical adapter vmnic# offline and then back online via esxcli.

YYYY-MM-DDT16:03:33.446Z In(166) Hostd[2100770]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-##-##7e sid=52fc370a] Event 21349 : User [email protected] logged in as pyvmomi #.#.#.#.# internal Python/3.11.12 (VMkernel; #.#.#; x86_64)
YYYY-MM-DDT16:03:33.516Z In(166) Hostd[2100795]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-##-##86 sid=52fc370a user=root] Dispatch network.nic.down
YYYY-MM-DDT16:03:34.004Z In(166) Hostd[2100786]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 21350 : Physical NIC vmnic0 linkstate is down.
.
YYYY-MM-DDT16:07:25.130Z In(166) Hostd[2100787]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-45-c6f8 sid=5286e5bf user=root] Dispatch network.nic.up
YYYY-MM-DDT16:07:25.288Z In(166) Hostd[2100787]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-45-c6f8 sid=5286e5bf user=root] Dispatch network.nic.up done
YYYY-MM-DDT16:07:26.002Z In(166) Hostd[2100769]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 21403 : Physical NIC vmnic0 linkstate is up.

vobd.log: Demonstrates that the physical NIC link failure directly triggered VMFS storage heartbeat timeouts, which persisted during the LACP timeout window before recovering.

YYYY-MM-DDT16:03:33.535Z In(14) vobd[2097812]:  [netCorrelator] 2820231871469us: [vob.net.vmnic.linkstate.down] vmnic <vmnic#> linkstate down
YYYY-MM-DDT16:03:34.000Z In(14) vobd[2097812]:  [netCorrelator] 2820366816395us: [esx.problem.net.vmnic.linkstate.down] Physical NIC <vmnic#> linkstate is down
YYYY-MM-DDT16:03:34.000Z In(14) vobd[2097812]:  The event ([esx.problem.net.vmnic.linkstate.down] Physical NIC <vmnic#> linkstate is down) was sent immediately to hostd;
YYYY-MM-DDT16:03:45.491Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820243827144us: [vob.vmfs.heartbeat.timedout] 68ff190d-########-####-b8599fcbc800 0d19ff68-####-####-####-b8599fcbc800
YYYY-MM-DDT16:03:45.491Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820378307413us: [esx.problem.vmfs.heartbeat.timedout] 68ff190d-########-####-b8599fcbc800 0d19ff68-####-####-####-b8599fcbc800
.
YYYY-MM-DDT16:05:17.710Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820336041342us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume 68ce43c5-########-####-b8599fcbc800

lacp.log: Shows the disconnected physical NIC successfully detaching from the Link Aggregation Group (LAG), leaving one active port and resulting in a loss of redundancy.

YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 3461, Receive uplink vmnic# event 12, smVars 0x1
YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 2002, Detach uplink vmnic# from aggregator 1, numPorts 1, agg b8:##:##:##:##:49
YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 1398,  1 active port of group ID ####30259

fdm.log: Reveals that HA communication was lost (triggering an election state change), but successfully recovered and stabilized once the 90-second LACP timer expired and traffic failed over to the healthy link.

YYYY-MM-DDT16:03:43.431Z In(166) Fdm: Election State Change Slave => Startup : Lost master
.
YYYY-MM-DDT16:05:03.548Z In(166) Fdm: Startup: Got AmMaster with goodness value 2

clomd.log: Highlights a loss of vSAN cluster membership that persisted during the traffic blackhole period, with nodes successfully re-added to the database immediately after the 90-second LACP convergence timer expired.

YYYY-MM-DDT16:03:47.597Z No(29) clomd: clomdb-CdbHandleRemoveEntry: Removing node from CLOMDB
.
YYYY-MM-DDT16:05:08.836Z No(29) clomd: clomdb-CdbAddTableEntry: Added node back to CLOMDB

vsansystem.log: Details the host becoming isolated from the vSAN cluster (nodeCount: 1) during the outage, before successfully self-healing to full membership (nodeCount: <total_nodes>) once the 90-second LACP timeout forced the upstream switch to stop sending traffic to the dead link.

YYYY-MM-DDT15:51:32.156Z In(166) vsansystem[2100151]: [vSAN@6876 sub=VsanSystemProvider opId=33b13639-d248] Complete, cluster: 52e47e51-####-####-####-d44f4781e837, nodeCount: <total_nodes>, status: (vim.vsan.host.ClusterStatus) { 
YYYY-MM-DDT16:03:47.660Z In(166) vsansystem[2100167]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-d4e7] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) { 
. 
YYYY-MM-DDT16:05:08.812Z In(166) vsansystem[2100081]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSNodeUpdate-d51a] Complete, nodeCount: <total_nodes>, runtime info: (vim.vsan.host.VsanRuntimeInfo) {

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

Executing esxcli network nic down performs a software-level driver teardown that halts VMkernel frame processing while leaving physical NIC hardware powered on and emitting carrier signals to the physical switch.

The upstream switch senses active physical link light and continues hashing ~50% of southbound ingress traffic to the non-responsive link until 3 consecutive LACPDU heartbeats time out under default LACP Slow Timers (3 x 30s = 90s). Once the timers expire, the switch removes the dead link from the LAG bundle, and the environment self-heals.

Resolution

Permanent Architecture Fix: Configure LACP to FAST (1-second transmission interval, 3-second hold-down timeout) symmetrically on both the physical switch ports and the vSphere Distributed Switch LAG profile to reduce convergence time to maximum of ~3 seconds.
Maintenance Workaround: Avoid using esxcli network nic down to test path failover or perform maintenance when LACP Slow Timers are configured. Instead, perform administrative shutdown commands directly on the upstream physical switch ports to terminate optical/electrical carrier signals and trigger instantaneous ESXi link-state failover.

For further assistance Contact Broadcom support.

Additional Information

vSAN VMs Restart Unexpectedly During Single NIC Failure.

ESXi host not responding.

HA failover over of VMs on an ESXi host.

VMs restart due to vSphere HA.

Total Network Outage After Administrative Uplink Shutdown Due to Switch LACP Standalone Configuration.

LACP Timer Failover & Asymmetric State Matrix When configuring LACP timers, it is critical that both the physical switch and the vSphere Distributed Switch (vDS) match. The table below illustrates the expected failover times and the specific network impacts of symmetric versus asymmetric timer configurations during an un-signaled link failure:

Scenario	Switch Side	vDS Side	Failover Time	Asymmetric Gap	Network Operational Behavior
1	SLOW (90s)	SLOW (90s)	90 Seconds	None (Symmetric)	Complete Outage (Symmetric): Traffic using the affected link is completely blackholed in both directions (Northbound and Southbound) for 90 seconds until both sides time out LACPDUs.
2	SLOW (90s)	FAST (3s)	90s (Rx-In) / 3s (Tx-Out)	87 Seconds Gap	Partial Degradation (Asymmetric): Northbound traffic (ESXi → Switch) fails over in 3s. Southbound traffic (Switch → ESXi) continues flooding the dead link for 90s, causing 87s of inbound packet loss.
3	FAST (3s)	SLOW (90s)	90s (Tx-Out) / 3s (Rx-In)	87 Seconds Gap	Partial Degradation (Asymmetric): Southbound traffic (Switch → ESXi) fails over in 3s. Northbound traffic (ESXi → Switch) continues sending to the dead link for 90s, causing 87s of outbound packet loss.
4	FAST (3s)	FAST (3s)	3 Seconds	None (Symmetric)	Healthy / Optimal (Symmetric): Symmetrical 3-second convergence in both directions without asymmetric traffic blackholing.