Total Network Outage After Administrative Uplink Shutdown Due to Switch LACP Standalone Configuration
search cancel

Total Network Outage After Administrative Uplink Shutdown Due to Switch LACP Standalone Configuration

book

Article ID: 451155

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi VMware NSX VMware vSAN

Issue/Introduction

In VMware vSphere ESXi environments utilizing Link Aggregation Control Protocol (LACP) Link Aggregation Groups (LAGs) connected to multi-chassis vPC switch fabrics, executing an administrative link-down command (esxcli network nic down -n <vmnic#>) causes complete, unrecovered network isolation for the host across all traffic types until the interface is manually re-enabled using esxcli network nic up -n <vmnic#>.

Unlike standard LACP timer convergence issues where traffic recovers within 3 to 90 seconds, this network outage persists indefinitely until manual administrative intervention is performed.

Observed symptoms and service events include:

  • Complete and continuous loss of network connectivity occurs across all Virtual Machine and VMkernel portgroups on the vSphere Distributed Switch (vDS).

  • Heartbeat loss events were recorded across one or more of the following logs: vSphere HA (/var/log/fdm.log), vSAN (/var/log/clomd.log and /var/log/vsansystem.log), and VMFS storage (/var/log/vobd.log).

  • The ESXi host transitions to a "Not Responding" state in vCenter Server if the management VMkernel interface is mapped to the impacted vDS.

  • Terminal command esxcfg-nics -l reports the operational link status of the disabled physical NIC as down.

  • Review of the /var/log/lacp.log indicates a successful failover, resulting only in a loss of network redundancy.

  • Executing an administrative switch port shutdown (shutdown) directly on the physical switch port does not impact the network and results in an immediate failover.

  • The network outage persists until the physical network adapter is manually brought online using the command: esxcli network nic up -n <vmnic#>

Log entries in /var/log/:

shell.log: Indicates that an administrator (root user) manually disabled and subsequently re-enabled the physical network adapter vmnic# using esxcli commands.

YYYY-MM-DDT16:03:33.213Z In(14) shell[12113029]: [root]: esxcli network nic down -n <vmnic#>
.
YYYY-MM-DDT16:07:24.836Z In(14) shell[12113029]: [root]: esxcli network nic up -n <vmnic#>

hostd.log: Records a local administrator (root) manually toggling the physical adapter vmnic# offline and then back online via esxcli network commands.

YYYY-MM-DDT16:03:33.446Z In(166) Hostd[2100770]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=esxcli-##-##7e sid=52fc370a] Event 21349 : User [email protected] logged in as pyvmomi #.#.#.#.# internal Python/3.11.12 (VMkernel; #.#.#; x86_64)
YYYY-MM-DDT16:03:33.516Z In(166) Hostd[2100795]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-##-##86 sid=52fc370a user=root] Dispatch network.nic.down
YYYY-MM-DDT16:03:34.004Z In(166) Hostd[2100786]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 21350 : Physical NIC vmnic0 linkstate is down.
.
YYYY-MM-DDT16:07:25.130Z In(166) Hostd[2100787]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-45-c6f8 sid=5286e5bf user=root] Dispatch network.nic.up
YYYY-MM-DDT16:07:25.288Z In(166) Hostd[2100787]: [Originator@6876 sub=Solo.VmwareCLI opID=esxcli-45-c6f8 sid=5286e5bf user=root] Dispatch network.nic.up done
YYYY-MM-DDT16:07:26.002Z In(166) Hostd[2100769]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 21403 : Physical NIC vmnic0 linkstate is up.

vobd.log: Demonstrates that a physical NIC link failure directly triggered VMFS storage heartbeat timeouts, which persisted until the network adapter was brought back online and switch connectivity was restored.

YYYY-MM-DDT16:03:33.535Z In(14) vobd[2097812]:  [netCorrelator] 2820231871469us: [vob.net.vmnic.linkstate.down] vmnic <vmnic#> linkstate down
YYYY-MM-DDT16:03:34.000Z In(14) vobd[2097812]:  [netCorrelator] 2820366816395us: [esx.problem.net.vmnic.linkstate.down] Physical NIC <vmnic#> linkstate is down
YYYY-MM-DDT16:03:34.000Z In(14) vobd[2097812]:  The event ([esx.problem.net.vmnic.linkstate.down] Physical NIC <vmnic#> linkstate is down) was sent immediately to hostd;
YYYY-MM-DDT16:03:45.491Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820243827144us: [vob.vmfs.heartbeat.timedout] 68ff190d-########-####-b8599fcbc800 0d19ff68-####-####-####-b8599fcbc800
YYYY-MM-DDT16:03:45.491Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820378307413us: [esx.problem.vmfs.heartbeat.timedout] 68ff190d-########-####-b8599fcbc800 0d19ff68-####-####-####-b8599fcbc800
.
YYYY-MM-DDT16:05:17.710Z In(14) vobd[2097812]:  [vmfsCorrelator] 2820336041342us: [vob.vmfs.heartbeat.recovered] Reclaimed heartbeat for volume 68ce43c5-########-####-b8599fcbc800
YYYY-MM-DDT16:07:25.269Z In(14) vobd[2097812]:  [netCorrelator] 2820463594545us: [vob.net.vmnic.linkstate.up] vmnic vmnic0 linkstate up
YYYY-MM-DDT16:07:26.002Z In(14) vobd[2097812]:  [netCorrelator] 2820598818062us: [esx.clear.net.connectivity.restored] Network connectivity restored on virtual switch

lacp.log: Shows the disconnected physical NIC successfully detaching from the Link Aggregation Group (LAG), leaving one active port and resulting in a loss of redundancy before re-joining the bundle when the link recovered.

YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 3461, Receive uplink vmnic# event 12, smVars 0x1
YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 2002, Detach uplink vmnic# from aggregator 1, numPorts 1, agg b8:##:##:##:##:49
YYYY-MM-DDT16:03:33Z No(29) lacp[2098271]: 1398,  1 active port of group ID ####30259
.
YYYY-MM-DDT16:07:25Z No(29) lacp[2098271]: 3530, Receive uplink vmnic# event (physical link up), smVars 0x3
YYYY-MM-DDT16:07:25Z No(29) lacp[2098271]: 1398,  1 active port of group ID ####30259
YYYY-MM-DDT16:07:25Z No(29) lacp[2098271]: 1294, The host status property already set to green for portset DvsPortset-0
YYYY-MM-DDT16:07:27Z No(29) lacp[2098271]: 2064, Add uplink vmnic# into aggregator 1 (Selected), numPorts 2, agg b8:##:##:##:##:49, priority 2000/1

fdm.log: Reveals that while HA communication briefly recovered after the standard LACP convergence timer expired (3 or 90 seconds, depending on configuration), the host dropped from the cluster again shortly after, remaining isolated until manual administrative intervention.

YYYY-MM-DDT16:03:43.431Z In(166) Fdm[2103094]: [Originator@6876 sub=Election opID=clusterElection.cpp:1570-####e18e] Election State Change Slave => Startup : Lost master
.
YYYY-MM-DDT16:05:03.548Z In(166) Fdm[2103094]: [Originator@6876 sub=Election opID=clusterElection.cpp:1570--####e18e] Startup: Got AmMaster with goodness value 2
.
YYYY-MM-DDT16:05:23.565Z In(166) Fdm[2103094]: [Originator@6876 sub=Election opID=clusterElection.cpp:1570-####e18e] Election State Change Slave => Startup : Lost master
.
YYYY-MM-DDT16:07:29.762Z In(166) Fdm[2103094]: [Originator@6876 sub=Election opID=clusterElection.cpp:1570--####e18e] Startup: Got AmMaster with goodness value 2

clomd.log: Highlights a temporary recovery of vSAN cluster membership following the LACP convergence timer, after which the host dropped from the cluster once more and remained isolated pending manual intervention.

YYYY-MM-DDT16:03:47.597Z No(29) clomd[2098943]: [Originator@6876] clomdb-CdbHandleRemoveEntry: Removing 00000000-0000-0000-0000-b8########00 of type CdbObjectNode from CLOMDB.
.
YYYY-MM-DDT16:05:08.836Z No(29) clomd[2098943]: [Originator@6876] clomdb-CdbAddTableEntry: Added 00000000-0000-0000-0000-b8599fcbc800 of type CdbObjectNode to CLOMDB, FD:00000000-0000-0000-0000-b8599fcbc800.
.
YYYY-MM-DDT16:05:27.602Z No(29) clomd[2098943]: [Originator@6876] clomdb-CdbHandleRemoveEntry: Removing 00000000-0000-0000-0000-b8########00 of type CdbObjectNode from CLOMDB.
.
YYYY-MM-DDT16:07:35.854Z No(29) clomd[2098943]: [Originator@6876] clomdb-CdbAddTableEntry: Added 00000000-0000-0000-0000-b8########00 of type CdbObjectNode to CLOMDB, FD:00000000-0000-0000-0000-b8########00.

vsansystem.log: Details the host becoming isolated from the vSAN cluster (nodeCount: 1), briefly recovering to full membership (nodeCount: <total_nodes>) via the LACP convergence timer, and subsequently dropping from the cluster again until manual intervention was applied.

YYYY-MM-DDT15:51:32.156Z In(166) vsansystem[2100151]: [vSAN@6876 sub=VsanSystemProvider opId=33b13639-d248] Complete, cluster: 52e47e51-####-####-####-d44f4781e837, nodeCount: 6, status: (vim.vsan.host.ClusterStatus) {
YYYY-MM-DDT16:03:47.660Z In(166) vsansystem[2100167]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-d4e7] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
.
YYYY-MM-DDT16:05:08.812Z In(166) vsansystem[2100081]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSNodeUpdate-d51a] Complete, nodeCount: <total_nodes>, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
YYYY-MM-DDT16:07:46.458Z In(166) vsansystem[2100168]: [vSAN@6876 sub=VsanSystemProvider opId=refresh-d640] Complete, nodeCount: <total_nodes>, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
YYYY-MM-DDT16:07:51.574Z In(166) vsansystem[2100167]: [vSAN@6876 sub=VsanSystemProvider opId=65645e89-d656] Complete, cluster: 52e47e51-####-####-####-d44f4781e837, nodeCount: <total_nodes>, status: (vim.vsan.host.ClusterStatus) {
.
YYYY-MM-DDT16:05:27.635Z In(166) vsansystem[2100078]: [vSAN@6876 sub=VsanSystemProvider opId=CMMDSMembershipUpdate-d584] Complete, nodeCount: 1, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
.
YYYY-MM-DDT16:07:51.574Z In(166) vsansystem[2100167]: [vSAN@6876 sub=VsanSystemProvider opId=65645e89-d656] Complete, cluster: 52e47e51-####-####-####-d44f4781e837, nodeCount: <total_nodes>, status: (vim.vsan.host.ClusterStatus) {

Environment

VMware vCenter Server.

VMware vSphere ESXi.

Cause

This issue occurs because the upstream physical switch port-channel is configured with no lacp suspend-individual (or a vendor-equivalent fallback setting). This feature is typically used to prevent LACP from suspending ports that stop receiving LACP packets, allowing bare-metal servers to complete PXE or DHCP network bootstrapping before the operating system fully loads and starts sending LACP data.

However, when this configuration is applied to active ESXi LACP uplinks, it causes a severe state mismatch. If an LACP PDU timeout occurs while the physical NIC transceiver remains powered on, the switch port inappropriately falls back into a standalone forwarding state. This mismatch induces hardware frame reflections and Layer 2 loop dynamics, completely isolating the ESXi host from the network until the interface is manually toggled.

Resolution

To permanently resolve this issue, the physical switch must be configured to suspend the port when LACP communication fails, preventing the standalone forwarding mismatch.

  1. Engage your internal network team or physical switch vendor to disable the standalone fallback settings (e.g., no lacp suspend-individual) on all upstream port-channel interfaces connected to ESXi LACP LAG uplinks.

  2. Verify that the physical switch ports automatically transition to a suspended state when LACP negotiation fails, or when an ESXi network adapter is administratively disabled.

Note: The following vendor-specific commands are provided as general reference examples. Because command syntax and default behaviors can vary across different hardware platforms and OS versions, please consult your switch vendor's official documentation to verify the exact commands required to enforce LACP port suspension in your environment.

Vendor / OSEnable Fallback (e.g., no lacp suspend-individual)Disable Fallback (Default Behavior)
Cisco NX-OSno lacp suspend-individuallacp suspend-individual
Arista EOSport-channel lacp fallback individualno port-channel lacp fallback
Juniper Junosset interfaces ae0 aggregated-ether-options lacp force-updelete interfaces ae0 aggregated-ether-options lacp force-up
Dell OS10port-channel lacp fallback individualno port-channel lacp fallback
Huawei VRPlacp pre-activation enableundo lacp pre-activation enable
NVIDIA Onyxlacp-individual enable forceno lacp-individual enable force


Workaround:  If your environment requires standalone fallback configurations to remain enabled (e.g., to support frequent bare-metal PXE provisioning, or during host migrations that involve temporarily moving uplinks to a Standard vSwitch), use the following maintenance workaround to prevent host isolation:

  • Avoid local ESXi commands: Do not use esxcli network nic down on the ESXi host to test failover or perform maintenance.

  • Shut down from the switch: Perform administrative shutdown commands directly on the upstream physical switch ports. This cleanly terminates the optical/electrical link signals and enforces an immediate, expected failover without triggering the LACP state mismatch.

For further assistance Contact Broadcom support.

Additional Information

vSAN VMs Restart Unexpectedly During Single NIC Failure.

Host network disconnected after LACP configuration.

Cisco Nexus LAG supportability with "no lacp suspend-individual".

Transient ~90 Second Outage Across All Traffic Types Following esxcli nic down on LACP Uplink.

Powered by Wolken Software