Intermittent NIC link flaps observed in host with failed criteria:128, but none are seen in physical switch end

Products

• VMware vSphere ESXi

Issue/Introduction

• Below events are observed in host logs.
  
  
  • NSX configuration being applied to host. Lcore configuration is reapplied every time NSX manager pushes TN config to a host.

          /var/run/log/nsxaVim.log

         7154 2024-11-07T14:45:36Z In(14) nsxaVim: [2102859]: INFO Applying nsx config [(vim.dvs.HostDistributedVirtualSwitchManager.NsxDvsConfigSpec)

• • ENS Lcore configuration is applied.

         7263 2024-11-07T14:45:36Z In(14) nsxaVim: [2102859]: INFO ApplyHostSwitchCpuConfig, params [## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##|0~8,1~8]^@

• • VMKernel executed Lcore configuration.
    
     /var/run/log/vmkernel.log

         2024-11-07T14:45:36.217Z In(182) vmkernel: cpu21:2097791)ENS: 1735: Configuring Lcores for DvsPortset-3, 0-8,1-8

• • This triggers NetIOC reconfiguration.

         2024-11-07T14:45:36.217Z In(182) vmkernel: cpu21:2097791)ENS: 5196: vmnic0: Reconfigure HCLK

• • HCLK reconfiguration triggers ENS uplink detach and attach operations. These are not fast operations, which could take some seconds, leading to network downtime.
    
    2024-11-07T14:45:36.710Z In(182) vmkernel: cpu14:2097533)ENS: Ens_DoAttachPort:4141: portID ##########, port->fcPortID 5
    2024-11-07T14:45:36.710Z In(182) vmkernel: cpu14:2097533)ENS: Ens_DoAttachPort:4389: Attach port: 0x8a0000## handle : 0x4500c0027b80
    2024-11-07T14:45:36.925Z In(182) vmkernel: cpu54:2097533)ENS: Ens_NetQDisablePnicData:10757: Disable pNIC Data for port 5
    2024-11-07T14:45:37.244Z In(182) vmkernel: cpu30:2097533)ENS: Ens_DoDetachPort:5329: Detach port 0x8a0000## from ENS, handle: 0x4500c0027b80
    2024-11-07T14:45:37.244Z In(182) vmkernel: cpu30:2097533)ENS: Ens_DoAttachPort:4141: portID ##########, port->fcPortID 5
    
    
  • Link up/down events are observed.
    
     In(182) vmkernel: cpu54:2097581)netschedHClk: NetSchedHClkNotify:5067: vmnic5: link down notification
     In(182) vmkernel: cpu54:2097581)netschedHClk: NetSchedHClkNotify:5059: vmnic5: link up notification
    
    

     In(14) vobd[2097956]:  [netCorrelator] 2899101151876us: [vob.net.dvport.uplink.transition.down] Uplink: vmnic5 is down. Affected dvPort: ######-####-####-####-############/## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##. 0 uplinks up. Failed criteria: 128

     

Environment

VMware NSX 

VMware ESXi

Cause

ENS Lcore configuration is reapplied every time NSX manager pushes TN config / update messages

Resolution

Current workaround is to Disable NIOC on the Distributed Switch. Steps to disable NIOC is :

1. On the vSphere Client Home page, click Networking and navigate to the distributed switch. 
2. From the Actions menu, select Settings> Edit Settings.
3. From the Network I/O Control drop-down menu, select Disable. 
4. Click OK.

Refer Enable/Disable Network I/O Control on a vSphere Distributed Switch

The issue is resolved in vSphere/ESXi version 9.0.