VMs connected to a vSwitch in ENS Polling Mode Fail to Start with a vCenter Power-On VM Task Status NIOC Message: Network I/O admission control (NIOC) check failed for Ethernet device(s)

Products

• VMware NSX

Issue/Introduction

• VMs intermittently fail to start and vCenter power-on VM task status messages similar to the below are observed:
  

  Network I/O admission control (NIOC) check failed for Ethernet device(s): 1(bitMap). 
  Check the host network configuration for available bandwidth for reservation. 
  Failed to start the virtual machine. 
  Module NIOC power on failed.

• VM host logs indicate that the VM port failed to connect to opaque network:
  
  /var/log/vmkernel.log
  

  <Timestamp> In(182) vmkernel: cpu107:41407651)Net: 2238: connected <Client Name>.eth0 eth0 to vDS, portID 0x6005dee
  <Timestamp> In(182) vmkernel: cpu107:41407651)Net: 3834: dissociate dvPort <dvPort ID> from port 0x6005dee
  <Timestamp> In(182) vmkernel: cpu107:41407651)Net: 3841: disconnected client from port 0x6005dee
  <Timestamp> In(182) vmkernel: cpu107:41407651)Net: 3410: Failed to connect port to opaque network: Success

  Error 'Net: 3410: Failed to connect port to opaque network: Success' occurs when Ens_ReserveENSPort in Net_ConnectOpaqueNetwork fails:

  <Timestamp> In(182) vmkernel: cpu107:41407651)Net: 3410: Failed to connect port to opaque network: Success

• Portset is in ENS poll mode, which can be confirmed by running the below CLI command from the VMs host:
  
  esxcfg-nics -e
  

  Name    Driver      ENS Capable      ENS Driven       INTR Capable     INTR Enabled     MAC Address             Description                   
  vmnic1  bnxtnet     True             True             True             False            <pNIC MAC Address>      <pNIC Description>
  vmnic2  bnxtnet     True             True             True             False            <pNIC MAC Address>      <pNIC Description

• The number of ports connected to the ENS poll mode vSwitch exceeds 108:
  • This can be confirmed by viewing the 'numPorts' column using the below CLI commands from the VM host:
    • Note that 'maxPorts' is 128 but 20 ports are reserved for non vnic/vmknic usage.
    • The below command should be run as close to the VM power-on failure as posisble for an accurate 'numPorts' count.
      
      [root@esx:~] nsxcli
      esx-04.corp.local> get ens switch list
      
      

      ENS switch list:
      name            swID   maxPorts   numActivePorts   numPorts   mtu   numLcores   lcoreIDs
      ------------------------------------------------------------------------------------------
      DvsPortset-0    0       128          109            109        9000        1           0
      ==========================================================================================
      

      
      

 

Environment

VMware NSX

VMware NSX-T Data Center

Cause

There are two potential causes for a VM power-on failure due to NIOC.  This KB is specific to 'Cause 1' below:

Cause 1
ENS-Polling mode does not support more than 108 VMs per vSwitch:

• Ens_ReserveENSPort can fail in ENS-Polling mode when the number of ports in a vswitch exceeds 108.

Cause 2

NIOC is not supported by smartNICs.

• See KB for further details:  VMs on SmartNIC Hosts Fail to Start with a vCenter power-on VM Task Status NIOC Message: Network I/O admission control (NIOC) check failed for Ethernet device(s)

Resolution

This is a known issue impacting VMware NSX.

 

Cause 1 Workaround

To workaround this limitation use either of the below options:

1. Reduce the number of ports connected to the ENS poll mode vSwitch below 108.

2.  Configure ENS Interrupt mode instead of ENS Polling mode for the relevant vSwitch.