Understanding the SNMP Layer 2 networking in the VMware ESXi
search cancel

Understanding the SNMP Layer 2 networking in the VMware ESXi

book

Article ID: 316380

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

This article provides information on obtaining information about Layer 2 network configuration between the VMware ESXi physical network interface card NICs and the virtual NICs of a given virtual machine and how to obtain port status and statistics.

The MIB modules are:

  • IETF IF-MIB 2863
  • IEEE8021-Q-BRIDGE-MIB Revision 200810150000Z
VMware SNMP agent follows the IETF and IEEE standards for reporting and does not provide additional documentation other than the MIB modules. For more information, see Determining the MIB module listing, name, and type of an SNMP OID.

Resolution

The MIB modules are:

  1. IETF IF-MIB 2863
  2. IEEE8021-Q-BRIDGE-MIB Revision 200810150000Z

In the IfTable/ifXTable table, the SNMP reports for ESXi physical NICs which are vSwitch uplinks. The ESXi agent also report vSwitch aggregate statistics and virtual interfaces used by the ESXi host. A stock ESXi host configuration on a system with one physical NIC report:


ifIndex 1, ifType.1 6(ethernet), ifDecr.1 "vmnic0 at 0:0:.."
ifIndex 2, ifType.2 53(propVirtual), ifDescr.2 "vswitch vSwitch0 type: traditional"
ifIndex 3, ifType.3 258(vmwareVirtualNic), ifDescr.3 "virtual ethernet interface vmk0 at ..."

For more information on ifTypes table, see https://www.iana.org/assignments/ianaiftype-mib/ianaiftype-mib.


ifStackTable provides how these three ifTable entries are related:

IF-MIB::ifStackStatus.1.0 = INTEGER: active(1)
IF-MIB::ifStackStatus.2.1 = INTEGER: active(1)
IF-MIB::ifStackStatus.3.2 = INTEGER: active(1)
IF-MIB::ifStackStatus.0.3 = INTEGER: active(1)

Note: 0 represents nothing below or above this item. For example, the entry .1.0 means there is nothing below the physical ethernet and .2.1 means the physical ethernet is connected to a virtual switch. Also, if uplinks are joined into link aggregation, that is reported in the ifTable/ifXTable table according to the IEEE/IETF standards.
 
For virtual machines whose virtual NICs are connected to virtual switch ports, access an instance of the bridge MIB to obtain l2 stats (in/out discard frames). The IEEE standard reports per port, and per VLAN.

Bridge ports (IEEE) identifiers are not the same as ifIndex (IETF). The bridge MIB, for any given port, provides a mapping from the bridge port back to ifIndex if it exists else its reported as 0.

For example, to map from vswitch 1 port 3:

ifIndex 1displays the uplink. The first index is the identifier of the vSwitch, unlike the older IETF BRIDGE-MIB (RFC 1493), IEEE8021-Q-BRIDGE-MIB supports multiple virtual bridges per SNMP agent. Furthermore, the ENTITY-MIB entLogicalTable provides entries to identify distributed virtual switch instances across multiple ESXi SNMP agents,which are distributed.

For example, the DVSID: is the same across ESXi systems:

ENTITY-MIB::entLogicalDescr.1 = STRING: ieee8021BridgeBaseComponentId: 3, DVSID: 64 76 73 77 69 74 63 68-00 00 00 00 00 00 00 00
ENTITY-MIB::entLogicalType.1 = OID: IEEE8021-BRIDGE-MIB::ieee8021BridgeMib
ENTITY-MIB::entLogicalTAddress.1 = ""
ENTITY-MIB::entLogicalTDomain.1 = OID: TRANSPORT-ADDRESS-MIB::transportDomainUdpIpv4

Start with the physical ESXi NICs which may be connected to virtual switches. If they are not, ifStackTable reports this:


snmpwalk -mall -v3 -u joe -l noAuthNoPriv 192.0.2.1 ieee8021BridgeBasePortIfIndex

IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex.1.3 = INTEGER: 1
IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex.1.5 = INTEGER: 7
IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex.1.11 = INTEGER: 0
IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex.2.3 = INTEGER: 2

If vSwitch 1 vswitch port 11 (ieee8021BridgeBasePortIfIndex.1.11) does not have an entry ifTable, there is probability that it is connected to a virtual machine.

For example, to report per virtual machine NIC stats from VMware ESXi configured with either traditional virtual switch or distributed virtual switch:
  1. Get virtual machine MAC address.

    VMWARE-VMINFO-MIB::vmwVmMAC[2][14] = STRING: 0:c:29:38:90:29
    VMWARE-VMINFO-MIB::vmwVmDisplayName[2] = STRING: cloudvm build 2467266
     
  2. Find the port this MAC address was learned on by making queries of each vSwitch's forwarding table (ieee8021QBridgeTpFdbPort) until its found.

    IEEE8021-Q-BRIDGE-MIB::ieee8021QBridgeTpFdbPort[1][1][STRING: 0:c:29:38:90:29] = Gauge32: 13
     
Get the counters sent per VLAN for Switch 1 Port “13” [1][13] and sum them by counter type to produce a per-port metric.

Note: 4096 means no VLAN assigned per IEEE8021-TC-MIB
 
IEEE8021-Q-BRIDGE-MIB::ieee8021QBridgeTpVlanPortInFrames[1][13][4096] = Counter64: 37801 framesIEEE8021-Q-BRIDGE-MIB::ieee8021QBridgeTpVlanPortOutFrames[1][13][4096] = Counter64: 5368872 frames IEEE8021-Q-BRIDGE-MIB::ieee8021QBridgeTpVlanPortInDiscards[1][13][4096] = Counter64: 4 frames

Otherwise, just the port level statistics, no need to aggregate VLANs seen:

IEEE8021-BRIDGE-MIB::ieee8021BridgeTpPortInFrames[1][13] = Counter64: 37801 frames
IEEE8021-BRIDGE-MIB::ieee8021BridgeTpPortOutFrames[1][13] = Counter64: 5368872 frames
IEEE8021-BRIDGE-MIB::ieee8021BridgeTpPortInDiscards[1][13] = Counter64: 4 frames

To get the port identifier:

IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortName[1][13] = STRING: 33554444

To get the MTU size this port accepts:

IEEE8021-BRIDGE-MIB::ieee8021BridgeTpPortMaxInfo[1][13] = INTEGER: 1500 bytes

IETF IF-MIB does not contain entries for virtual machine ports, only vmk(ifType 258) and physical NICs (ifType 6). Must use this IEEE Bridge mib to obtain statistics from virtual machine virtual NICs from the virtual switch ports:

IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex[1][3] = INTEGER: 1
IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex[1][5] = INTEGER: 6
IEEE8021-BRIDGE-MIB::ieee8021BridgeBasePortIfIndex[1][13] = INTEGER: 0
where ifIndex 3 is the uplink
IF-MIB::ifDescr[3] = STRING: Device vmnic2 at 03:00.2 tg3
IF-MIB::ifType[3] = INTEGER: ethernetCsmacd(6)
and ifIndex 6 is the vmk0
IF-MIB::ifDescr[6] = STRING: Virtual interface: vmk0 on vswitch vSwitch0 portgroup: Management Network
IF-MIB::ifType[6] = INTEGER: vmwareVirtualNic(258)

Q: So why all this complexity? Why not provide enterprise objects in the agent for ESXi that fetched these counters from a VMware enterprise MIB?

The preceeding mechanism is how it has been done for the past two decades in the physical world, operators query switches to find out what is connected to a given switch port. If there is no LLDP/CDP, then they look to the switch to find what MAC addresses are learned per port/per VLAN and then search their address database to match the MAC address or go to arp/nd tables on networking gear to match the layer 3 IP address.



Additional Information

For more information on standard documents that VMware ESXi SNMP follows, see:

Powered by Wolken Software