Interface Down event when the interface on the device is actually up

Products

VMware Smart Assurance VMware Telco Cloud Service Assurance

Issue/Introduction

Symptoms:

Smarts IP domain is reporting an Interface Down event when the interface on the device is actually up
False Interface Down alert in Smarts IP domain.
Interface Status in the Containment view is reported as NotPresent

Environment

SMARTS - 10.1.x

SMARTS - 2.x

TCSA -2.x

Cause

This behavior may have other causes, but is typically seen when the same IP address is configured on multiple devices where only one of the devices is in the Smarts IP topology.

Cause details:

A device may have multiple IP addresses (up to 10) on the SNMPAgentAddress list. If the IP address specified in the SNMPAgent attribute of the device stops responding to SNMP requests, the Smarts IP domain goes down the list specified in the SNMPAgentAddressList attribute until it finds an IP address that responds to SNMP polls. If the IP domain starts polling an IP address that is also configured on another device that is not in the Smarts IP topology, and that device starts replying to the SNMP polls, false alerts can be generated.

Resolution

To determine if this is the issue, you need to enable tracing and run an sm_snmpwalk as described in the following step-by-step instructions. Enabling tracing for the device will allow you to find out which IP address is used to poll the device, while the sm_snmpwalk will allow you to determine if more than one device is configured with the same IP address. If you find that the same IP address is configured on more than one device, you will need to reconfigure the devices so that each IP is configured only on one device.

Step-by-step instructions

To enable tracing, enter the following from the Smarts command line:

cd <BASEDIR>/smarts/bin/
Run the trace from the /smarts/bin/ directory as follows (all on one line):

./dmctl -s <IP_DM_NAME> invoke ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller setTrace ICIM_UnitaryComputerSystem::<device_name> TRUE
The trace will generate messages similar to the following in the Smarts IP domain log file. The IP address used to poll the device is specified in the "Sent 178 bytes to 192.168.15.18 port 161" of the following message.

The message below is the actual SNMP packet trace to the device.

SNMP Message:
   38:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.7.1"
   50:          INTEGER-32 (0x02), 1 byte == 1
   53:        VarBind -> SEQUENCE (0x30), 15 bytes:
   55:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.8.1"
   67:          INTEGER-32 (0x02), 1 byte == 1
   70:        VarBind -> SEQUENCE (0x30), 15 bytes:
   72:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.7.2"
   84:          INTEGER-32 (0x02), 1 byte == 1
   87:        VarBind -> SEQUENCE (0x30), 15 bytes:
   89:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.8.2"

Sent 178 bytes to 192.168.15.18, port 161.

The response of the Agent follows

Received 187 bytes from 192.168.15.18, port 161.
   38:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.7.1"
   50:          INTEGER-32 (0x02), 1 byte == 1
   53:        VarBind -> SEQUENCE (0x30), 15 bytes:
   55:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.8.1"
   67:          INTEGER-32 (0x02), 1 byte == 1
   70:        VarBind -> SEQUENCE (0x30), 15 bytes:
   72:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.7.2"
   84:          Exception (IMPLICIT NULL) (0x81), 0 bytes == <noSuchInstance>

   87:        VarBind -> SEQUENCE (0x30), 15 bytes:
   89:          OBJ-ID (0x06), 10 bytes == ".1.3.6.1.2.1.2.2.1.8.2"
101:          Exception (IMPLICIT NULL) (0x81), 0 bytes == <noSuchInstance>
104:        VarBind -> SEQUENCE (0x30), 15 bytes:

In the preceding output we see that the agent is replying with <noSuchInstance> to the requests for the index of this interface. This means that Agent is not aware of this interface being present on the device.
Disable the trace as follows from the command line (all on one line):

$./dmctl -s <IP_DM_NAME> invoke ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller setTrace ICIM_UnitaryComputerSystem::<device_name> FALSE
Run an sm_snmpwalk against the IP address specified in the AgentAddress Attribute and against the IP address specified in the trace. If the responses received are from two different devices, you must reconfigure the devices so that each IP address is configured only on one device.