Smarts is not polling interfaces properly. Many random interfaces are showing Admin or Oper status as NOTPRESENT, which is not the actual status on the MIB.
No changes have been done in device configuration or Smarts.
Smarts 10.X
1) Checked one interface as an example, it is instrumented for Fault by MIB2 in Smarts, with admin status UP and oper status NOTPRESENT.
-------------------------------------------
dmctl -s IPAM get Interface::IF-DSA/67149824::InstrumentedBy
{ Interface_Fault_MIB2::I-Interface_Fault_MIB2-IF-DSA/67149824 }
dmctl -s IPAM get Interface::IF-DSA/67149824::AdminStatus
UP
dmctl -s IPAM get Interface::IF-DSA/67149824::OperStatus
NOTPRESENT
-------------------------------------------
2) Checked the snmp output from device, both have value 1, which means they are UP:
-------------------------------------------
Admin Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.7.67149824
.1.3.6.1.2.1.2.2.1.7.67149824 = 1
Oper Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.8.67149824
.1.3.6.1.2.1.2.2.1.8.67149824 = 1
-------------------------------------------
3) Enabled trace on the device in Smarts
From the log, there is no evidence of polling details of Oper status of the interface, only Admin Status is captured.
-------------------------------------------
cat IPAM_en_US_UTF-8.log_trace | grep 67149824
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
-------------------------------------------
4) Captured debug data of SNMP Poller on the domain with below command:
dmctl -s <domain name> exec dmdebug --dumpAccessor=DEVSTAT-SNMP-Poller --output=accessor.log
From the log, there is an error on polling of Oper Status of the interface and the polling was last successful 1 day prior.
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifAdminStatus
Polling Period: 240
Last polled At: November 14, 2017 2:33:49 PM GMT+01:00
Cached Value: 1
Last Error: <OK>-E-<OK>-Normal successful completion
Unconstrained Polling Period: 15
Able To Poll: TRUE
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifOperStatus
Polling Period: 240
Last polled At: November 13, 2017 2:39:35 PM GMT+01:00
Cached Value:
Last Error: SNMP-E-ERR_NOSUCHNAME-No such name
Unconstrained Polling Period: 15
Able To Poll: FALSE
-------------------------------------------
5) The SNMP version in use for the device is V1:
-------------------------------------------
dmctl -s IPAM get SNMPAgent::SNMPAgent-DSA | grep SNMP
SNMPStatus = OK
SNMPVersion = V1
-------------------------------------------
For all the Non-V1 devices, Smarts never disables polling if it encounters any error. Which means for V2c or V3 discovered devices, if SMARTS gets any error while polling for an instance index, it just ignores them and continues to poll the index during next polling cycle.
But for V1 devices Smarts is default having the value (i.e. V1_continue_poll_after_error) set to FALSE. Which means that if there is any error while polling an instance index, Smarts will not poll that any more.
dmctl -s <IP domain name> get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
dmctl -s IPAM get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
FALSE
dmctl -s <IP domain name> put ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error TRUE
dmctl -s <IP domain name> invoke ICF_PolicyManager::ICF-PolicyManager reconfigure
sm_tpmgr -s <IP domain name> save