Smarts IP: Interfaces on few devices in Smarts have status NOTPRESENT when the status on MIB is UP
Article ID: 331771
Updated On:
Products
VMware Smart Assurance
Issue/Introduction
Symptoms:
Smarts is not polling interfaces properly. Many random interfaces are showing Admin or Oper status as NOTPRESENT, which is not the actual status on the MIB.
No changes have been done in device configuration or Smarts.
Smarts is not polling interfaces properly. Many random interfaces are showing Admin or Oper status as NOTPRESENT, which is not the actual status on the MIB.
No changes have been done in device configuration or Smarts.
Environment
VMware Smart Assurance - SMARTS
Cause
1) Checked one interface as an example, it is instrumented for Fault by MIB2 in Smarts, with admin status UP and oper status NOTPRESENT.
-------------------------------------------
dmctl -s IPAM get Interface::IF-DSA/67149824::InstrumentedBy
{ Interface_Fault_MIB2::I-Interface_Fault_MIB2-IF-DSA/67149824 }
dmctl -s IPAM get Interface::IF-DSA/67149824::AdminStatus
UP
dmctl -s IPAM get Interface::IF-DSA/67149824::OperStatus
NOTPRESENT
-------------------------------------------
2) Checked the snmp output from device, both have value 1, which means they are UP:
-------------------------------------------
Admin Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.7.67149824
.1.3.6.1.2.1.2.2.1.7.67149824 = 1
Oper Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.8.67149824
.1.3.6.1.2.1.2.2.1.8.67149824 = 1
-------------------------------------------
3) Enabled trace on device in Smarts with command listed in KB 321711, https://support.emc.com/kb/321711
From the log, there is no evidence of polling details of Oper status of the interface, only Admin Status is captured.
-------------------------------------------
cat IPAM_en_US_UTF-8.log_trace | grep 67149824
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
-------------------------------------------
4) Captured debug data of SNMP Poller on the domain with below command:
dmctl -s <domain name> exec dmdebug --dumpAccessor=DEVSTAT-SNMP-Poller --output=accessor.log
From the log, there is an error on polling of Oper Status of the interface and the polling was last successful 1 day prior.
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifAdminStatus
Polling Period: 240
Last polled At: November 14, 2017 2:33:49 PM GMT+01:00
Cached Value: 1
Last Error: <OK>-E-<OK>-Normal successful completion
Unconstrained Polling Period: 15
Able To Poll: TRUE
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifOperStatus
Polling Period: 240
Last polled At: November 13, 2017 2:39:35 PM GMT+01:00
Cached Value:
Last Error: SNMP-E-ERR_NOSUCHNAME-No such name
Unconstrained Polling Period: 15
Able To Poll: FALSE
-------------------------------------------
5) The SNMP version in use for the device is V1:
-------------------------------------------
dmctl -s IPAM get SNMPAgent::SNMPAgent-DSA | grep SNMP
SNMPStatus = OK
SNMPVersion = V1
-------------------------------------------
For all the Non-V1 devices, Smarts never disables polling if it encounters any error. Which means for V2c or V3 discovered devices, if SMARTS gets any error while polling for an instance index, it just ignores them and continues to poll the index during next polling cycle.
But for V1 devices Smarts is default having the value (i.e. V1_continue_poll_after_error) set to FALSE. Which means that if there is any error while polling an instance index, Smarts will not poll that any more.
-------------------------------------------
dmctl -s IPAM get Interface::IF-DSA/67149824::InstrumentedBy
{ Interface_Fault_MIB2::I-Interface_Fault_MIB2-IF-DSA/67149824 }
dmctl -s IPAM get Interface::IF-DSA/67149824::AdminStatus
UP
dmctl -s IPAM get Interface::IF-DSA/67149824::OperStatus
NOTPRESENT
-------------------------------------------
2) Checked the snmp output from device, both have value 1, which means they are UP:
-------------------------------------------
Admin Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.7.67149824
.1.3.6.1.2.1.2.2.1.7.67149824 = 1
Oper Status:
sm_snmp -c public -d DSA get .1.3.6.1.2.1.2.2.1.8.67149824
.1.3.6.1.2.1.2.2.1.8.67149824 = 1
-------------------------------------------
3) Enabled trace on device in Smarts with command listed in KB 321711, https://support.emc.com/kb/321711
From the log, there is no evidence of polling details of Oper status of the interface, only Admin Status is captured.
-------------------------------------------
cat IPAM_en_US_UTF-8.log_trace | grep 67149824
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
73: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
76: OBJ-ID (0x06), 13 bytes == ".1.3.6.1.2.1.2.2.1.7.67149824"
-------------------------------------------
4) Captured debug data of SNMP Poller on the domain with below command:
dmctl -s <domain name> exec dmdebug --dumpAccessor=DEVSTAT-SNMP-Poller --output=accessor.log
From the log, there is an error on polling of Oper Status of the interface and the polling was last successful 1 day prior.
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifAdminStatus
Polling Period: 240
Last polled At: November 14, 2017 2:33:49 PM GMT+01:00
Cached Value: 1
Last Error: <OK>-E-<OK>-Normal successful completion
Unconstrained Polling Period: 15
Able To Poll: TRUE
-------------------------------------------
Instance Name: I-Interface_Fault_MIB2-IF-DSA/67149824
Attribute Name: ifOperStatus
Polling Period: 240
Last polled At: November 13, 2017 2:39:35 PM GMT+01:00
Cached Value:
Last Error: SNMP-E-ERR_NOSUCHNAME-No such name
Unconstrained Polling Period: 15
Able To Poll: FALSE
-------------------------------------------
5) The SNMP version in use for the device is V1:
-------------------------------------------
dmctl -s IPAM get SNMPAgent::SNMPAgent-DSA | grep SNMP
SNMPStatus = OK
SNMPVersion = V1
-------------------------------------------
For all the Non-V1 devices, Smarts never disables polling if it encounters any error. Which means for V2c or V3 discovered devices, if SMARTS gets any error while polling for an instance index, it just ignores them and continues to poll the index during next polling cycle.
But for V1 devices Smarts is default having the value (i.e. V1_continue_poll_after_error) set to FALSE. Which means that if there is any error while polling an instance index, Smarts will not poll that any more.
Resolution
The default setting of V1_continue_poll_after_error needs to be set to TRUE on the IP domain, for Smarts to continue to poll the instance index, in case of any error received during previous polling.
Follow the below steps to set the same:
1) Check for the current value of the flag with below command:
dmctl -s <IP domain name> get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
Ex:
dmctl -s IPAM get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
FALSE
2) Set the value to TRUE with below command:
dmctl -s <IP domain name> put ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error TRUE
3) Execute the command of Step(1) to check for the value update.
4) Execute Reconfigure on the IP domain with below command:
dmctl -s <IP domain name> invoke ICF_PolicyManager::ICF-PolicyManager reconfigure
Check for Reconfigure status with command in KB 334942, https://support.emc.com/kb/334942
5) Once reconfigure is complete, save the IP domain repository with below command:
sm_tpmgr -s <IP domain name> save
6) Rediscover the device (or) Execute "Discover All" on the domain.
Follow the below steps to set the same:
1) Check for the current value of the flag with below command:
dmctl -s <IP domain name> get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
Ex:
dmctl -s IPAM get ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error
FALSE
2) Set the value to TRUE with below command:
dmctl -s <IP domain name> put ICIP_SNMPAccessorInterface::DEVSTAT-SNMP-Poller::V1_continue_poll_after_error TRUE
3) Execute the command of Step(1) to check for the value update.
4) Execute Reconfigure on the IP domain with below command:
dmctl -s <IP domain name> invoke ICF_PolicyManager::ICF-PolicyManager reconfigure
Check for Reconfigure status with command in KB 334942, https://support.emc.com/kb/334942
5) Once reconfigure is complete, save the IP domain repository with below command:
sm_tpmgr -s <IP domain name> save
6) Rediscover the device (or) Execute "Discover All" on the domain.
Feedback
Yes
No
Powered by
