IPSLA test data missing in DX NetOps Portal reports
Article ID: 388117
Updated On:
Products
Issue/Introduction
IPSLA tests are discovered and being polled for SNMP based metric data. The reports in DX NetOps Performance Management Portal UI used to show the data.
There is no longer current data seen as of a few days ago.
The devices are using the correct Metric Family and Vendor Certification. The devices have IPSLA test items discovered, created and polled. All other polled metrics on the devices are current except for the test data.
Some devices have tests that continue to show data where tests on the same device are missing data.
Gathering detailed poll logging for an impacted device, covering a 2 hour time frame, reveals the problem. See the KB Enable and download logging from DCDebug in Performance Management for steps to enable it. Enable it with filtering for the Poll Group ID of the target items to limit the logging generated.
Environment
All supported DX NetOps Performance Management releases.
Cause
The following is sample data showing the issue from a single IPSLA test index. The data referenced was generated through the DA DCDebug page. See the KB referenced in the Issue/Introduction for steps.
This is where we see the attempt to discover a new bucket. This has us doing an INITIAL read for bucket for OID.Test "1.3.6.1.4.1.9.9.42.1.3.2.1.7.130", based on the previous known bucket being "1180120240.1.1" and it's returning "1.3.6.1.4.1.9.9.42.1.3.2.1.7.140.1179766094.1.1".
Jan 23 10:24:42.256: CiscoIPSLADynamicDiscovery processResponse called: baseOIDWithTestIndex=1.3.6.1.4.1.9.9.42.1.3.2.1.7.130, responseEvent=[ResponseEvent [,
source=Snmp,
address=#.#.#.#/161,
request=GETNEXT[requestID=1894649069, errorStatus=Success(0), errorIndex=0, VBS[,
1.3.6.1.4.1.9.9.42.1.3.2.1.7.130.1180120240.1.1 = Null,
1.3.6.1.2.1.1.3 = Null,
]],
response=RESPONSE[requestID=1894649069, errorStatus=Success(0), errorIndex=0, VBS[,
1.3.6.1.4.1.9.9.42.1.3.2.1.7.140.1179766094.1.1 = 0,
1.3.6.1.2.1.1.3.0 = 136 days, 14:14:32.08,
]],
userObject=ItemBasedRequestState[responseReceivedTimestamp=1737638682256(Thu Jan 23 10:24:42 ART 2025), itemList=[10355848]],
error=null, ]]
That is test index 140. This means the device is telling us there are no newer buckets for test index 130. The next one available is for test index 140.
With bucket ID "1179766094.1.1" being lower than the current bucket "1180120240.1.1", we continue to use "130.1180120240.1.1" for polling of that test index.
When then query the device for the current bucket and Index we use what we still know, the "130.1180120240.1.1" value. The device responds that it's not found. We see that in the device responses that read noSuchInstace. These are a sample of those messages.
Jan 23 10:24:42.260: response=RESPONSE[requestID=1894649075, errorStatus=Success(0), errorIndex=0, VBS[
Jan 23 10:24:42.260: 1.3.6.1.4.1.9.9.42.1.3.1.1.7.130.1180120240.1.1.1 = noSuchInstance
Jan 23 10:24:42.260: 1.3.6.1.4.1.9.9.42.1.3.1.1.5.130.1180120240.1.1.1 = noSuchInstance
Jan 23 10:24:42.260: 1.3.6.1.4.1.9.9.42.1.3.1.1.10.130.1180120240.1.1.1 = noSuchInstance
Jan 23 10:24:42.260: 1.3.6.1.4.1.9.9.42.1.3.1.1.11.130.1180120240.1.1.1 = noSuchInstance
Resolution
The test results for these IPSLA tests are written to buckets on the device. The device is supposed to create a new bucket for the test index to store new test data on an hourly cycle. The old bucket is dropped or deleted and a new one is created.
The NetOps SNMP polling of these tests continually attempts to discover a new bucket to use for each tests index entry. This takes place on an hourly basis. We update the item with the new bucket ID and poll the new bucket for the index for data. The device should be creating a new bucket, dropping the old one.
The device responses to our hourly bucket index discovery is returning the same known bucket, indicating there is no new one available. When we try to poll the one we still have as 'current' we get a response of noSuchInstance. That suggests the bucket index polled doesn't exist, meaning the device got rid of it.
The device needs investigation, by it's admin or the vendor, to determine why it is no longer creating new buckets to represent the tests data for the current test cycle. Once resolved the data will return.
Feedback
