A significant number of IPSLA Response Path Tests on Cisco devices are not reporting data in Performance Management.
All Views for the data from these items return "No Data To Display" or "Invalid Data".
The tests are functioning at the device level.
Re-discovering the device and the metric families and even deleting the device and rediscovering it makes no difference.
The cause for the issue is a device based problem. The test bucketID values should be less than the devices sysUpTime. The problem devices were restarted but the bucketID values didn't get reset.
Detailed Poll Logging through the Data Aggregator DCDebug page (<DA_HOST>:8581/dcdebug) shows the following error for all related items.
Feb 18 21:16:16.489: CiscoIPSLAFirstPollListener processing GETNEXT response=ResponseEvent [source=Snmp, address=<Device_IP>/161, request=GETNEXT[requestID=951785610, errorStatus=Success(0), errorIndex=50, VBS[184.108.40.206.220.127.116.11.18.104.22.168.1.4.103 = Null; 22.214.171.124.126.96.36.199 = Null]], response=RESPONSE[requestID=951785610, errorStatus=Success(0), errorIndex=0, VBS[188.8.131.52.184.108.40.206.220.127.116.11.18.104.22.1680360000 = 1200; 22.214.171.124.126.96.36.199.0 = 62 days, 5:55:54.05]], userObject=ItemBasedRequestState[responseReceivedTimestamp=1582060576489, nextIndex=1, itemList=]
Feb 18 21:16:16.489: Initial bucket discovery failed for IP=<Device_IP>, item ID=3615901, discoveredOID=188.8.131.52.184.108.40.206.220.127.116.11.1.4, testIndex=103. Not sending SNMP GET request and sending UNEXPECTED_END_OF_TABLE response
What we should see, from a working device with the same kind of tests is the following.
Feb 18 21:17:46.243: CiscoIPSLASubsequentBucketDiscoveryListener processing successful GETNEXT response=ResponseEvent [source=Snmp, address=<Device_IP>/161, request=GETNEXT[requestID=1790384248, errorStatus=Success(0), errorIndex=50, VBS[18.104.22.168.22.214.171.124.126.96.36.199.188.8.131.5259166294 = Null; 184.108.40.206.220.127.116.11 = Null]], response=RESPONSE[requestID=1790384248, errorStatus=Success(0), errorIndex=0, VBS[18.104.22.168.22.214.171.124.126.96.36.199.188.8.131.5258806294 = 2400; 184.108.40.206.220.127.116.11.0 = 435 days, 2:34:34.33]], userObject=ItemBasedRequestState[responseReceivedTimestamp=1582060666243, nextIndex=1, itemList=]
Feb 18 21:17:46.243: OID wasn't Sys Up Time OR a new Bucket, it was : 18.104.22.168.22.214.171.124.126.96.36.199.188.8.131.5258806294 also, var.getOid shows 184.108.40.206.220.127.116.11.18.104.22.168.22.214.171.12458806294
Broadcom Engineering examined the code to determine what is done when the UEOT (UNEXPECTED_END_OF_TABLE) message is printed in releases r3.6.x and earlier. We get bucketid from the response, and we also store off sysuptime from the response. Then we compare bucketid and sysuptime to confirm the bucketid < sysuptime value. If it's not, we encounter a conditional clause where we see if we have a previous discovered bucket. If not, we set the UEOT error.
For the failure example above, if we calculate the values out we see:
To translate the timeticks value to days we use "timeticks/8640000=days" where 8640000 represents number of seconds in a day. We end up with:
Here we see the bucket ID equals 145.875 days and sysUpTime shows just over 62 days.
Does this prove out for the working device example above at test index 211?
Crunch the numbers and we get:
Bucket ID is just less than sysUpTime and checks out as correct.
Performance Management r3.6.9 or 3.7.7 and earlier releases
This issue is resolved in changes to IPSLA polling where we allow this device to issue and still provide the expected metric data.
The changes were introduced in release 3.6.10 and 3.7.8.
To resolve this, you can choose to: