IPSLA Response Paths stop showing data even though the devices themselves continue to be polled, UNEXPECTED_END_OF_TABLE error in logs

Products

• DX NetOps
• CA Performance Management - Usage and Administration

Issue/Introduction

In some situations in CA Performance Management (CAPM), IPSLA Response Paths (Cisco RTTMON) stop showing data even though the device itself continues to be polled and continues to show data for other device elements (such as CPU, memory, Bandwidth Utilization etc).

When looking at the logs, it shows an UNEXPECTED_END_OF_TABLE error in the Response Path MIB.

Environment

DX NetOps CAPM all currently supported releases

Cause

Looking at the dcdebug data and the Response Path VC that is configured is CiscoRttMonStatsMib:

Polling Config Item ID=3462230
Poll Group ID=426
Normalized Facet Type={http://im.ca.com/normalizer}NormalizedICMPInfo
Certification Facet Type={http://im.ca.com/certifications/snmp}CiscoRttMonStatsMib
PollingConfig's DcmID=DC1:db2c97c7-7d97-4ae9-a513-84240c0013c1
Polled Items(1)
        Item ID=3076942, Name=Cisco Rttmon ICMP: 0.0.0.0-103.230.16.1 : 5555, Index List={[5555, 5555, 5555]}
</br>

Looking at the OIDs in this VC and then searching the detailedPolling log and we find many instances of the following type of error:

Aug 02 16:07:12.536: CiscoIPSLADynamicDiscovery sending ERROR, baseOID: 1.3.6.1.4.1.9.9.42.1.3.1.1.7.5555, userObject=ItemBasedRequestState[responseReceivedTimestamp=1722586032536(Fri Aug 02 16:07:12 SGT 2024), itemList=[3076942]], error=UNEXPECTED_END_OF_TABLE

This UNEXPECTED_END_OF_TABLE error shows that the device did not send the complete MIB table in response (SNMP GET_RESPONSE) to CAPM's polling (SNMP GET_REQUEST).

Running a MIB walk using sapwalk2, it ends with:

...
1.3.6.1.2.1.4.24.4.1.1.192.168.0.2.255.255.255.0.0.192.168.1.1, IpAddress   , 192.168.0.2

1.3.6.1.2.1.4.24.4.1.1.192.168.0.1.255.255.252.0.0.192.168.1.1, IpAddress   , 192.168.0.1

#Error: Device not responding

So device is stopping at this point to any SNMP polling regardless of source.

The MIB walk doesn't contain any IPSLA data (OID 1.3.6.1.4.1.9.9.42.1.3.2.1) and in this case, it stops abruptly at the ipCidrRouteEntry (1.3.6.1.2.1.4.24.4.1.1) table.

The VC for IPSLA uses:

<UsesDynamicIndex>true</UsesDynamicIndex>
          <Source>1.3.6.1.4.1.9.9.42.1.3.1.1.7</Source>

But this OID is not in the eMIB walk, it comes after the point where the device stops responding. So all values for this are NULL.

CAPM tries to set the value of dynamicIndex in CiscoIPSLADynamicDiscoveryResult. But since full OID in the class was null, dynamicIndex was set to null also. So no data is returned and none shown.

Hence, this is a device issue.

Resolution

When CAPM tried to query 1.3.6.1.4.1.9.9.42.1.3.2.1.7.5555 (rttMonStatsCollectVerifyErrors) with GetNext, the device in this case returned 1.3.6.1.4.1.9.9.43.1.1.1.0 = 399 days, 16:28:30.08 (ccmHistoryRunningLastChanged). However, sysUpTime in this case is 1.3.6.1.2.1.1.3.0 = 148 days, 14:04:10.97.

 

The sysUpTime should be the same as ccmHistoryRunningLastChanged (or greater) but since it's much less, it shows that the counter has overflowed and rolled over (i.e. a number too large for the device to handle in this field)

 

So it appears this device hasn't been rebooted in over 500 days when combining the two and there is a problem with the SNMP agent running on it.

 

To fix this, the device may need to be rebooted or at least restart the SNMP agent on it. It maybe a bug in the device or SNMP agent itself, but this is something that needs to be corrected on the device as it cannot be overcome in CAPM.