Smarts IP: IPSEC Alerts showing wrong status; cipSecTunStatus polling with SNMP-E-EXP_NOSUCHINSTANCE-No such instance

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


IPSec tunnel status is incorrect

IPSec tunnels status are not updating promptly enough following a change in index caused by a DOWN then UP condition .

Environment

VMware Smart Assurance - SMARTS

Cause

From the SNMP dump accessor we can see the following error:

Instance Name: I-Interface_Fault_CiscoIPSec-IF-abc-iface101/IPSEC-Tunnel-177.77.77.37
Attribute Name: cipSecTunStatus
Polling Period: 120
Last polled At: October 31, 2014 3:21:41 PM GMT
Cached Value: 
Last Error: SNMP-E-EXP_NOSUCHINSTANCE-No such instance

The last error indicates that the SNMP agent responded with NOSUCHINSTANCE which we map it to critical state and flag it DOWN.
Basically the tunnel interfaces are non-reliable and might go DOWN and then come UP at times. But at the same time when it comes UP, it gets a different index assigned to itself.
In the current scenario, its likely that this tunnel interface had gone DOWN once and came up with the different interface index. But since SMARTS does not know about the new index, it continues polling the old index and hence the device agent is responding with NOSuchInstance.
 
We need to enable Smarts so that the re-indexed interface is updated and can continue to monitor the same tunnel interface with a new index to it.

Resolution

The issue has been fixed in the follow patches:

Smarts IP version 9.3 patch 10 (9.3.0.10).
Smarts IP version 9.4 Patch 1 (9.4.0.1).

Please upgrade to one of these versions to attain the fix.

If you cannot upgrade, please see workaround below:

Workaround:
To update the new index to the tunnel interface, use a short discovery probe which is minimized version of the discovery probe.
This short discovery selectively walks a particular table and re-indexes the interface that has got changed.


To do this please follow the below procedure:

Note: Please used sm_edit in the <InstallDir>/smarts/bin/ directory to edit these files.

1. In order to discover IP-Sec tunnels, enable below parameter in the "tpmgr-param.conf"
EnableIPSecDiscovery TRUE

2. and in "discovery.conf" enable short discovery by setting 
autoReprobe_short = TRUE.

3. In tpmgr-param.conf, add Interface_Fault_CiscoIPSec to ShortDiscoveryInstrPattern as follows:
ShortDiscoveryInstrPattern Card_Fault_CiscoONSCPU|Card_Fault_CiscoEntityFRU|Interface_Fault_CiscoIPSec

4. In discovery.conf, set 
numberShortProbeThreads = 7
autoReprobe_short = TRUE
reprobePeriod_short = 180

Please set the above parameter and following a restart of the server, check the short discovery is happening for every 3 minutes (The logs will indicate this). 

Note: The default for reprobePeriod_short=900 which is 15 mins. We have reduced this so that the index is picked up. 

5. If the device in question has Sysoid : .1.3.6.1.4.1.9.1.544, the containment driver is :  Cisco-Router-CardSelect ,  and we need to add
the short containment driver for this device as follows: 

Please sm_edit the  DISCOVERY_CISCO.import  present under local/conf/discovery directory and add the below short containment driver at the end of the file:

GA_CompoundDriver::Short-Containment-Cisco-Router-CardSelect-Driver {
drivers = {
{"Cisco-Check-IpSec-Driver", 10}
}
waitForCompletion = TRUE
}

6. Restart the server for the changes to take effect

Additional Information

For reference please see the Read Me notes for patches below:

https://support.emc.com/docu52261_Smarts-SAM,-IP,-NPM,-MPLS,-ESM,-and-VoIP-9.3.0-Patch-Readme-Installation-Instructions.pdf?language=en_US
https://support.emc.com/docu58687_Smarts-SAM,-IP,-NPM,-MPLS,-ESM,-OTM-and-VoIP-Version-9.4.0-Cumulative-Patch-Readme-Installation-Instructions.pdf?language=en_US