Smarts SAM: Number of devices monitored by a Smarts SAM subscribing to multiple IP-AMPM domains changes regularly at the SAM level but remain constant at the underlying IP-AMPM domain level

Products

• VMware

Issue/Introduction

Symptoms:




Number of devices monitored by a Smarts Service Assurance Manager (Smarts SAM) subscribing to multiple IP-AMPM domains changes regularly at the SAM level but remains constant at the underlying IP-AMPM domain level
Added the following dmctl command to a cron job to regularly log the number of devices in SAM and IP topologies:

dmctl -b <broker> -s <domain> geti UnitaryComputerSystem | tr ' ' '\012' | grep -v "^$" | wc -l

Command output shows fluctuation in SAM device numbers but the AM domain device numbers stay constant:

Mon May 21 07:15:01 BST 2012
SAM 794
AM1 204
AM2 197
AM3 182
AM4 211
Mon May 21 07:30:02 BST 2012
SAM 794
AM1 204
AM2 197
AM3 182
AM4 211
Mon May 21 07:45:01 BST 2012
SAM 703
AM1 204
AM2 197
AM3 182
AM4 211
Mon May 21 08:00:02 BST 2012
SAM 703
AM1 204
AM2 197
AM3 182
AM4 211
Mon May 21 08:15:02 BST 2012
SAM 794
AM1 204
AM2 197
AM3 182
AM4 211
Mon May 21 08:30:01 BST 2012
SAM 794
AM1 204
AM2 197
AM3 182
AM4 211

Smarts SAM log shows that Concurrent Topology Synchronizations to the underlying AM domains were in progress at the time of the device number fluctuation as indicated in the following:

May 21, 2012 07:29:13 PM GMT+01:00 Topology-Sync Started From: AM1 is using: /opt/InCharge8/SAM/smarts/local/conf/ics/dxa-conn-perf.conf
May 21, 2012 07:38:28 PM GMT+01:00 Topology-Sync Started From: AM2 is using: /opt/InCharge8/SAM/smarts/local/conf/ics/dxa-conn-perf.conf
May 21, 2012 07:43:42 PM GMT+01:00 Topology-Sync Started From: AM3 is using: /opt/InCharge8/SAM/smarts/local/conf/ics/dxa-conn-perf.conf

Environment

VMware Smart Assurance - SMARTS

Cause

There is a race condition that is encountered in FastDXA Concurrent Mode of topology synchronization which is allowing a reconciliation operation (END_IMPORT) of the book keeping data structure to be processed while BEGIN_IMPORT operation is still in progress. This condition is leading to deletion of some objects in the SAM repository and inconsistency between the SAM topology and the underlying domain topologies. See 15860 ("What are the different modes of FastDXA Topology Synchronization") for more information on FastDXA TopoSynch operations

Resolution

This issue has been addressed in Smarts SAM 9.1 and Smarts SAM 8.1.4. EMC recommends that you upgrade your Smarts SAM installation to one of these (or later) versions to address this issue. The latest software releases available for this product are available for download on the EMC Online Support site (https://support.emc.com) as follows:

1. From the EMC Online Support site, select Support > Support By Product.
2. In the "Find a product" field, search for "Smarts Service Assurance Manager" (field will auto-suggest as you type).
3. From the product page, select Downloads and browse to the software download you require.

Workaround
If you encounter this issue in your environment and cannot upgrade Smarts SAM, you can change the FastDXA mode used by the Smarts SAM environment to Serial Mode TopoSynch. To enable Serial Mode TopoSynch in your environment, do the following:

1. Configure the following setting in the local/conf/runcmd_env.sh file:
   
   SM_ENABLE_SERIAL_SYNCS=TRUE
   
    
2. Restart the Smarts SAM domain.

Contact EMC Customer Support and reference this solution ID if you need further assistance with this workaround.