SDN Tunnels remain in Portal Inventory after VNA Gateway deletion

Products

• DX NetOps
• CA Performance Management - Usage and Administration
• CA Virtual Network Assurance

Issue/Introduction

The DX NetOps Virtual Network Assurance (VNA) Gateway was deleted from the DX NetOps Performance Management Portal configuration.

In the Data Aggregator (DA) karaf.log file, after the VNA Gateway was deleted, we see the message that indicates it finished.

Finished cleaning up inventory associated with SDN Gateway <gatewayID>

The DA /rest/tunnels endpoint returns an empty result set showing none remain in DA inventory.

The Portal web server still shows an Inventory->Tunnels entry. There are 64 tunnels remaining after 20K+ were present prior to the VNA Gateway being deleted.

Running these MySql commands against the netqosportal DB on the Portal web server shows tunnels that remain in Portal Inventory from the DA Data Source (DS).

• Identify DS IDs with this query:
  • select SourceID,ConsoleName from data_sources2;
• List all tunnels present from the DA DS:
  • select * from dst_tunnel where SourceID= <DA_DS_ID>;
  • Replace <DA_DS_ID> with the ID from the prior query.
• To see just a count of items:
  • select count(*) from dst_tunnel where sourceid = <DA_DS_ID>;
  • select count(*) from ds_items where itemtypename = 'tunnel' and sourceid = <DA_DS_ID>;
  • Replace <DA_DS_ID> with the ID from the prior query above.

Environment

All supported DX NetOps Performance Management releases

Cause

Logging doesn't show an error during the deletion of the VNA Gateway that would prevent complete tunnel item removal.

Once the deletion was considered complete and successful, only manual intervention can remove the left over tunnels from Portal Inventory.

Resolution

To remove the tunnels from Portal inventory complete the following steps. Replace <DA_DS_ID> with the DA DS ID. 

1. Log into the MySql prompt using this command. Enter the password when prompted. Assumes default home path is used.
   • /opt/CA/MySql/bin/mysql -uroot -p netqosportal
2. Identify the DA DS ID with this query:
   • select SourceID,ConsoleName from data_sources2;
3. In the prompt run these to see how many are in each table.
   
   • select count(*) from dst_tunnel where sourceid = <DA_DS_ID>;
   • select count(*) from ds_items where itemtypename = 'tunnel' and sourceid = <DA_DS_ID>;
4. Set UpdatedOn=0 to trigger removal of the problem items in both tables:
   
   • update dst_tunnel set updatedOn=0;
   • update ds_items set updatedon=0 where ItemTypeName='tunnel' and sourced = <DA_DS_ID>;
5. Wait for the next incremental sync for the DA DS to complete.
6. Wait for the next Global Synchronization cycle to complete.

After completing those steps, after a DA sync is followed by a Global Sync, the items should be removed from Portal inventory.