Aria Operations stops collecting data due to large number of distinct NSX BGP Neighbor Instanced Metric metric keys.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Aria Operations shows problems collecting data on all adapters and as a result dashboards are showing no data.  
• You have NSX-T management pack installed and is configured to collect in your environment
• The issue with collection is temporarily resolved when the cluster is taken offline/online but it will reoccur after sometime.
• /storage/db is high on some or all of the nodes and is increasing gradually. 
• You may find the cluster goes offline automatically due to running out of storage if storage/db is above 96% usage. 
• /storage/log/vcops/log/analytics-xxxxxxxxxxxxx.log show performance-based error

  YYYY-MM-DDTHH:MM:SS,958+0000 FATAL [DataForwarder]  com.vmware.vcops.platform.common.DataForwarder.sendData - Failed to send data: forward data queue is backing up, data is being lost!

• The BGPNeighborInstancedMetric cnt is substantially higher than other metrics when running the following command on root ssh session 

  su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433 -c \"SELECT LEFT(metric_key, POSITION(':' in metric_key) - 1) AS metric_key_prefix, count(*) AS cnt FROM metric_key GROUP BY metric_key_prefix HAVING COUNT(*) > 1 UNION ALL SELECT 'TOTAL_ROWS', COUNT(*) FROM metric_key ORDER BY cnt DESC;\""

Environment

Aria Operations 8.x

Cause

NSX-T Management Pack can generate a huge number of distinct BGP Neighbor Instanced Metric metric keys,  these metrics lead to a few objects having very large FSDB data files, which appears to impact the performance of saving these metrics, it leads to other performance issues, like collections not completing normally.

Resolution

Disable the problematic metrics below  in the  active policy in Aria Operations

 

Prior to 8.18

 

1. Configure  -> Policies -> Policy Definition

 

 

2. Select Active Policy (following change to be done on all Active policies)

 

 

3. Click EDIT POLICY

 

 

4. Click Metrics and Properties

 

 

5. Router Service > Metrics > BGP Neighbor > Advertised Routes > Transport Nodes > Routes:

 

  Deactivate these items and Save

• AS Path
• Next Hop IP Address

 

 

From 8.18

 

1. Go to Operations -> Configurations -> Policy Definition

 

2. Select Active Policy

 

3. Edit Policy

 

4. Select Metrics and Properties

 

5. Router Service > Metrics > BGP Neighbor > Advertised Routes > Transport Nodes > Routes:

 

  Deactivate these items and Save

• AS Path
• Next Hop IP Address