High Disk Utilization is seen on Aria Operations for Networks GUI

Products

VCF Operations for Networks

Issue/Introduction

This is seen on all the platform Clustered Nodes. Refer to error screenshots as below:

/var/ partition shows 89% when executed below command

./run_all.sh df -h

Collector logs at location /var/log/arkin/collector below error are seen:

2025-05-28T17:26:42.271Z INFO nsxt.tasks.ManageNSXTLatencyTask NSXT_<NSXT IPAddress/FQDN>_Config_OpMgr-0 createNSGroupForTNs:613 New Child NSGroup added, id /infra/domains/default/groups/vRNI-Node_Group_Profile_TN_829###ef-###-###-8453-#######362, member count 40

2025-05-28T17:26:42.326Z ERROR dataprovider.utils.HttpUtils NSXT_<NSXT IP Address/FQDN>_Config_OpMgr-0 checkCodeAndThrow:53 Could not get response for /policy/api/v1/infra/domains/default/groups/vRNI-Node_Group_Profile_RFF####==, status 400
2025-05-28T17:26:42.326Z ERROR dataprovider.utils.HttpUtils NSXT_<NSXT IPAddress/FQDN>_Config_OpMgr-0 checkStatusAndThrow:41 API /policy/api/v1/infra/domains/default/groups/vRNI-Node_Group_Profile_RFF####== error response {
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 500127,
  "module_name" : "Policy",
  "error_message" : "Cannot create an object with path=[/infra/domains/default/groups/vRNI-Node_Group_Profile_RFF####==] as it already exists."
}

Login to NSXT GUI and Navigate to Inventory>Groups

Search with vRNI or vRNI Node_Group_Profile_TN you should see a huge number of profiles created.

Refer to screenshots below:

Environment

Aria Operations for Networks 6.12.0
Aria Operations for Networks 6.12.1
Aria Operations for Networks 6.13.0
Aria Operations for Networks 6.14.0

Cause

An investigation of collector logs revealed that improper cleanup procedures resulted in the removal of vRNI\_Groups from the vRNI KV Store, while corresponding entries were not deleted from NSX-T.
Consequently, during latency enablement, while TN_Groups were successfully created, an exception was encountered during the creation of vRNI_Group, indicating a pre-existing group.

During this time the Watcher task operates continuously in the background, generating a substantial number of TN_Groups. Because of which we saw a high number of TN_Groups.

Resolution

VCF Operation for Networks Product team is aware of this issue and is working on a fix for this.

To workaround this issue this issue we need to:

1. Disable latency metric from Aria Operations from Network GUI

2. Clean up the stale entries for the profile vRNI-Node_Group_Profile_TN_* on the NSXT GUI.

Open a support case with Broadcom Support to review your Aria Operations for Networks deployment. For more information, see Creating and managing Broadcom support cases.

Attachments

nsx_t_group_delete-1.py get_app