Underlying SAN storage workloads shows high Utilization up to 100%

Products

• VCF Operations for Networks

Issue/Introduction

1. Aria Operations for Networks Appliances are deployed with underlying SAN Storage.
   
   
2. Underlying SAN storage workloads shows high utilization up to 100% 
   
   Refer to below screenshots showing high Read and Write throughputs for underlying SAN Storage:
   
   
   
   
3. System Health shows Poor with High Lag is seen on GUI such as shown i below screenshot
   
   
   
   Note: In above screenshot Lags shows 4.2 h, however, higher lags can also be seen. 
   
   
4. Aria Operations for Networks database (Foundation DB) service shows running and healthy, however moving data component shows beyond 50 GB. A Higher moving data can also been seen raining between  250 GB to 500 GB or beyond.

 

Environment

Aria Operations for Networks 6.13.0
Aria Operations for Networks 6.14.0
Aria Operations for Networks 6.14.1

 

Cause

Aria Operations for Networks shows Higher Indexer Lag and Foundation Moving Data showing beyond 50 GB. ( Ranging between either 150 GB to 250 GB or beyond)
Under such as situation underlying SAN storage is expected to see high utilization upto 100%
This could be due combination of below performance related issues :
Lower IOPS, high churn of flow data or high flow versions where current deployment brick size is not able to process the data, Incorrect cpu/memory resources and incorrect reservation allocated.

Resolution

If you experience issue as described in Issue/Introduction section then Aria Operations for Networks deployment needs to be review by GS support further.

1. Open a support case with Broadcom Support to review your Aria Operations for Networks deployment. For more information, see Creating and managing Broadcom support cases. and refer to this KB
2. Capture below details:
   a) On Aria Operations for Networks GUI , Navigate to Settings>Infrastructure and support>Infrastructure and Updates pages, from there take 1-2 screenshots covering the entire page, additionally if you see any Problems Click on it and capture another screenshots showing all the problems.
   b) Open a SSH/Putty session on VMware Aria Operations for Networks Platform Node1, login with username support

   
   Execute below commands:

   ub
   ./run_all.sh uptime
   ./run_all.sh df -h
   ./run_all.sh sudo /home/ubuntu/check-service-health.sh -p -d
   sudo -u hbase hbase hbck
   sudo cat /home/ubuntu/build-target/deployment/patch.txt
   sudo cat /home/ubuntu/build-target/deployment/appliance.status
   sudo grep id: /etc/vnera/deployment/deployment.def
   fdbcli
   status details

   
   Note: Outputs of above commands are expected to be longer hence copy/paste the outputs to a Notepad file, save it and upload or sent as email attachment to this Case. 

Additional Information

Refer to System Recommendations and Requirements documentation here