Collector disk Space issues - the WARN file uses more space in the Aria Operations for Networks nodes than is allocated

Products

• VCF Operations for Networks

Issue/Introduction

This article helps in resolving high disk usage for /var/log/

• Aria Operations for Networks may stop working when the disk is full.
• In the Aria Operations for Networks platform GUI, the alert shows not all the services are healthy and running.
• When the /var/log/ partition is more than 89% utilized, Samza containers services will stop to prevent the disk from going out of space.

Running the df -h command on the vRNI platform or collector node shows /var/log is utilized more than 90%




Navigate cd /var/log/

Run to sort the files with sizes 

 ls -laSh 

Environment

vRealize Network Insight

Aria Operations for Networks 6.##

Cause

This issue occurs when the warn file(s) utilizes more space in the Aria Operations for Networks Platform / Collector nodes. 

The warn file would have filled up with martian, rsyslogd and sometimes, corrupted IPFIX data flow sets.

Resolution

The Aria Operations for Networks Engineering is aware of the issue and working on fixing this issue which would be made available in future releases.

Workaround:
Follow the below steps for a workaround.

We can zero out the warn file by executing the following command 

1. Perform a manual cleanup of the warn file(s), see the steps below to be executed.
2. Take a Putty/SSH session on the problematic vRNI appliance/VM and log in with the user name 'support'
3. Execute the below command:
   
   

   sudo dd if=/dev/null of=/var/log/warn

   
   Refer to the screenshot below:
   
   
   
   
4. Run the below command to validate the disk usage. 
   

   df -h 

   
   
   
   
   
5. Run the following command to check the services on either the Platform or Collector node where the edits were made
   
   support@vrni-proxy-release:cd
support@vrni-proxy-release:~$ ub
ubuntu@vrni-proxy-release:~$ ./check-service-health.sh -p -d
   
   
   
   

Additional Information

• Aria Operations for Networks may stop working when the disk is full.
• In the Aria Operations for Networks platform GUI, the alert shows not all the services are healthy and running.
• When the /var/log partition is more than 89% utilized, samza containers services will stop to prevent the disk from going out of space.