"Edge node disk usage is high" alarm is generated on NSX manager due to /var/log/frr/frr.log

Products

• VMware NSX

Issue/Introduction

• Alarms are generated on the NSX manager for the edge indicating the disk usage is high or very high.
• Log in to the Edge node indicated in the alarm as root user and run df -h which shows the capacity utilization of the directory/var/log/ is high.

• To know which files or folders are consuming high space in /var/log/ run the following commands.

cd /var/log/
du -csxh * | sort -rh --> This will sort and display the files/folders consuming more space.

OR

find -size +10M -print  --> This will display the files that are consuming more than 10M space under /var/log/ directory

• As per the output below, its evident that frr.log is consuming more space

         

• Due to this disk space usage, edge node log extraction will fail on NSX UI and there might be intermittent packet drops on traffic traversing through the corresponding edge node.

Environment

VMware NSX

Cause

The log rotate function is missing on FRR service for NSX managers that are upgraded from 3.x to 4.x or from 4.x to 4.2.1.x  (less than 4.2.1.3)

The issue is not seen for freshly deployed NSX managers i.e greenfield deployment of NSX 4.2.1.x (less than 4.2.1.3)

Resolution

This issue is resolved in VMware NSX 4.2.1.3, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:-

• Download the frr.txt file attached to this KB
• Login to the NSX edge node as user admin via putty/ssh. Switch to the root account by running the command st en
• Navigate to the directory cd /etc/logrotate.d/ 
• Create a new file using vi or vim editor by running the command vi frr and press i key to enter into insert mode.
• Copy paste the contents of frr.txt file that you downloaded from this KB. 
• Ensure the contents are properly copied and press Esc key
• Save the file by pressing :wq! and exit.
• To ensure the proper permission and ownership of the newly created file frr, run the commands below
  chown root:root frr
chmod 644 frr
  
  
• The file permission should look like:

-rw-r--r-- 1 root root  973 Oct 25  2018 frr  

• The next time log rotate runs, it will read this file and rotate the /var/log/frr/frr.log file(s) every hour.

 

Additional Information

• If utilization of /var/log/ is 100% due to frr.log file occupying high space, the log rotation script will not work.
• The file frr.log needs to be truncated i.e moved to any other directory or folder.
• If the file is needed for historical reasons or for troubleshooting purpose, use Winscp/SCP and copy the file to the local system and move/delete the file /var/log/frr.log