BGP Flapping and "Edge node disk usage is high" alarms caused by /var/log/frr/frr.log growth

Products

• VMware NSX

Issue/Introduction

• Alarms are generated on the NSX manager for the edge indicating the disk usage is high or very high.
• Log in to the Edge node indicated in the alarm as root user and run df -h which shows the capacity utilization of the directory/var/log/ is high.

• To know which files or folders are consuming high space in /var/log/ run the following commands.

cd /var/log/
du -csxh * | sort -rh --> This will sort and display the files/folders consuming more space.

OR

find -size +10M -print  --> This will display the files that are consuming more than 10M space under /var/log/ directory

• As per the output below, it is evident that frr.log in /var/log/ directory is consuming more space

         

• Due to this disk space usage, edge node log extraction will fail on NSX UI.
• There will be intermittent packet drops on traffic traversing through the corresponding edge node. BGP peering might also be impacted due to high disk usage. 

Environment

VMware NSX 4.0.x - 4.1.x
VMware NSX 4.2.0 - 4.2.1.2

Cause

• The log rotate function is missing on FRR (Free Range Routing) service for NSX managers that are upgraded from 3.x to 4.x or from 4.x to 4.2.1.x  (less than 4.2.1.3)
• The issue is not seen for freshly deployed NSX managers i.e greenfield deployment of NSX 4.2.1.x (less than 4.2.1.3)

Resolution

This issue is resolved in VMware NSX 4.2.1.3 and later versions of NSX, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Follow the steps to ensure that log rotate for FRR is properly configured.

• Download the frr.txt file attached to this KB
• Login to the NSX edge node as user admin via putty/ssh.
• Switch to the root account by running the command st en
• Navigate to the directory cd /etc/logrotate.d/ 
• Create a new file using vi or vim editor by running the command vi frr and press i key to enter into insert mode.
• Copy paste the contents of frr.txt file that you downloaded from this KB. 
• Ensure the contents are properly copied and press Esc key
• Save the file by pressing :wq! and exit.
• To ensure the proper permission and ownership of the newly created file frr, run the commands below
  
chown root:root frr
chmod 644 frr
  
  
• The file permission should look like:

-rw-r--r-- 1 root root  973 Oct 25  2018 frr  

• The next time log rotate runs, it will read this file and rotate the /var/log/frr/frr.log file(s) every hour.

Additional Information

• If utilization of /var/log/ is 100% due to frr.log file occupying high space, the log rotation script will not work.
• The file frr.log needs to be truncated i.e moved to any other directory or folder.
• You can use the following command to truncate the frr.log file:
  > /var/log/frr/frr.log
• If the file is needed for historical reasons or for troubleshooting purpose, use WinSCP/SCP and copy the file to the local system and move/delete the file /var/log/frr.log