The /var/log partition grows exponentially without rotating on VMware NSX-T Controller Node

Products

• VMware
• VMware NSX Networking

Issue/Introduction

Symptoms:

• In the NSX-T 2.3.x dashboard, you see disk usage alert for controller node similar to:
  Disk usage is high for filesystem /dev/mapper/nsx-var+log

• When you check the volume mount information on the controller, you see that /var/log/moot-server/Zookeeper_0  directory taking up most of the space.

• You see that zookeeper.log, zookeeper-cli.log files does not get compressed.

Environment

VMware NSX-T Data Center
VMware PKS 1.x
VMware NSX-T Data Center 2.x

Cause

The issue occurs as the log rotation does not work as expected. 

Resolution

This is a known issue with VMware NSX-T 2.3.0 and 2.3.1.

This issue is resolved in VMware NSX-T 2.4.1 and later.

Workaround:

To work around this issue in NSX-T 2.3.x, change the log file rotation parameters as mentioned below on the problematic Controller node:

1. Log in to Controller node as root user.

2. Edit the log4j.properties file located at:
   /config/moot-server/localServiceManager/containers/Zookeeper/persistentStore/zkWorkspace

3. Update the following parameters:
   zookeeper.log.maxfilesize=256MB
   zookeeper.log.maxbackupindex=20
   Set these values as per the requirement and free space on controller node.

4. Validate the changes:
   # /var/log/moot-server/Zookeeper_0# grep "zookeeper.log.maxfilesize\|zookeeper.log.maxbackupindex" /config/moot-server/localServiceManager/containers/Zookeeper/persistentStore/zkWorkspace/log4j.properties
   zookeeper.log.maxfilesize=128MB
   zookeeper.log.maxbackupindex=5

5. Restart the controller to apply the changes.

Note: Changes on the log4j.properties will be persistent after the reboot of the controller only. Changes will not be reflecting if you do service restart like this /etc/init.d/moot-server restart

Additional Information

Impact/Risks:

This will cause the control plane to malfunction as the moot service (which is a container for zookeeper) will stop. Due to this zookeeper may not start and control cluster will be impacted. So controller will not function as expected.