Logrotate service has issues in rotating logs for nginx
, syslog
, warn
, etv.
Log files of syslog
, warn
and nginx
grows and consumes disk space causing disk out of space issues.
syslog.1
instead of syslog
file. (Location is /var/log/
)warn.1
instead of warn
file. (Location of log is /var/log/
)access.log.1
instead of access.log
file (Location of log is /var/log/nginx/
)Since the service is writing to file with '1', logrotate never performs the log rotation, because it checks the main file for rotation and its always empty.
It doesn't perform rotation on empty files and hence syslog.1, warn.2, access.log.1 files grows bigger and bigger.
VMware Aria Operations for Networks 6.11.0
After log rotation is performed it triggers a reload call to the respective service to reload the log files, so that it will continue to write to correct file.
This mechanism is broken and as a result files grow huge in size making /var/log >90%
Aria Operations for Networks Engineering team is aware of this and a patch (6.11.0.P2.1699450092.patch.bundle) on top of Aria Operations for Networks Version 6.11 is made available which fixes the log rotation issue.
This issue is fixed in 6.12.1 release version.
Below is the is workaround which needs to be executed on all the Aria Operations for Networks affected node(s) (Platfrom(s) and collector(s), prior to applying the patch or upgrading to version 6.12.
1. Take a Putty/SSH session to Aria Operations for Networks affected node(s) (Platfrom(s) and collector(s)
2. Login with username support
3. Execute below command to switch to ubuntu user.
ub
4. You will need to run these commands on each node that is effected.
cd /var/log
To identify the size of the files such as warn,syslog.1 and auth.log.1
on the affected nodes
ls -lrth
5. Look at the last 3 to 4 files, eg as below:
-rw-r----- 1 syslog adm 2.5G Jul 19 16:19 warn
-rw-r----- 1 syslog adm 4.6G Jul 19 16:19 syslog
-rw-r----- 1 syslog adm 3.9G Jul 19 16:19 auth.log
6. Execute below command to rotate the logs manually
sudo dd if=/dev/null of=/var/log/secure
sudo dd if=/dev/null of=/var/log/warn
sudo dd if=/dev/null of=/var/log/syslog.1
sudo dd if=/dev/null of=/var/log/auth.log.1
7. Clean up the warn,syslog.1 and auth.log.1
on the affected nodes
To do so execute below commands:
sudo rm -rf warn
sudo rm -rf syslog.1
sudo rm -rf auth.log.1
8. Delete access.log, access.1.log
from /var/log/nginx
on all the nodes.
To do so execute below commands:
sudo su
cd nginx/
ls --lrth
sudo rm -rf access.log
sudo rm -rf access.log.1
9. Restart the syslog
and ngnix
service on all the nodes, execute below commands:
systemctl restart syslog.service
systemctl restart nginx.service
10. Post executing above mentioned steps, execute below command to validate the size of /var/log
using below command:
df -f
If there is a cluster setup then execute below command:
./run_all.sh df -h
The size of /var/log
should not show less than 60%
11. Now the P2 patch can now be applied to Aria Operations for Networks 6.11 as per steps mentioned in Resolution section above. If you are upgrading to Aria operations for Networks to version 6.12.0 or 6.12.1 then you can ignore to apply the P2 patch.
12. Download Aria Operations for Networks Version 6.11 P2 patch from the attachment section in this Knowledge Base Article.
File Name: VMware-AriaOpNetworks.6.11.0.P2.1699450092.patch.bundle
File Size: 668.2 MiB
CheckSums Values:
Note:
If the 6.11.0-P1 patch is already applied there is no need for this work around but if P2 patch is the first patch to be applied on top of 6.11 GA version then use the following steps:
ub
sudo su
mkdir -p /usr/local/lib/python3.6/dist-packages/cli
ln -sf /usr/local/lib/python3.8/dist-packages/cli/tool_manager_runner.py /usr/local/lib/python3.6/dist-packages/cli/tool_manager_runner.py
3. Upload patch bundle from Aria Operations for Networks GUI.