High Disk Usage in /var/log
Article ID: 323340
Updated On:
Products
VCF Operations for Networks
VCF Operations/Automation (formerly VMware Aria Suite)
Issue/Introduction
- Platform or Collector VM is running long, for example more than 3 months
/var/logdisk usage goes 90% or higheraccess.log.1file is large or empty
root@vrni-platform-release:/var/log/nginx# ls -l .
total 32792
-rw-r----- 1 www-data adm 964 Aug 2 08:55 access.log
-rw-r--r-- 1 root root 33570734 Aug 2 08:55 access.log.1
-rw-r----- 1 www-data adm 0 Aug 2 08:55 error.log
-rw-r--r-- 1 root root 221 Aug 2 08:54 error.log.1
total 32792
-rw-r----- 1 www-data adm 964 Aug 2 08:55 access.log
-rw-r--r-- 1 root root 33570734 Aug 2 08:55 access.log.1
-rw-r----- 1 www-data adm 0 Aug 2 08:55 error.log
-rw-r--r-- 1 root root 221 Aug 2 08:54 error.log.1
- nginx worker processes write to old access.log.1 after logrotation
root@vrni-platform-release:/var/log/nginx# lsof -c nginx|grep access.log
nginx 45587 root 4w REG 253,2 0 524292 /var/log/nginx/access.log
nginx 45588 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45589 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45590 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45591 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45592 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45587 root 4w REG 253,2 0 524292 /var/log/nginx/access.log
nginx 45588 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45589 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45590 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45591 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45592 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
- You may see auth.log.1 or syslog.1 files in large size
root@aria-networks-collector:~# sudo du -aBM /var/log | sort -nr | head -n1027223M /var/log11563M /var/log/warn6729M /var/log/auth.log.12929M /var/log/journal/641###########################db22929M /var/log/journal2775M /var/log/syslog.12526M /var/log/arkin795M /var/log/arkin/flow-processor602M /var/log/arkin/collector446M /var/log/arkin/launcher
Environment
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.x
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.11.0
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.10.0
VMware vRealize Network Insight 6.x
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.11.0
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.10.0
VMware vRealize Network Insight 6.x
Cause
nginx worker processes fail to write to access.log after logrotation and keep writing to old access.log.1.
Resolution
This issue is resolved in Aria Operations for Networks 6.12.
Workaround:
To workaround the issue, run following commands to restart nginx service.sudo service nginx stopsudo service nginx start
Additional Information
If the above steps in this KB do not resolve the issue, raise a support ticket with Broadcom support selecting Aria operations for network as the product.
Handling Log Bundles for offline review with Broadcom support.
Feedback
Yes
No
Powered by
