High Disk Usage in /var/log

Products

• VMware Aria Operations for Networks
• VMware Aria Suite

Issue/Introduction

Symptoms:

• Platform or Collector VM is running long, for example more than 3 months.
• /var/log disk usage goes 90% or higher.
• access.log.1 is large or empty.​​​​​​

root@vrni-platform-release:/var/log/nginx# ls -l .
total 32792
-rw-r----- 1 www-data adm 964 Aug 2 08:55 access.log
-rw-r--r-- 1 root root 33570734 Aug 2 08:55 access.log.1
-rw-r----- 1 www-data adm 0 Aug 2 08:55 error.log
-rw-r--r-- 1 root root 221 Aug 2 08:54 error.log.1

 

• nginx worker processes write to old access.log.1 after logrotation.

root@vrni-platform-release:/var/log/nginx# lsof -c nginx|grep access.log
nginx 45587 root 4w REG 253,2 0 524292 /var/log/nginx/access.log
nginx 45588 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45589 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45590 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45591 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
nginx 45592 www-data 4w REG 253,2 34581691 524291 /var/log/nginx/access.log.1
 

Environment

VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.x
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.11.0
VMware Aria Operations for Networks (formerly vRealize Network Insight) 6.10.0
VMware vRealize Network Insight 6.x

Cause

nginx worker processes fail to write to access.log after logrotation and keep writing to old access.log.1.

Resolution

This issue is resolved in Aria Operations for Networks 6.12.

Workaround:
Run following commands to restart nginx service.
sudo service nginx stop
sudo service nginx start