VMware Cloud Director root ("/") partition becomes full due to large /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log.

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director root ("/") partition is full or temporarily full particularly during support bundle generation.
• A vmware-vcd service may stop unexpectedly.
• Generating a support bundle fails.
• /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log consume a large amount of the "/" partition.
  • To confirm this, list the 200 largest files and directories in the "/" partition by the command below.
    
    $ du --time -aShx / | sort -rh | head -200
5.2G  20xx-xx-xx xx:xx        /opt/vmware/vcloud-director/logs
2.8G  20xx-xx-xx xx:xx        /var/log/journal/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2.8G  20xx-xx-xx xx:xx        /opt/vmware/var/log/vcd
1.5G  20xx-xx-xx xx:xx        /var/log
1.2G  20xx-xx-xx xx:xx        /var/log/nginx/access.log
1.2G  20xx-xx-xx xx:xx        /var/log/nginx
1007M 20xx-xx-xx xx:xx        /var/log/wtmp
622M  20xx-xx-xx xx:xx        /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log-20210510
    
    
• /var/log/nginx/error.log may also consume a large amount of the "/" partition.
• APIs starting with "/api/1.0.0" are frequently called using 3rd party tools for purposes such as service health monitoring.

Environment

• VMware Cloud Director 10.5
• VMware Cloud Director 10.6

Cause

The API requests are logged in /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log, but these logs are not rotated properly due to improper configuration.
So, in environments where the APIs are called frequently, the size of the logs may increase.

Resolution

This issue is resolved in VMware Cloud Director 10.6.1.2 or later. VMware Cloud Director 10.6.1.2 Release Notes

To workaround this issue, follow the steps below:

1. /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log
   1. To correct the log file that a vcd_ova_ui service is writing, restart the service by the command below.
      
      systemctl restart vcd_ova_ui
      
      Note : During the service restart, a management UI connected via port 5480 is unavailable and API calls starting with "/api/1.0.0" fail.
   2. Modify /etc/logrotate.d/vcd-applyance-sync as follows.
      ----------------------------------------------------------------------------------------
      /opt/vmware/var/log/vcd/*.log {
        su vcloud vcloud
        daily
        minsize 10M <-- Replace "size" with "minsize"
        create 0664 vcloud vcloud
        rotate 10
        nocompress
        prerotate
            /usr/bin/chown -R vcloud:vcloud /opt/vmware/var/log/vcd/
            /usr/bin/chmod -R 0664 /opt/vmware/var/log/vcd/*
        endscript
        copytruncate <-- Add this line
}
      ----------------------------------------------------------------------------------------
2. /var/log/nginx/access.log

1. 1. Create /etc/logrotate.d/nginx with the following content.
      No further action, such as service restart, isn't required.
      ----------------------------------------------------------------------------------------
      /var/log/nginx/*.log {
        daily
        minsize 10M
        rotate 10
        compress
        copytruncate
}
      ----------------------------------------------------------------------------------------