VMware Cloud Director root ("/") partition becomes full due to large /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log.

Products

• VMware Cloud Director

Issue/Introduction

• VMware Cloud Director root ("/") partition is full or temporarily full particularly during support bundle generation.
• A vmware-vcd service may stop unexpectedly.
• Generating a support bundle fails.
• /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log consume a large amount of the "/" partition.
  • To confirm this, list the 200 largest files and directories in the "/" partition by the command below.
    $ du --time -aShx / | sort -rh | head -200
    5.2G  20xx-xx-xx xx:xx        /opt/vmware/vcloud-director/logs
    2.8G  20xx-xx-xx xx:xx        /var/log/journal/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    2.8G  20xx-xx-xx xx:xx        /opt/vmware/var/log/vcd
    1.5G  20xx-xx-xx xx:xx        /var/log
    1.2G  20xx-xx-xx xx:xx        /var/log/nginx/access.log
    1.2G  20xx-xx-xx xx:xx        /var/log/nginx
    1007M 20xx-xx-xx xx:xx        /var/log/wtmp
    622M  20xx-xx-xx xx:xx        /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log-20210510
    (snip)
• APIs starting with "/api/1.0.0" are frequently called using 3rd party tools for purposes such as service health monitoring.

Environment

VMware Cloud Director 10.5
VMware Cloud Director 10.6

Cause

The API requests are logged in /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log and /var/log/nginx/access.log, but these logs are not rotated properly due to improper configuration.
So, in environments where the APIs are called frequently, the size of the logs may increase.

Resolution

This issue is resolved in VMware Cloud Director 10.6.1.2 or later.
To workaround this issue, follow the steps below:

1. /opt/vmware/var/log/vcd/vcd_ova_ui_uwsgi.log
   1. To correct the log file that a vcd_ova_ui service is writing, restart the service by the command below.
      systemctl restart vcd_ova_ui
      Note : During the service restart, a management UI connected via port 5480 is unavailable and API calls starting with "/api/1.0.0" fail.
   2. Modify /etc/logrotate.d/vcd-applyance-sync as follows.
      ----------------------------------------------------------------------------------------
      /opt/vmware/var/log/vcd/*.log {
              su vcloud vcloud
              daily
              minsize 10M <-- Replace "size" with "minsize"
              create 0664 vcloud vcloud
              rotate 10
              nocompress
              prerotate
                  /usr/bin/chown -R vcloud:vcloud /opt/vmware/var/log/vcd/
                  /usr/bin/chmod -R 0664 /opt/vmware/var/log/vcd/*
              endscript
              copytruncate <-- Add this line
      }
      ----------------------------------------------------------------------------------------
2. /var/log/nginx/access.log
   Create /etc/logrotate.d/nginx with the following content.
   No further action, such as service restart, isn't required.
   ----------------------------------------------------------------------------------------
   /var/log/nginx/*.log {
           daily
           minsize 10M
           rotate 10
           compress
           copytruncate
   }
   ----------------------------------------------------------------------------------------