The vCenter logs partition is full because /var/log/ is consuming most of the space.

Products

• VMware vCenter Server

Issue/Introduction

• The vCenter Server reached capacity because the /var/log/messages file had grown to several gigabytes in size.
• A review of the df -h output shows that the /var/log/ partition is at capacity.

df -h

Example ouptut:
Filesystem                                   Size  Used Avail Use% Mounted on
devtmpfs                                     4.0M     0  4.0M   0% /dev
tmpfs                                         11G  980K   11G   1% /dev/shm
tmpfs                                        4.1G   12M  4.1G   1% /run
tmpfs                                        4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0               47G   24G   23G  54% /
tmpfs                                         11G  3.0M   11G   1% /tmp
/dev/mapper/autodeploy_vg-autodeploy         9.8G   40K  9.3G   1% /storage/autodeploy
/dev/mapper/lifecycle_vg-lifecycle            98G  4.1G   89G   5% /storage/lifecycle
/dev/mapper/core_vg-core                      49G   40K   47G   1% /storage/core
/dev/mapper/vg_lvm_snapshot-lv_lvm_snapshot  196G   28K  186G   1% /storage/lvm_snapshot
/dev/mapper/netdump_vg-netdump               982M  108M  807M  12% /storage/netdump
/dev/mapper/db_vg-db                         9.8G  205M  9.1G   3% /storage/db
/dev/mapper/imagebuilder_vg-imagebuilder     9.8G   36K  9.3G   1% /storage/imagebuilder
/dev/mapper/dblog_vg-dblog                    15G  129M   14G   1% /storage/dblog
/dev/sda3                                    488M   60M  392M  14% /boot
/dev/mapper/vtsdblog_vg-vtsdblog              15G   33M   14G   1% /storage/vtsdblog
/dev/sda2                                     10M  2.1M  7.9M  22% /boot/efi
/dev/mapper/vtsdb_vg-vtsdb                    25G   36M   24G   1% /storage/vtsdb
/dev/mapper/updatemgr_vg-updatemgr            98G  990M   92G   2% /storage/updatemgr
/dev/mapper/seat_vg-seat                      25G  441M   23G   2% /storage/seat
/dev/mapper/archive_vg-archive                49G  2.5G   44G   6% /storage/archive
/dev/mapper/log_vg-log                        10G  10G   0G   100% /storage/log

• Run the command below to list the 10 largest files on the system.
  
  du -ahx . | sort -rh | head -10

Example ouptut:
2234M   /var/log/messages
98M    /var/log/vmware/vpxd.log
85M    /var/core/core.12345
72M    /var/log/vmware/sps.log
65M    /var/log/vmware/vmware-vpx/vpxd-profiler.log
54M    /var/log/vmware/vmware-sps/sps-profiler.lo

Environment

vCenter Server 8.0 Update 3g

Cause

This issue occurs because the /etc/logrotate.d/syslog configuration file is missing from vCenter Server. Without this file, log rotation does not take place, causing the /var/log/messages file to grow uncontrollably and eventually consume available partition space.

Resolution

To resolve this issue, recreate the missing logrotate configuration file for syslog.

1. SSH to vCenter server.
2. Create the missing file using following cmd,
   
   vi /etc/logrotate.d/syslog
   
   
3. Add the following configuration content in text editor.
   
   /var/log/warn /var/log/messages /var/log/mail /var/log/mail.info /var/log/mail.warn /var/log/mail.err {
    compress
    nodateext
    size 5M
    rotate 15
    missingok
    notifempty
    create 640 root root
    sharedscripts
    postrotate
        /usr/bin/systemctl restart syslog.service > /dev/null
    endscript
}
/var/log/cron /var/log/auth.log /var/log/faillog /var/log/tallylog {
    compress
    nodateext
    size 5M
    rotate 15
    missingok
    notifempty
    create 600 root root
    sharedscripts
    postrotate
        /usr/bin/systemctl restart syslog.service > /dev/null
    endscript
}
/var/log/lastlog {
    compress
    nodateext
    size 5M
    rotate 15
    missingok
    notifempty
    create 640 root tty
    sharedscripts
    postrotate
        /usr/bin/systemctl restart syslog.service > /dev/null
    endscript
}
   
   
4. Once the configuration file has been created, execute this command to apply log rotation:
   
   logrotate -f -v /etc/logrotate.d/syslog