vmafdd.log is not being compressed which eventually leads to "log disk exhaustion" warning on the vCenter

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• You see a "log disk exhaustion" warning on the vCenter
• Checking storage logs for disk utilization you find the vmafdd.log is extremely large.

Example:

sudo du -ah /storage/log/ | sort -h -r | head -n 20

7.0G    /storage/log/vmware
7.0G    /storage/log/
1.4G    /storage/log/vmware/vmafd/vmafdd.log
1.4G    /storage/log/vmware/vmafd
768M    /storage/log/vmware/lookupsvc
762M    /storage/log/vmware/sso
757M    /storage/log/vmware/eam

 

• Additionally you may see the following messages in the vmafdd log:

  YYYY-MM-DDTHH:MM:SS [vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/ssl/certs/d70df8d2.0, so will not write again.
  YYYY-MM-DDTHH:MM:SS [vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/ssl/certs/6349821d.0, so will not write again. 
  YYYY-MM-DDTHH:MM:SS[vmafdd][INFO] VecsSrvWriteCertOrCrlToDisk: cert/cl already exists as /etc/vmware-vpx/docRoot/certs/ d70df8d2.0, so will not write again. 
  YYYY-MM-DDTHH:MM:SS [vmafdd][INFO] VmAfdProcessCACerts: force flushing.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

This happens because the logrotate.d configuration for the vmafd service expects the logs to be located in the following path:

Expected path	/var/log/vmware/vmafdd/vmafdd.log
Current path	/var/log/vmware/vmafd/vmafdd.log

# cat /etc/logrotate.d/vmware-vmafd.lr

/var/log/vmware/vmafdd/vmafdd.log <<< Here's where the log file is expected.
{
    size 10M
    nodateext
    rotate 10
    missingok
    notifempty
    compress
    create 644 root root
    postrotate
        /usr/sbin/killproc -L -HUP /usr/lib/vmware-vmafd/sbin/vmafdd
    endscript
}

Checking the vmafdd registry you see the incorrect path:

/opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"

root@vcsa1 [ ~ ]# /opt/likewise/bin/lwregshell list_values "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]"
+  "CAPath"                  REG_SZ          "/etc/ssl/certs"
+  "DCName"                  REG_SZ          "{vCenter's FQDN}"
+  "DCPort"                  REG_DWORD       0x000001bb (443)
+  "DomainGUID"              REG_SZ          "########-####-####-####-########2496"
+  "DomainName"              REG_SZ          "vsphere.local"
+  "DomainState"             REG_DWORD       0x00000001 (1)
+  "LDU"                     REG_SZ          "########-####-####-####-########e651"
+  "LogFile"                 REG_SZ          "/var/log/vmware/vmafd/vmafdd.log"

Resolution

This issue is resolved in the following versions of vCenter Server.

• 7.0 Update 3o (7.0.3.01700)
• 8.0 Update 1c (8.0.1.00300)

Workaround:

Note: Please ensure to create a fresh backup or offline snapshot (in powered off state) of the vCenter Server Appliance before implementing the steps below. In case the affected vCenter Server Appliance is part of an Enhanced Linked Mode (ELM) replication group, the backup or offline snapshots must be created for all of its replication partners as well. When restoring an ELM vCenter, all members of the ELM replication need to be restored too, otherwise there will be inconstencies in the VMDirectory LDAP database.

To work around this issue, please apply the following steps:

1. Update the registry to match the log rotate configuration path, using the following command:

   # /opt/likewise/bin/lwregshell set_value "[HKEY_THIS_MACHINE\Services\vmafd\Parameters]" "LogFile" "/var/log/vmware/vmafdd/vmafdd.log"

2. Restart all services:

   # service-control --stop --all && service-control --start --all

3. Afterwards you should move the old log 

   /storage/log/vmware/vmafd/vmafdd.log

   out of the /storage/log partition or delete it to reduce storage utilization in the log partition.