Increasing Debug Log Retention Time in VMware Cloud Director

Products

• VMware Cloud Director

Issue/Introduction

The default logging setup for Cloud Director is 10 files at 10MB each. Depending on how busy your environment is, these logs can roll over very quickly. Industry best practice is to send your logs to a Syslog server, VRLI or a o11y tool such as Tanzu Observability.

The other option is to increase the size and number of files that the system logs. Please note that this will increase the size of the files so ensure you have enough capacity on the shared NFS space.

 

Environment

All Cloud Director versions.

Resolution

The location of the file that you need to edit is:

/opt/vmware/vcloud-director/etc/log4j.properties

We'll focus on the vcloud-container-debug.log block in that file as the debug level log files are the most important piece of evidence a TSE needs when triaging a case.

This is the default:

# Component appender for container debug
log4j.appender.vcloud.system.debug=org.apache.log4j.RollingFileAppender
log4j.appender.vcloud.system.debug.File=logs/vcloud-container-debug.log
log4j.appender.vcloud.system.debug.MaxFileSize=10240KB
log4j.appender.vcloud.system.debug.MaxBackupIndex=9
log4j.appender.vcloud.system.debug.layout=com.vmware.vcloud.logging.layout.CustomPatternLayout
log4j.appender.vcloud.system.debug.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.75t | %-30.50c{1} | %m | %x%n
log4j.appender.vcloud.system.debug.threshold=DEBUG

What does this all mean?

Appender Name and Type

• log4j.appender.vcloud.system.debug - is the name of the appender.
• org.apache.log4j.RollingFileAppender - is the type of appender, in this case this is a rolling file appender.

File Settings

• log4j.appender.vcloud.system.debug.File=logs/vcloud-container-debug.log - specifies the file name and location for the log file. In this case, it's logs/vcloud-container-debug.log under $VCLOUD_HOME
• log4j.appender.vcloud.system.debug.MaxFileSize=10240KB - sets the maximum size of the log file to 10,240 KB, or 10MB. When the log reaches this size, it will be rolled over i.e., a new file will be created with the same name, but with a incremented numerical suffix.
• log4j.appender.vcloud.system.debug.MaxBackupIndex=9 - sets the maximum number of backup files to keep. When the maximum file size is reached, the oldest file will be deleted, up to a maximum of 9 files.

Layout and Pattern

• log4j.appender.vcloud.system.debug.layout=com.vmware.vcloud.logging.layout.CustomPatternLayout - specifies the layout for the log messages. In this case, it's a custom layout provided by com.vmware.vcloud.logging.layout.
• log4j.appender.vcloud.system.debug.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.75t | %-30.50c{1} | %m | %x%n - specifies the pattern for formatting the log messages.

We'll break this line down below.

Pattern Breakdown

• %d{ISO8601}: Format the date and time using the ISO 8601 format (e.g., 2023-03-17T14:30:00.000Z)
• %-8.8p: Format the log level (e.g., DEBUG, INFO, WARNING, etc.) using a minimum width of 8 characters and a maximum width of 8 characters. The minus sign means that the value will be padded with spaces on the left if it's shorter than 8 characters.
• %-25.75t: Format the thread name using a minimum width of 25 characters and a maximum width of 25 characters. The minus sign means that the value will be padded with spaces on the left if it's shorter than 25 characters.
• %-30.50c{1}: Format the logger name using a minimum width of 30 characters and a maximum width of 30 characters. The minus sign means that the value will be padded with spaces on the left if it's shorter than 30 characters. The {1} specifies that the logger name should be truncated to the first level of nesting.
• %m: Format the log message itself.
• %x: Format the mapped diagnostic context information.
• %n: Format a newline character.

Threshold

• log4j.appender.vcloud.system.debug.threshold=DEBUG - sets the threshold level for this appender to DEBUG. This means that only log messages with a level of DEBUG or higher will be logged to this appender.

Figuring out how many files you need

Let's say you want to retain 72 hours of activity in the logs, given that the default logging level is good for 8 hours.

10 logs files at 10MB each cover an 8-hour period, which means:

• The total log size for 8 hours is 10 files * 10MB/file = 100MB
• The log size per hour is 100MB / 8 hours = 12.5MB per hour

Since the logs are capped at 10MB, we can assume that the actual log size per hour is approximately 10MB.

To cover a 72-hour period, we need to calculate the total log size required:

• 72 hours * 10MB per hour = 720MB

We want to store the logs in 50MB files. To calculate how many files we need, we can divide the total log size by the file size:

• 720MB (total log size) ÷ 50MB (file size) = 14.4 files

Round up to the nearest whole number. In this case we would need 15 files, each 50MB in size to cover a 72-hour period plus an extra 5 for good measure.

Editing the log4j.properties file

Ensure you have a backup of the default settings.

Open the file in your favourite editor and look for these lines:

log4j.appender.vcloud.system.debug.MaxFileSize=10240KB
log4j.appender.vcloud.system.debug.MaxBackupIndex=9

Replace them with:

log4j.appender.vcloud.system.debug.MaxFileSize=51200KB
log4j.appender.vcloud.system.debug.MaxBackupIndex=20

Perform the same steps on all Cloud Director appliances and then restart the vmware-vcd service:

service vmware-vcd restart