vCenter Appliance /storage/log partition full due to excessive pod-startup.log files

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• The vmware-pod service will not start.
• /var/log/vmware/pod contains thousands of pod-startup.log.###### files
• The /var/log/vmware/pod/pod-console.log contains the following errors:

2020-08-18T 7:01:01+0000 [__main__#INFO] Starting twistd service
2020-08-18T 7:01:04+0000 [__main__#ERROR] Got error while starting twistd service: Specified chain file '/etc/vmware/vmware-vmafd/machine-ssl.crt' doesn't contain any valid certificates in PEM format.

Environment

VMware vCenter Server 7.0.x

Cause

This issue occurs when the pod service repeatedly attempts to start but is not able to complete the process.

One known reason for this is that the service is unable to validate the certificate located in /etc/vmware/vmware-vmafd/machine-ssl.crt. This should be apparent by investigating the pod-console.log

Resolution

VMware is aware of this issue and working towards a fix.

Workaround:

To workaround the issue, resolve the problem which is stopping the vmware-pod service from starting.

In the case of certificate validation failing on  /etc/vmware/vmware-vmafd/machine-ssl.crt, there may be an issue with the certificate format, where the file contains "^M" characters after each line. Back up the certificate file and use a file editor such as VIM to remove the extra characters.

1. create a backup copy of the machine-ssl.crt file:

# cp -p /etc/vmware/vmware-vmafd/machine-ssl.crt /etc/vmware/vmware-vmafd/orig-machine-ssl.crt.bak

 

2. Edit the file and remove all ^M characters:

# vi /etc/vmware/vmware-vmafd/machine-ssl.crt
 

3. Save the file

Additional Information

The vmware-pod service is not required to run vCenter Server in "on-premise" deployments. It can be disabled temporarily while troubleshooting. To disable the service, run the following command:
 

# systemctl disable vmware-pod.service
 

To enable the service again, run:
 

# systemctl enable vmware-pod.service
 

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full