vCenter Appliance /storage/log partition full due to excessive pod-startup.log files

Products

• VMware vCenter Server 7.0

Issue/Introduction

Symptoms:

• The vmware-pod service will not start in the vCenter server.
• /var/log/vmware/pod/ contains thousands of pod-startup.log.###### files
• The /var/log/vmware/pod/pod-console.log contains the following errors:

YYYY-MM-SST HH:MM:SS [__main__#INFO] Starting twistd service
YYYY-MM-SST HH:MM:SS [__main__#ERROR] Got error while starting twistd service: Specified chain file '/etc/vmware/vmware-vmafd/machine-ssl.crt' doesn't contain any valid certificates in PEM format.

Environment

VMware vCenter Server 7.x

Cause

This issue occurs when the pod service repeatedly attempts to start but is not able to complete the process.

This is due to the service unable to validate the certificate located in /etc/vmware/vmware-vmafd/machine-ssl.crt. This should be apparent by investigating the pod-console.log

Resolution

This issue has been resolved in vCenter Server 7.0 Update 2.

Workaround:

If the issue is caused due to certificate validation failing in /etc/vmware/vmware-vmafd/machine-ssl.crt, there may be an issue with the certificate format, where the file contains "^M" characters after each line.
Hence, back up the certificate file and use a file editor such as VIM to remove the extra characters.

1. Create a backup copy of the machine-ssl.crt file:
   cp -p /etc/vmware/vmware-vmafd/machine-ssl.crt /etc/vmware/vmware-vmafd/orig-machine-ssl.crt.bak
   
   
2. Edit the file and remove all ^M characters:
   vi /etc/vmware/vmware-vmafd/machine-ssl.crt
   
   
3. Start the POD service
   service-control --start vmware-pod

Additional Information

The vmware-pod service is not required to run vCenter Server in "on-premise" deployments. It can be disabled temporarily while troubleshooting. To disable the service, run the following command:
 

# systemctl disable vmware-pod.service

Check the service status using:

# systemctl status vmware-pod.service
 

To enable the service again, run:
 

# systemctl enable vmware-pod.service
 

Troubleshooting vCenter Appliance /storage/log directory is 80% or more full