Unable to access the VCF Automation Portal: multiple pods evicted or pending due to high root partition usage

Products

• VCF Automation

Issue/Introduction

• Unable to access the VCF Automation Portal.
  
  
• 'kubectl get pods -A'  shows multiple pods in Evicted or Pending state.
  
  
• 'df -h /' shows the root partition usage is 88% or higher.
  
  

Environment

VCF Automation 9.0.0.0

Cause

Pods are not running properly due to DiskPressure on the nodes.

 

Resolution

Check large files on the root partition:

Example:

du -a -x -h / | sort -rh | head -100

If logs under /var/log/vmsp-logs/systemd/ are large, vacuum the journal and truncate files:

First, truncate journal size

journalctl --vacuum-size=500M

Then, truncate output logs

truncate -s 0 /var/log/vmsp-logs/systemd/<file-name>



If audit logs under /var/log/vmsp-logs/apiserver-audit/ are large:

First, truncate apiserver-audit logs

truncate -s 0 /var/log/apiserver-audit/<file-name>.log

Then, truncate files under vmsp-logs

truncate -s 0 /var/log/vmsp-logs/apiserver-audit/<file-name>.log