Internal Server Error while accessing VCF Management page in VCF Operations UI

Products

• VCF Operations

Issue/Introduction

• VCF Management page under Operations Manager > Build > Lifecycle fails to load with below error

The server sent back an error: 500 Internal Server Error

• vcf-fleet-build or vcf-fleet-upgrade pods are in CrashLoopBackOff (kubectl get pods -n vcf-fleet-lcm)
• One or more vcf-fleet-lcm-db pods are in Containercreating status
• vcf-fleet-lcm-db pod logs report below error (kubectl logs vcf-fleet-lcm-db-0 -n vcf-fleet-lcm | grep -i "No space left on device" )

FATAL: could not create lock file "postmaster.pid:. No space left on device

Environment

• VMware Cloud Foundation (VCF) 9.1
• VCF Operations 9.1

Cause

 Insufficient storage capacity in the Persistent Volume Claim (PVC) assigned to the vcf-fleet-lcm database pods.

Resolution

1.Verify Error: Check vcf-fleet-lcm pod logs to confirm the "no space left on the device" error.

1. Connect to VMSP control plane node using SSH and log in as vmware-system-user
2. Elevate your privileges to root using sudo -i
3. Identity the fleet lcm database pod which is in Containercreating status using kubectl get pods -n vcf-fleet-lcm
4. Review the pod logs using kubectl logs vcf-fleet-lcm-db-# -n vcf-fleet-lcm | grep -i "No space left on device"

Note: vcf-fleet-lcm-db-# will be the database pod which has a Containercreating status

2.Increase Storage: 

1. Retrieve the pvc details of vcf-fleet-lcm using kubectl get pvc -n vcf-fleet-lcm
2. Expand the pvc size of all the vcf-fleet-lcm database pods to 90GB using 

kubectl patch pvc <pvc-name> -n  vcf-fleet-lcm -p '{"spec":{"resources":{"requests":{"storage":"90Gi"}}}}'

Note: The size of all the pvc's in vcf-fleet-lcm needs to be expanded. Replace <pvc-name> with the PVC identifier found using kubectl get pvc -n vcf-fleet-lcm

3.Reinitialize Pods: 

• Reinitialize the pods which are in  CrashLoopBackOff or  Containercreating using kubectl delete pod <pod-name> -n vcf-fleet-lcm

4.Validate: 

• Confirm that all the vcf-fleet-lcm pods are in a running status using kubectl get pods -n vcf-fleet-lcm
• Verify that the VCF Management UI page loads successfully