VCFA UI unavailable due to a full partition/PVC mount

Products

• VCF Automation
• VMware Cloud Foundation

Issue/Introduction

• VCF Auto user interface / webpage is down
• Running df -h on the VCFA VM as the root user returns a mount at 100% similar to:
  • /dev/sdf         50G   50G   48K 100% /var/lib/kubelet/pods/########-####-####-####-############/volumes/kubernetes.io~csi/pvc-########-####-####-####-############/mount

Environment

VCFA: 9.1

VCF: 9.1

Cause

Currently under investigation by VMware engineering.

Resolution

Workaround

Increase the CAPACITY of the postgresql PVC mount

1. Edit the prelude postgresql pod

   • kubectl edit postgresql vcfapostgres -n prelude

2. Locate the env volume size entry under -env:

   1. /- env

   2. /size

3. Enter the size by 20 Gi and save/exit the config file:

   1. i (enters insert mode)

   2. Edit the current size: value by increasing the value by 20 Gi

   3. <esc>:wq! (save and quit)

4. Restart the zalando-postgres-operator deployment

   • kubectl rollout restart deployment/zalando-postgres-operator -n vmsp-platform

5. Confirm the CAPACITY of the PVC reflects the new size:

   • kubectl get pvc -n prelude -l application=spilo

6. The VCFA UI should become available following the restart of the deployment.  The associated table should clean itself up after 24 hours.

• Note: The storage space used to increase the PVC cannot be reclaimed. 

Additional Information

Confirm the issue is with the postgres mount:

1. SSH into the VCFA appliance as the vmware-system-user

2. Switch to the root user

   • sudo -i

3. Identify the postgres PVC

   • kubectl get pvc -A | grep postgres

   • Example:

     prelude    pgdata-vcfapostgres-0    Bound pvc-########-####-####-####-############    30Gi    RWO    vmsp-default    111d

4. Confirm this is the PVC that is full

   • df -h | grep pvc-208849e9-01f2-467a-948c-0b7875b5730d

   • example results:

     /dev/sdf         50G   50G   48K 100% /var/lib/kubelet/pods/########-####-####-####-############/volumes/kubernetes.io~csi/pvc-########-####-####-####-############/mount

• If this partition is 100% full, proceed with the Resolution listed above.

• If not, proceed with the next steps to identify the mount, POD UUID, and PVC UUID details and use KB Creating and managing Broadcom cases to open a support case to investigate further:

1. Using the output:

   • Example:

     /dev/sdf         50G   50G   48K 100% /var/lib/kubelet/pods/########-####-####-####-############/volumes/kubernetes.io~csi/pvc-########-####-####-####-############/mount

2. Find the namespace for the PVC:

   • kubectl get pvc -A | grep pvc-########-####-####-####-############

3. Find the name of the pod using the PVC

   • kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.metadata.uid}{"\n"}{end}' | grep ########-####-####-####-############

   • Example:

     prelude vcfapostgres-0  ########-####-####-####-############
i.e <namespace> <podname> ########-####-####-####-############

4. Find the mount which is filling up the PVC:

   • kubectl exec -it -n <namespace> <podname> -- df -h

5. Look for large files/logs on the mount:

   • kubectl exec -it -n <namespace> <podname> -- du -sh