Error: "No Healthy Upstream" while accessing VCF Automation user interface due to partition full.

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

VCF Automation becomes inaccessible and reports a "no healthy upstream" error.

• /dev/sdg partition used for the database, reaches 100% capacity.
  /dev/sdg 65G 65G 96K 100% /var/lib/kubelet/pods/[pod-id]/volumes/kubernetes.io~csi/[pvc-id]/mount
• VCF Automation VMs were moved to different folder than where they were originally deployed.
• Attempting to resize disk fails with following error:
  

  com. vmware.vrealize.lcm.vmsp. common.exception.VmspUpdateStorageVolumesException: Storage re-size failed for ycfa-bundle with error: Failed to re-size storage. Output:
  {"volumes":[{"capacity":{"key":"storage.volumeGroups.pgdataV/cfapostgres.capacity","value":"######"}}]}.
  at com.vmware.vrealize.lcm.vmsp.plugin.util. VMSPDay2Util. updateStorageVolumes (VMSPDay2Util. java: 228)
  at com. vmware.vrealize. lcm.vmsp.plugin. tasks. VmspUpdateStorageVolumeTask. execute(VmspUpdateStorageVolumeTask. java: 58)
  at com.vmware.vrealize.lcm.automata.core.TaskThread.cup(TaskThread.java:62)
  at java.base/java.util.concurrent.ThreadPoolExecutor.cunWorker(Unknown Source)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.cun(Unknown Source)
  at java.base/java.lang.Thread.cun(Unknown Source)

Environment

VCF Automation 9.x

Cause

The VCF Automation VMs were moved to a different folder, rendering the UI inaccessible. As this configuration is unsupported according to the Tech Doc it triggered a series of cascading failures.

Resolution

Move the VCF Automation machines back to the folder where machines were originally deployed 

1. Log in to SSH session of the VCF Automation.
2. Execute the following command to patch the storage capacity for the pgdataVcfapostgres volume group. Replace 101Gi with your new desired capacity:
   Note: If increasing disk space was tried using vCenter, ensure to add +1 GB to the capacity it was increased by and replace 101Gi with new capacity.
   kubectl patch pd vcfa-bundle -n prelude --type=merge -p '{"spec":{"values":{"storage":{"volumeGroups":{"pgdataVcfapostgres":{"capacity":"101Gi"}}}}}}'
3. Monitor the update progress to ensure the package deployment is successful
   kubectl describe pd vcfa-bundle -n prelude
4. If a database node (e.g., vcfapostgres-0) remains in unhealthy state follow KB: How to remediate an unhealthy postgres cluster.