Scale out operation fails during deployment of cgs-ui-app

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Fresh deployments of 8.16.1 and 8.16.2 will not have the cgs-ui deployed
• Upgraded environments from versions prior to 8.16.1 to 8.161 or 8.16.2 will still have the cgs-ui deployed

This discrepancy in behavior will lead to issues when an environment of the latter type is being scaled out or a node is being replaced.

Environment

VMware Aria Automation 8.16.x

Cause

The cgs-ui service has been recently removed and is unused post-upgrade but not removed by rpm during upgrade.

Resolution

This issue is resolved in versions 8.17.0 and above.

If you are upgrading from a version prior to 8.16.1 to 8.17.0 or later, do not apply the workaround.

Workaround:

Prerequisites

• You have backups of the VMware Aria Automation 8.x appliance(s).
  • You must back up all VMware Aria Automation appliances, at the same time - simultaneously for all nodes.
  • If you are taking the snapshots manually, you must start the snapshots of the second and the third node no  more than 40 seconds after you start the snapshots for the first node.
  • When you back up the VMware Aria Automation appliance, disable in-memory snapshots and enable quiescing.

Procedure

1. Before scaling-out or replacing a node, on the the existing upgraded node(s) where the cgs-ui still exists, run the following command:

   rpm -q prelude-cgs-ui && rpm -e prelude-cgs-ui

2. Locate the node with the latest deploy.log (in case of an HA environment) either manually or by using the following command:

   vracli cluster exec -- bash -c "current_node; ls -l /var/log/deploy.log || true"

   1. SSH to this node, as it has the latest deploy.log, then run the following command to extract environment specific parameters to be used in the following steps:

      grep 'helm-upstall ui' /var/log/deploy.log | sed -e 's/+ helm-upstall/helm-upstall/g'

   2. Run the following command to navigate to the charts directory:

      cd /opt/charts

   3. Run the following commands:

      source /opt/scripts/helm_utils.sh
      InsertOutputfromStep2.1

3. Proceed with the scale-out or node replacement operation.

Additional Information

Scale out VMware Aria Automation from one to three nodes

Replacing a VMware Aria Automation appliance node

Impact/Risks:
The cgs-ui pod will fail to come up when the environment is upgraded to 8.16.1 or 8.16.2 and a scale out is performed.

You may see the following errors in the journal or deploy.log:

• Warning FailedMount 3m31s (x21 over 30m) kubelet MountVolume.Setup failed for volume "nginx-dist : hostPath type check failed: /opt/tarballs/content-gateway-ui is not a directory.