Scale out operation fails during deployment of cgs-ui-app
search cancel

Scale out operation fails during deployment of cgs-ui-app

book

Article ID: 325851

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

Symptoms:
  • Fresh deployments of 8.16.1 and 8.16.2 will not have the cgs-ui deployed
  • Upgraded environments from versions prior to 8.16.1 to 8.161 or 8.16.2 will still have the cgs-ui deployed
This discrepancy in behavior will lead to issues when an environment of the latter type is being scaled out or a node is being replaced.

Environment

VMware Aria Automation 8.16.x

Cause

The cgs-ui service has been recently removed and is unused post-upgrade but not removed by rpm during upgrade.

Resolution

This issue is resolved in versions 8.17.0 and above.

If you are upgrading from a version prior to 8.16.1 to 8.17.0 or later, do not apply the workaround.


Workaround:

Prerequisites

  • You have backups of the VMware Aria Automation 8.x appliance(s).
    • You must back up all VMware Aria Automation appliances, at the same time - simultaneously for all nodes.
    • If you are taking the snapshots manually, you must start the snapshots of the second and the third node no¬† more than 40 seconds after you start the snapshots for the first node.
    • When you back up the VMware Aria Automation appliance, disable in-memory snapshots and enable quiescing.

Procedure

  1. Before scaling-out or replacing a node, on the the existing upgraded node(s) where the cgs-ui still exists, run the following command:
    rpm -q prelude-cgs-ui && rpm -e prelude-cgs-ui
  2. Locate the node with the latest deploy.log (in case of an HA environment) either manually or by using the following command:
    vracli cluster exec -- bash -c "current_node; ls -l /var/log/deploy.log || true"
    1. SSH to this node, as it has the latest deploy.log, then run the following command to extract environment specific parameters to be used in the following steps:
      grep 'helm-upstall ui' /var/log/deploy.log | sed -e 's/+ helm-upstall/helm-upstall/g'
    2. Run the following command to navigate to the charts directory:
      cd /opt/charts
    3. Run the following commands:
      source /opt/scripts/helm_utils.sh
      InsertOutputfromStep2.1
  3. Proceed with the scale-out or node replacement operation.


Additional Information

Scale out VMware Aria Automation from one to three nodes

Replacing a VMware Aria Automation appliance node


Impact/Risks:
The cgs-ui pod will fail to come up when the environment is upgraded to 8.16.1 or 8.16.2 and a scale out is performed.

You may see the following errors in the journal or deploy.log:
  • Warning FailedMount 3m31s (x21 over 30m) kubelet MountVolume.Setup failed for volume "nginx-dist : hostPath type check failed: /opt/tarballs/content-gateway-ui is not a directory.