Upgrade for the vRO service is stale on CExP VA and no new versions of vRO service are showing

Products

• VMware Aria Suite

Issue/Introduction

This article provides steps to prevent an upgrade vRO job from failing continuously after an automatic vRO service upgrade initiated by vRA SaaS.

Symptoms:

• The CExP VA version should be older < 8.8 similar to the below

vracli version
Version - 8.4.0.16308 Build 17850259

• When checking the vRO service version on the CExP VA, it is too old or it is not the expected one, the VA version could be checked by

  curl -k https://<CExP-fqdn>/vco/api/about

• When navigating to the directory /data/upgrade/prelude and listing contents, you will see similar to data-<number>, and also pending-upgrade.flag file may be present

  root@es-sof-vc-vm-224-194 [/data/upgrade/prelude]# ls -l
  total 3228
  drwxr-xr-x 6 root root 4096 Sep 5 16:13 data
  drwxr-xr-x 6 root root 4096 Sep 9 15:12 data-1662736375
  drwxr-xr-x 6 root root 4096 Sep 17 12:13 data-1663416782
  drwxr-xr-x 6 root root 4096 Sep 17 16:42 data-1663432975
  drwxr-xr-x 6 root root 4096 Sep 19 21:12 data-1663621975
  drwxr-xr-x 6 root root 4096 Sep 20 10:25 data-1663669548
  drwxr-xr-x 6 root root 4096 Sep 21 14:12 data-1663769573
  drwxr-xr-x 6 root root 4096 Sep 22 07:13 data-1663830784
  drwxr-xr-x 6 root root 4096 Sep 22 23:45 data-1663890353
  drwxr-xr-x 6 root root 4096 Sep 29 13:12 data-1664457175
  drwxr-xr-x 6 root root 4096 Sep 30 02:11 data-1664503902
  drwxr-xr-x 6 root root 4096 Sep 30 04:22 data-1664511756
  drwxr-xr-x 6 root root 4096 Sep 30 11:12 data-1664536366
  drwxr-xr-x 6 root root 4096 Oct 1 08:12 data-1664611972
  drwxr-xr-x 6 root root 4096 Oct 3 13:57 data-1664805436
  rw-rr- 1 root root 0 Sep 5 16:13 pending-upgrade.flag
  rw-rr- 1 root root 3235489 Sep 5 16:42 prelude-upgrade.log
  rw-rr- 1 root root 161 Apr 5 2021 README

Cause

During automatic vRO service upgrade initiated by vRA SaaS, when

• an intermittent environment failure
• Non-responsive Kubernetes infrastructure services

or

• a manual intervention to a particular VA

occurs, the upgrade vRO job may fail continuously.

This job cannot be reattempted because of a temporary file indicating pending-upgrade is not deleted from the vRO upgrade folder.

Resolution

The issue is resolved in 8.8 and later versions of the Cloud Extensibility Proxy.

Workaround:

1. SSH / PuTTy into Cloud Extensibility proxy virtual appliance.
2. Ensure the vRO service on the VA is up and running by checking the following URL

    https://va-fqdn/vco/api/healthstatus

3. Manually trigger the update

   prelude-vami-upgrade –force