Upgrading Aria Automation Orchestrator from 8.18.1 U3/U4/U5 to VCF Operations Orchestrator 9.0.x fails with dependency errors

Products

• VCF Automation

Issue/Introduction

The following symptoms might be observed when upgrading VMware Aria Orchestrator:

• Upgrade from Aria Automation Orchestrator 8.18.1 U3 (8.18.1.36994),  U4 or U5 to VCF Operations Orchestrator 9.0.x fails.
• The /opt/vmware/var/log/vami/updatecli.log shows dependency errors such as:

    - libreadline.so.7()(64bit) is needed by (installed) socat-2.0.0.b9-3.ph4.x86_64

  [ERROR] Failed with exit code 65024

• The upgrade-report shows: 

  "VAMI has exited with error code: 1. Upgrade has failed."

Environment

VMware Aria Automation Orchestrator 8.18.1 Patch 3, Patch 4 or Patch 5

Cause

The Orchestrator VA upgrade engine relies on the source version being declared in the target 9.0.x build.
The 8.18.1 U3/U4/U5 patch releases have version numbers (e.g., 8.18.1.36994) that are not recognized as valid upgrade source versions by the 9.0.x upgrade package, causing the dependency check phase to fail.

Resolution

Workaround

The following steps should be executed to downgrade the reported version to the base
8.18.1 version, which allows the upgrade to proceed correctly:

1. Take a VM snapshot (without memory, with quiesced option selected) of all Orchestrator VMs.
   This can be done manually from vCenter.
2. SSH into one of the Orchestrator appliances as root.
3. Execute the following command to apply the patch on all cluster nodes:
   

   vracli cluster exec -- bash -c 'base64 -d <<< IyEvYmluL2Jhc2gKCiMgQ29weXJpZ2h0IChjKSAyMDI2IEJyb2FkY29tLiBBbGwgUmlnaHRzIFJlc2VydmVkLgojIEJyb2FkY29tIENvbmZpZGVudGlhbC4gVGhlIHRlcm0gIkJyb2FkY29tIiByZWZlcnMgdG8gQnJvYWRjb20gSW5jLgojIGFuZC9vciBpdHMgc3Vic2lkaWFyaWVzLgoKbG9nRGlyPSIvdmFyL2xvZy92bXdhcmUvcHJlbHVkZSIKdGltZXN0YW1wPSQoZGF0ZSAnKyVZLSVtLSVkLSVILSVNLSVTJykKcHI9IlZDT1BTLTY4MjU3IgpCQUNLVVBfRElSPSIvZGF0YS9wYXRjaC1iYWNrdXAvJHByLSR0aW1lc3RhbXAiCgpta2RpciAtcCAiJGxvZ0RpciIKZXhlYyAyPiA+KHRlZSAtYSAiJGxvZ0Rpci9wYXRjaC0kcHItJHRpbWVzdGFtcC5sb2ciKQpleGVjID4mMgoKaD0kKGhvc3RuYW1lKQpsb2coKSB7CiAgICBlY2hvICJbJDFdWyQoZGF0ZSAiKyVZLSVtLSVkLSVILSVNLSVTIildWyRoXSAkMiIgPiYyCn0KCmxvZyBJTkZPICJBcHBseWluZyBwYXRjaCBmb3IgUFIgJHByIgoKIyBDb25maWd1cmF0aW9uIC0gOC4xOC4xIEdBIHZlcnNpb24vYnVpbGQgdG8gYWxsb3cgdGhlIHVwZ3JhZGUgdG8gZW50ZXIgdGhlIGNvcnJlY3QgZmxvdwpUQVJHRVRfVkVSU0lPTj0iOC4xOC4xLjM2NzkxIgpUQVJHRVRfQlVJTEQ9IjI0MjgxNjAyIgoKIyBGaWxlIHBhdGhzCk1BTklGRVNUX0ZJTEU9Ii9vcHQvdm13YXJlL3Zhci9saWIvdmFtaS91cGRhdGUvZGF0YS9pbmZvL21hbmlmZXN0LWluc3RhbGxlZC54bWwiClZBX1JFTEVBU0VfRklMRT0iL2V0Yy92bXdhcmUtcHJlbHVkZS92YS1yZWxlYXNlLmNvbmYiCgojIFZlcnNpb24gY2hlY2sgLSBvbmx5IGFwcGxpY2FibGUgZm9yIDguMTguMS54IHZlcnNpb25zID4gOC4xOC4xLjM2ODY4IChVMy9VNCkKIyBWZXJzaW9ucyB1cCB0byA4LjE4LjEuMzY4Njggc2hvdWxkIHVwZ3JhZGUgd2l0aG91dCB0aGlzIHdvcmthcm91bmQKTUlOX0FGRkVDVEVEX1ZFUlNJT049IjguMTguMS4zNjg2OSIKY3VycmVudF92ZXJzaW9uPSQodnJhY2xpIHZlcnNpb24gMj4vZGV2L251bGwgfCBncmVwIC1vRSAnWzAtOV0rXC5bMC05XStcLlswLTldK1wuWzAtOV0rJyB8IGhlYWQgLTEpCmlmIFtbIC1uICIkY3VycmVudF92ZXJzaW9uIiBdXTsgdGhlbgogICAgIyBDaGVjayBpZiBwYXRjaCBpcyBhbHJlYWR5IGFwcGxpZWQgKHZlcnNpb24gYWxyZWFkeSBzZXQgdG8gdGFyZ2V0KQogICAgaWYgW1sgIiRjdXJyZW50X3ZlcnNpb24iID09ICIkVEFSR0VUX1ZFUlNJT04iIF1dOyB0aGVuCiAgICAgICAgbG9nIElORk8gIlBhdGNoIGFscmVhZHkgYXBwbGllZC4gQ3VycmVudCB2ZXJzaW9uIGlzIGFscmVhZHkgJFRBUkdFVF9WRVJTSU9OLiBOb3RoaW5nIHRvIGRvLiIKICAgICAgICBleGl0IDAKICAgIGZpCiAgICAKICAgICMgRXh0cmFjdCBtYWpvci5taW5vci5wYXRjaCAoZS5nLiwgOC4xOC4xIGZyb20gOC4xOC4xLjM2OTk0KQogICAgbWFqb3JfbWlub3JfcGF0Y2g9JChlY2hvICIkY3VycmVudF92ZXJzaW9uIiB8IGN1dCAtZCcuJyAtZjEtMykKICAgIAogICAgIyBDaGVjayBpZiB2ZXJzaW9uIGlzIGV4YWN0bHkgOC4xOC4xLnggKG11c3Qgc3RhcnQgd2l0aCA4LjE4LjEpCiAgICBpZiBbWyAiJG1ham9yX21pbm9yX3BhdGNoIiAhPSAiOC4xOC4xIiBdXTsgdGhlbgogICAgICAgIGxvZyBJTkZPICJUaGlzIHBhdGNoIGlzIG9ubHkgYXBwbGljYWJsZSBmb3IgQXJpYSBPcmNoZXN0cmF0b3IgOC4xOC4xLnggdmVyc2lvbnMuIEN1cnJlbnQgdmVyc2lvbjogJGN1cnJlbnRfdmVyc2lvbiIKICAgICAgICBleGl0IDAKICAgIGZpCiAgICAKICAgICMgQ2hlY2sgaWYgdmVyc2lvbiBpcyBhZmZlY3RlZCAoPiA4LjE4LjEuMzY4NjgpCiAgICBpZiBbWyAiJChwcmludGYgJyVzXG4nICIkTUlOX0FGRkVDVEVEX1ZFUlNJT04iICIkY3VycmVudF92ZXJzaW9uIiB8IHNvcnQgLVYgfCBoZWFkIC1uMSkiICE9ICIkTUlOX0FGRkVDVEVEX1ZFUlNJT04iIF1dOyB0aGVuCiAgICAgICAgbG9nIElORk8gIlRoaXMgcGF0Y2ggaXMgb25seSBuZWVkZWQgZm9yIHZlcnNpb25zID4gOC4xOC4xLjM2ODY4LiBDdXJyZW50IHZlcnNpb246ICRjdXJyZW50X3ZlcnNpb24iCiAgICAgICAgZXhpdCAwCiAgICBmaQogICAgCiAgICBsb2cgSU5GTyAiQ3VycmVudCB2ZXJzaW9uOiAkY3VycmVudF92ZXJzaW9uIC0gcGF0Y2ggaXMgYXBwbGljYWJsZSIKZWxzZQogICAgbG9nIFdBUk4gIkNvdWxkIG5vdCBkZXRlcm1pbmUgY3VycmVudCB2ZXJzaW9uLCBwcm9jZWVkaW5nIGFueXdheSIKZmkKCmxvZyBJTkZPICJDcmVhdGluZyBiYWNrdXAgZGlyZWN0b3J5OiAkQkFDS1VQX0RJUiIKbWtkaXIgLXAgIiRCQUNLVVBfRElSIgoKIyBVcGRhdGUgbWFuaWZlc3QtaW5zdGFsbGVkLnhtbAppZiBbWyAtZiAiJE1BTklGRVNUX0ZJTEUiIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiQmFja2luZyB1cCAkTUFOSUZFU1RfRklMRSB0byAkQkFDS1VQX0RJUiIKICAgIGNwICIkTUFOSUZFU1RfRklMRSIgIiRCQUNLVVBfRElSL21hbmlmZXN0LWluc3RhbGxlZC54bWwiCiAgICAKICAgIGxvZyBJTkZPICJVcGRhdGluZyAkTUFOSUZFU1RfRklMRS4uLiIKICAgIAogICAgIyBVcGRhdGUgPHZlcnNpb24+WC5YLlguWFhYWFg8L3ZlcnNpb24+CiAgICBzZWQgLWkgLUUgInN8PHZlcnNpb24+WzAtOV0rXC5bMC05XStcLlswLTldK1wuWzAtOV0rPC92ZXJzaW9uPnw8dmVyc2lvbj4ke1RBUkdFVF9WRVJTSU9OfTwvdmVyc2lvbj58ZyIgIiRNQU5JRkVTVF9GSUxFIgogICAgCiAgICAjIFVwZGF0ZSA8ZnVsbFZlcnNpb24+WC5YLlguWFhYWFggQnVpbGQgWFhYWFhYWFg8L2Z1bGxWZXJzaW9uPgogICAgc2VkIC1pIC1FICJzfDxmdWxsVmVyc2lvbj5bMC05XStcLlswLTldK1wuWzAtOV0rXC5bMC05XSsgQnVpbGQgWzAtOV0rPC9mdWxsVmVyc2lvbj58PGZ1bGxWZXJzaW9uPiR7VEFSR0VUX1ZFUlNJT059IEJ1aWxkICR7VEFSR0VUX0JVSUxEfTwvZnVsbFZlcnNpb24+fGciICIkTUFOSUZFU1RfRklMRSIKICAgIAogICAgbG9nIElORk8gIlVwZGF0ZWQgbWFuaWZlc3QtaW5zdGFsbGVkLnhtbCIKZWxzZQogICAgbG9nIEVSUk9SICIkTUFOSUZFU1RfRklMRSBub3QgZm91bmQiCiAgICBleGl0IDEKZmkKCiMgVXBkYXRlIHZhLXJlbGVhc2UuY29uZgppZiBbWyAtZiAiJFZBX1JFTEVBU0VfRklMRSIgXV07IHRoZW4KICAgIGxvZyBJTkZPICJCYWNraW5nIHVwICRWQV9SRUxFQVNFX0ZJTEUgdG8gJEJBQ0tVUF9ESVIiCiAgICBjcCAiJFZBX1JFTEVBU0VfRklMRSIgIiRCQUNLVVBfRElSL3ZhLXJlbGVhc2UuY29uZiIKICAgIAogICAgbG9nIElORk8gIlVwZGF0aW5nICRWQV9SRUxFQVNFX0ZJTEUuLi4iCiAgICAKICAgICMgVXBkYXRlIHZhLnZlcnNpb24uZnVsbD1YLlguWC5YWFhYWAogICAgc2VkIC1pIC1FICJzfF52YVwudmVyc2lvblwuZnVsbD1bMC05XStcLlswLTldK1wuWzAtOV0rXC5bMC05XSt8dmEudmVyc2lvbi5mdWxsPSR7VEFSR0VUX1ZFUlNJT059fGciICIkVkFfUkVMRUFTRV9GSUxFIgogICAgCiAgICAjIFVwZGF0ZSB2YS52ZXJzaW9uLmJ1aWxkPVhYWFhYWFhYCiAgICBzZWQgLWkgLUUgInN8XnZhXC52ZXJzaW9uXC5idWlsZD1bMC05XSt8dmEudmVyc2lvbi5idWlsZD0ke1RBUkdFVF9CVUlMRH18ZyIgIiRWQV9SRUxFQVNFX0ZJTEUiCiAgICAKICAgIGxvZyBJTkZPICJVcGRhdGVkIHZhLXJlbGVhc2UuY29uZiIKZWxzZQogICAgbG9nIEVSUk9SICIkVkFfUkVMRUFTRV9GSUxFIG5vdCBmb3VuZCIKICAgIGV4aXQgMQpmaQoKbG9nIElORk8gIlBhdGNoICRwciBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5LiIKbG9nIElORk8gIkJhY2t1cCBmaWxlcyBzdG9yZWQgaW4gJEJBQ0tVUF9ESVIiCmxvZyBJTkZPICJZb3UgY2FuIG5vdyByZXRyeSB0aGUgdXBncmFkZSB0byA5LjAueCIK | bash -'
   

4. Verify the patch was applied on all nodes by checking the version:
   

      vracli cluster exec -- bash -c 'grep -E "^va\.version\.(full|build)" /etc/vmware-prelude/va-release.conf'

   Expected output (on each node):
   

      va.version.full=8.18.1.36791
      va.version.build=24281602

5. Retry the upgrade to 9.0.x. For detailed instructions on the upgrade path, follow the official Broadcom documentation