VCF Operations Orchestrator Upgrade from 8.18.1 U3 to 9.0.1 fails with "overall upgrade has failed" error.

Products

• VCF Operations

Issue/Introduction

• The /var/log/vmware/prelude/upgrade.log contains errors similar to:

  [ERROR] VAMI overall upgrade has failed.
  [ERROR] Upgrade failed on this node. Monitoring on this node will be deactivated.

• The /opt/vmware/var/log/vami/updatecli.log contains errors similar to:

  [INFO] Running /opt/vmware/var/lib/vami/update/data/job/4/test_command
  warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/Operations-Logs-Agent-9.0.0.0.24597150.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID fdf4dc95: NOKEY
  warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/kubectl-1.32.0-1.el9.vmware.2.fips.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 001e5cc9: NOKEY
  error: Failed dependencies:
  libreadline.so.7()(64bit) is needed by (installed) socat-2.0.0.b9-3.ph4.x86_64
  [ERROR] Failed with exit code 65024

Environment

• VMware vCenter 8.0.1 U3
• VMware Automation Orchestrator 8.18.1 U3/U4

Cause

The Orchestrator VA upgrade engine relies on the source version being declared in the target 9.0.x build.

The 8.18.1 U3/U4 patch releases have version numbers (e.g., 8.18.1.36994) that are not recognized as valid upgrade source versions by the 9.0.x upgrade package, causing the dependency check phase to fail.

Resolution

Apply the workaround below before attempting the upgrade.

Workaround

The following steps should be executed to downgrade the reported version to the base 8.18.1 version, which allows the upgrade to proceed correctly:

1. Take a VM snapshot (without memory, with quiesced option selected) of all Orchestrator VMs. This can be done manually from vCenter.
2. SSH into one of the Orchestrator appliances as root.
3. Execute the following command to apply the patch on all cluster nodes:

   vracli cluster exec -- bash -c 'base64 -d <<< IyEvYmluL2Jhc2gKCmxvZ0Rpcj0iL3Zhci9sb2cvdm13YXJlL3ByZWx1ZGUiCnRpbWVzdGFtcD0kKGRhdGUgJyslWS0lbS0lZC0lSC0lTS0lUycpCnByPSJWQ09QUy02ODI1NyIKQkFDS1VQX0RJUj0iL2RhdGEvcGF0Y2gtYmFja3VwLyRwci0kdGltZXN0YW1wIgoKbWtkaXIgLXAgIiRsb2dEaXIiCmV4ZWMgMj4gPih0ZWUgLWEgIiRsb2dEaXIvcGF0Y2gtJHByLSR0aW1lc3RhbXAubG9nIikKZXhlYyA+JjIKCmg9JChob3N0bmFtZSkKbG9nKCkgewogICAgZWNobyAiWyQxXVskKGRhdGUgIislWS0lbS0lZC0lSC0lTS0lUyIpXVskaF0gJDIiID4mMgp9Cgpsb2cgSU5GTyAiQXBwbHlpbmcgcGF0Y2ggZm9yIFBSICRwciIKCiMgQ29uZmlndXJhdGlvbiAtIHRhcmdldCB2ZXJzaW9ucyB0byBkb3duZ3JhZGUgdG8KVEFSR0VUX1ZFUlNJT049IjguMTguMS4zNjc5MSIKVEFSR0VUX0JVSUxEPSIyNDI4MTYwMiIKCiMgRmlsZSBwYXRocwpNQU5JRkVTVF9GSUxFPSIvb3B0L3Ztd2FyZS92YXIvbGliL3ZhbWkvdXBkYXRlL2RhdGEvaW5mby9tYW5pZmVzdC1pbnN0YWxsZWQueG1sIgpWQV9SRUxFQVNFX0ZJTEU9Ii9ldGMvdm13YXJlLXByZWx1ZGUvdmEtcmVsZWFzZS5jb25mIgoKIyBWZXJzaW9uIGNoZWNrIC0gb25seSBhcHBsaWNhYmxlIGZvciA4LjE4LjEueCB2ZXJzaW9ucyAobm90IDkueCBvciBoaWdoZXIpCmN1cnJlbnRfdmVyc2lvbj0kKGdyZXAgJ152YVwudmVyc2lvblwuZnVsbD0nICIkVkFfUkVMRUFTRV9GSUxFIiAyPi9kZXYvbnVsbCB8IGN1dCAtZCc9JyAtZjIpCmlmIFtbIC1uICIkY3VycmVudF92ZXJzaW9uIiBdXTsgdGhlbgogICAgIyBDaGVjayBpZiBwYXRjaCBpcyBhbHJlYWR5IGFwcGxpZWQgKHZlcnNpb24gYWxyZWFkeSBzZXQgdG8gdGFyZ2V0KQogICAgaWYgW1sgIiRjdXJyZW50X3ZlcnNpb24iID09ICIkVEFSR0VUX1ZFUlNJT04iIF1dOyB0aGVuCiAgICAgICAgbG9nIElORk8gIlBhdGNoIGFscmVhZHkgYXBwbGllZC4gQ3VycmVudCB2ZXJzaW9uIGlzIGFscmVhZHkgJFRBUkdFVF9WRVJTSU9OLiBOb3RoaW5nIHRvLiIKICAgICAgICBleGl0IDAKICAgIGZpCiAgICAKICAgICMgRXh0cmFjdCBtYWpvci5taW5vci5wYXRjaCAoZS5nLiwgOC4xOC4xIGZyb20gOC4xOC4xLjM2OTk0KQogICAgbWFqb3JfbWlub3JfcGF0Y2g9JChlY2hvICIkY3VycmVudF92ZXJzaW9uIiB8IGN1dCAtZCcuJyAtZjEtMykKICAgIAogICAgIyBDaGVjayBpZiB2ZXJzaW9uIGlzIGV4YWN0bHkgOC4xOC4xLnggKG11c3Qgc3RhcnQgd2l0aCA4LjE4LjEpCiAgICBpZiBbWyAiJG1ham9yX21pbm9yX3BhdGNoIiAhPSAiOC4xOC4xIiBdXTsgdGhlbgogICAgICAgIGxvZyBJTkZPICJUaGlzIHBhdGNoIGlzIG9ubHkgYXBwbGljYWJsZSBmb3IgQXJpYSBPcmNoZXN0cmF0b3IgOC4xOC4xLnggdmVyc2lvbnMuIEN1cnJlbnQgdmVyc2lvbjogJGN1cnJlbnRfdmVyc2lvbiIKICAgICAgICBleGl0IDAKICAgIGZpCiAgICBsb2cgSU5GTyAiQ3VycmVudCB2ZXJzaW9uOiAkY3VycmVudF92ZXJzaW9uIC0gcGF0Y2ggaXMgYXBwbGljYWJsZSIKZWxzZQogICAgbG9nIFdBUk4gIkNvdWxkIG5vdCBkZXRlcm1pbmUgY3VycmVudCB2ZXJzaW9uLCBwcm9jZWVkaW5nIGFueXdheSIKZmkKCmxvZyBJTkZPICJDcmVhdGluZyBiYWNrdXAgZGlyZWN0b3J5OiAkQkFDS1VQX0RJUiIKbWtkaXIgLXAgIiRCQUNLVVBfRElSIgoKIyBVcGRhdGUgbWFuaWZlc3QtaW5zdGFsbGVkLnhtbAppZiBbWyAtZiAiJE1BTklGRVNUX0ZJTEUiIF1dOyB0aGVuCiAgICBsb2cgSU5GTyAiQmFja2luZyB1cCAkTUFOSUZFU1RfRklMRSB0byAkQkFDS1VQX0RJUiIKICAgIGNwICIkTUFOSUZFU1RfRklMRSIgIiRCQUNLVVBfRElSL21hbmlmZXN0LWluc3RhbGxlZC54bWwiCiAgICAKICAgIGxvZyBJTkZPICJVcGRhdGluZyAkTUFOSUZFU1RfRklMRS4uLiIKICAgIAogICAgIyBVcGRhdGUgPHZlcnNpb24+WC5YLlguWFhYWFg8L3ZlcnNpb24+CiAgICBzZWQgLWkgLUUgInN8PHZlcnNpb24+WzAtOV0rXC5bMC05XStcLlswLTldK1wuWzAtOV0rPC92ZXJzaW9uPnw8dmVyc2lvbj4ke1RBUkdFVF9WRVJTSU9OfTwvdmVyc2lvbj58ZyIgIiRNQU5JRkVTVF9GSUxFIgogICAgCiAgICAjIFVwZGF0ZSA8ZnVsbFZlcnNpb24+WC5YLlguWFhYWFggQnVpbGQgWFhYWFhYWFg8L2Z1bGxWZXJzaW9uPgogICAgc2VkIC1pIC1FICJzfDxmdWxsVmVyc2lvbj5bMC05XStcLlswLTldK1wuWzAtOV0rXC5bMC05XSsgQnVpbGQgWzAtOV0rPC9mdWxsVmVyc2lvbj58PGZ1bGxWZXJzaW9uPiR7VEFSR0VUX1ZFUlNJT059IEJ1aWxkICR7VEFSR0VUX0JVSUxEfTwvZnVsbFZlcnNpb24+fGciICIkTUFOSUZFU1RfRklMRSIKICAgIAogICAgbG9nIElORk8gIlVwZGF0ZWQgbWFuaWZlc3QtaW5zdGFsbGVkLnhtbCIKZWxzZQogICAgbG9nIEVSUk9SICIkTUFOSUZFU1RfRklMRSBub3QgZm91bmQiCiAgICBleGl0IDEKZmkKCiMgVXBkYXRlIHZhLXJlbGVhc2UuY29uZgppZiBbWyAtZiAiJFZBX1JFTEVBU0VfRklMRSIgXV07IHRoZW4KICAgIGxvZyBJTkZPICJCYWNraW5nIHVwICRWQV9SRUxFQVNFX0ZJTEUgdG8gJEJBQ0tVUF9ESVIiCiAgICBjcCAiJFZBX1JFTEVBU0VfRklMRSIgIiRCQUNLVVBfRElSL3ZhLXJlbGVhc2UuY29uZiIKICAgIAogICAgbG9nIElORk8gIlVwZGF0aW5nICRWQV9SRUxFQVNFX0ZJTEUuLi4iCiAgICAKICAgICMgVXBkYXRlIHZhLnZlcnNpb24uZnVsbD1YLlguWC5YWFhYWAogICAgc2VkIC1pIC1FICJzfF52YVwudmVyc2lvblwuZnVsbD1bMC05XStcLlswLTldK1wuWzAtOV0rXC5bMC05XSt8dmEudmVyc2lvbi5mdWxsPSR7VEFSR0VUX1ZFUlNJT059fGciICIkVkFfUkVMRUFTRV9GSUxFIgogICAgCiAgICAjIFVwZGF0ZSB2YS52ZXJzaW9uLmJ1aWxkPVhYWFhYWFhYCiAgICBzZWQgLWkgLUUgInN8XnZhXC52ZXJzaW9uXC5idWlsZD1bMC05XSt8dmEudmVyc2lvbi5idWlsZD0ke1RBUkdFVF9CVUlMRH18ZyIgIiRWQV9SRUxFQVNFX0ZJTEUiCiAgICAKICAgIGxvZyBJTkZPICJVcGRhdGVkIHZhLXJlbGVhc2UuY29uZiIKZWxzZQogICAgbG9nIEVSUk9SICIkVkFfUkVMRUFTRV9GSUxFIG5vdCBmb3VuZCIKICAgIGV4aXQgMQpmaQoKbG9nIElORk8gIlBhdGNoICRwciBjb21wbGV0ZWQgc3VjY2Vzc2Z1bGx5LiIKbG9nIElORk8gIkJhY2t1cCBmaWxlcyBzdG9yZWQgaW4gJEJBQ0tVUF9ESVIiCmxvZyBJTkZPICJZb3UgY2FuIG5vdyByZXRyeSB0aGUgdXBncmFkZSB0byA5LjAueCIK | bash -'

4. Verify the patch was applied on all nodes by checking the version:

   vracli cluster exec -- bash -c 'grep -E "^va\.version\.(full|build)" /etc/vmware-prelude/va-release.conf'

   Expected output (on each node):

   va.version.full=8.18.1.36791
   va.version.build=24281602

5. Retry the upgrade to 9.0.x

Uninstall/Rollback Instructions

To rollback the patch (if needed before upgrade):

1. Locate the backup files in /data/patch-backup/VCOPS-68257-<timestamp>/
2. Restore the original files on all nodes:

   vracli cluster exec -- bash -c 'cp /data/patch-backup/VCOPS-68257-<timestamp>/manifest-installed.xml /opt/vmware/var/lib/vami/update/data/info/manifest-installed.xml'
   vracli cluster exec -- bash -c 'cp /data/patch-backup/VCOPS-68257-<timestamp>/va-release.conf /etc/vmware-prelude/va-release.conf'

Note: Replace <timestamp> with the actual timestamp from the backup directory.