Error: "[ERROR] Failed with exit code 65024" and "error: Failed dependencies" causes upgrade to VCD 10.6.x to fail

Products

• VMware Cloud Director

Issue/Introduction

• While trying to perform an upgrade of VMware Cloud Director (VCD) to version 10.6 or later, it is failing with the error:
  
Error while running installation tests and post-install scripts
  
  
• In the /opt/vmware/var/log/vami/updatecli.log on the VCD Cell, you see errors similar to:
  
  installing package VMware-Postgres-upgrade-10-14.11.0-23473091.x86_64 needs 198MB on the / filesystem
DD/MM/YYYY HH:MM:SS [ERROR] Failed with exit code 65024
DD/MM/YYYY HH:MM:SS [INFO] Update status: Running post-install scripts
DD/MM/YYYY HH:MM:SS [INFO] Running /opt/vmware/var/lib/vami/update/data/job/12/post_install '10.5.1.10593' '10.6.1.11841' 2
Failed with status of 2 while installing version 10.6.1.11841
VM version is still 10.5.1.10593
DD/MM/YYYY HH:MM:SS [ERROR] Failed with exit code 256
DD/MM/YYYY HH:MM:SS [INFO] Update status: Running VMware tools reconfiguration
DD/MM/YYYY HH:MM:SS [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools
vmware-toolbox-cmd is /bin/vmware-toolbox-cmd
vmtoolsd wrapper not required on this VM with systemd.
DD/MM/YYYY HH:MM:SS [INFO] Update status: Done VMware tools reconfiguration
DD/MM/YYYY HH:MM:SS [INFO] Update status: Error while running installation tests and post-install scripts
DD/MM/YYYY HH:MM:SS [ERROR] Failure: updatecli exiting abnormally
DD/MM/YYYY HH:MM:SS [INFO] Install Finished
  
  
• Also in /opt/vmware/var/log/vami/updatecli.log, you might see following errors related to package xml-security-c-1.7.3-4.ph2.x86_64:
  
  /opt/likewise/bin/domainjoin-cli: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory
/opt/likewise/bin/domainjoin-cli: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory
/opt/likewise/bin/lwsm: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory
Removed /etc/systemd/system/multi-user.target.wants/lwsmd.service.
DD/MM/YYYY HH:MM:SS [INFO] Update status: Done pre-install scripts
DD/MM/YYYY HH:MM:SS [INFO] Update status: Running installation tests
DD/MM/YYYY HH:MM:SS [INFO] Running /opt/vmware/var/lib/vami/update/data/job/21/test_command
warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/audit-2.8.5-31.ph4.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8a6a826d: NOKEY
warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-vcloud-distribution-ovascripts-10.6.0-24055916.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 530c79e6: NOKEY
error: Failed dependencies:
 libcrypto.so.1.0.0()(64bit) is needed by (installed) xml-security-c-1.7.3-4.ph2.x86_64
 libssl.so.1.0.0()(64bit) is needed by (installed) xml-security-c-1.7.3-4.ph2.x86_64
  
  
• Similarly, you see dependency failure's for file tcpdump-4.9.2-1.ph3.x86_64 in /opt/vmware/var/log/vami/updatecli.log:
  
  DD/MM/YYYY HH:MM:SS [INFO] Update status: Running installation tests
DD/MM/YYYY HH:MM:SS [INFO] Running /opt/vmware/var/lib/vami/update/data/job/5/test_command
warning: /opt/vmware/var/lib/vami/update/data/package-pool/package-pool/vmware-vcloud-distribution-ovascripts-10.6.1-24532678.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 530c79e6: NOKEY
error: Failed dependencies:
        libcrypto.so.1.0.0()(64bit) is needed by (installed) tcpdump-4.9.2-1.ph3.x86_64
  
  
• You may also see the below failures:
  
  error: Failed dependencies:
python3 = 3.6.9-10.ph2 is needed by (installed) python3-tools-3.6.9-10.ph2.x86_64
/usr/bin/python3.6 is needed by (installed) python3-tools-3.6.9-10.ph2.x86_64

Environment

• VMware Cloud Director 10.6
• VMware Cloud Director 10.5
• VMware Cloud Director 10.4

Cause

This issue occurs due to the already installed packages which are not expected to be present:

xml-security-c-1.7.3-4.ph2.x86_64 - It is a Photon OS 2.0 package.
tcpdump-4.9.2-1.ph3.x86_64 - It is a network packet capture tool.
python3-tools-3.6.9-10.ph2.x86_64

Resolution

As these packages are not expected to be present, they can be removed before re-attempting the upgrade.

1. Roll back the VCD Cell to before the failed upgrade was attempted.
2. Prepare again for the upgrade attempt by taking the appropriate snapshots and backups as per the Before you begin in the VCD upgrade documentation.
3. Check if the packages reported in the error installed on all VCD cells by executing the command:
   
   rpm -qa | grep xml-security-c-1.7.3-4.ph2.x86_64
rpm -qa | grep tcpdump-4.9.2-1.ph3.x86_64
rpm -qa | grep python3-tools-3.6.9-10.ph2.x86_64
   
   
4. If the above packages are installed, remove the packages from all the VCD Cells by executing the below commands:
   
   rpm -e xml-security-c-1.7.3-4.ph2.x86_64
rpm -e tcpdump-4.9.2-1.ph3.x86_64
rpm -e python3-tools-3.6.9-10.ph2.x86_64
   
   
5. Once the packages are removed, proceed with the upgrade.