Upgrade to VCD 10.6 GA fails installing few RPMs though upgrade is reported as successful
search cancel

Upgrade to VCD 10.6 GA fails installing few RPMs though upgrade is reported as successful

book

Article ID: 372041

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

Issue will be seen only for VCD appliances which are not connected to the internet where STIG hardening invocation tries to install multiple packages and fails installation.

Commonly, upgrade from version 10.5.1.1 when running the command "vamicli update --install latest" seems to get stuck without completing. The upgrade can take ~3 hours to timeout retries of the STIG hardening tasks.

/opt/vmware/var/log/vami/updatecli.log reports the following or similar errors:

TASK [/usr/share/ansible/stig-hardening : PHTN-40-000013 - Check to see if OpenSSL FIPS Provider is installed] ***
ok: [127.0.0.1] => {"changed": false, "cmd": "set -o pipefail\nrpm -qa | grep ^openssl-fips-provider\n", "delta": "0:00:00.07xx88", "end": "YYYY-MM-DD xx:xx:xx.xxxxxx", "failed_when_result": false, "msg": "non-zero return code", "rc": 1, "start": "YYYY-MM-DD xx:xx:xx.xxxxxx", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [/usr/share/ansible/stig-hardening : PHTN-40-000013 - Install OpenSSL FIPS Provider] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": ["tdnf", "-y", "install", "openssl-fips-provider"], "delta": "x:xx:xx.02xx84", "end": "YYYY-MM-DD xx:xx:xx.xxxxxx", "msg": "non-zero return code", "rc": 243, "start": "YYYY-MM-DD xx:xx:xx.xxxxxx", "stderr": "Error(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_release_4.0_x86_64'\nError(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Extras 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_extras_4.0_x86_64'\nError(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64) Updates' from 'https://packages.vmware.com/photon/4.0/photon_updates_4.0_x86_64'\nopenssl-fips-provider package not found or not installed\nError(1011) : No matching packages", "stderr_lines": ["Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_release_4.0_x86_64'", "Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Extras 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_extras_4.0_x86_64'", "Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64) Updates' from 'https://packages.vmware.com/photon/4.0/photon_updates_4.0_x86_64'", "openssl-fips-provider package not found or not installed", "Error(1011) : No matching packages"], "stdout": "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Linux 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Extras 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Linux 4.0 (x86_64) Updates'", "stdout_lines": ["Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Linux 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Extras 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Linux 4.0 (x86_64) Updates'"]}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=19   changed=4    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

Adding default FIPS Off configuration for OpenSSL
Finished installing version xx.xx.xxxxxxx
DD/MM/YYYY 07:01:03 [INFO] Update status: Done post-install scripts
DD/MM/YYYY 07:01:03 [INFO] Update status: Running VMware tools reconfiguration
DD/MM/YYYY 07:01:03 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools 
vmware-toolbox-cmd is /bin/vmware-toolbox-cmd
vmtoolsd wrapper not required on this VM with systemd.
DD/MM/YYYY 07:01:03 [INFO] Update status: Done VMware tools reconfiguration
DD/MM/YYYY 07:01:03 [INFO] Update status: Running finalizing installation
DD/MM/YYYY 07:01:03 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/2/manifest_update 
DD/MM/YYYY 07:01:03 [INFO] Update status: Done finalizing installation
DD/MM/YYYY 07:01:03 [INFO] Update status: Update completed successfully
DD/MM/YYYY 07:01:03 [INFO] Install Finished

Environment

VMware Cloud Director 10.4
VMware Cloud Director 10.5
VMware Cloud Director 10.6

Cause

This issue can occur if the Cloud Director cell is in a private network with no internet access or behind a restrictive firewall during the upgrade. Several rpms that were not included in the 10.6 release bundle are unable to install without access to the public Photon repositories.

Resolution

This issue has been fixed in VMware Cloud Director 10.6.0.1: VMware Cloud Director 10.6.0.1 Release Notes

Workaround 1:

  1. Since, the upgrade of VMware Cloud Director was stopped because of the error or even being paused, you need to cancel the upgrade setup and REVERT to the snapshot of the primary cell that was taken before the upgrade process started.
  2. After reverting to the snapshot, you need to make sure that the VMware Cloud Director cells are able to access the internet.
  3. After you open the internet access to the cells, you can proceed with the regular upgrade process as mentioned in the document: Upgrade Your VMware Cloud Director Appliance by Using an Update Package

Workaround 2:

Note: Apply the below workaround, only if the environment is upgraded to 10.6 GA. Implementing this workaround will not work on lower versions, prior 10.6, as it will not install the RPMs.

If you are unable give the cells internet access, you need to download the archive.tar.gz file from attachments and follow the steps below:

  1. Copy the downloaded (attached below with this kb) archive.tar.gz to /tmp
  2. Extract the archive file.
    # tar -zxvf archive.tar.gz
  3. Run the script which will disable default repos in photon-os to allow installation of missing RPMs and reenables the repos
    # /tmp/archive/install.sh 
  4. After following the above workaround, you can continue with the Process of Upgrading the VMware Cloud Director Appliance following this document: Upgrade Your VMware Cloud Director Appliance by Using an Update Package

Attachments

archive.tar.gz get_app
Powered by Wolken Software