Upgrade to VCD 10.6 GA fails installing few RPMs though upgrade is reported as successful.

Products

• VMware Cloud Director

Issue/Introduction

Issue will be seen only for VCD appliances which are not connected to internet where STIG hardening invocation tries to install these packages and fails installation.

/opt/vmware/var/log/vami/updatecli.log reports following error:

TASK [/usr/share/ansible/stig-hardening : PHTN-40-000013 - Check to see if OpenSSL FIPS Provider is installed] ***
ok: [127.0.0.1] => {"changed": false, "cmd": "set -o pipefail\nrpm -qa | grep ^openssl-fips-provider\n", "delta": "0:00:00.07xx88", "end": "YYYY-MM-DD xx:xx:xx.xxxxxx", "failed_when_result": false, "msg": "non-zero return code", "rc": 1, "start": "YYYY-MM-DD xx:xx:xx.xxxxxx", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [/usr/share/ansible/stig-hardening : PHTN-40-000013 - Install OpenSSL FIPS Provider] ***
fatal: [127.0.0.1]: FAILED! => {"changed": false, "cmd": ["tdnf", "-y", "install", "openssl-fips-provider"], "delta": "x:xx:xx.02xx84", "end": "YYYY-MM-DD xx:xx:xx.xxxxxx", "msg": "non-zero return code", "rc": 243, "start": "YYYY-MM-DD xx:xx:xx.xxxxxx", "stderr": "Error(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_release_4.0_x86_64'\nError(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Extras 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_extras_4.0_x86_64'\nError(1229) : Timeout was reached\nError: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64) Updates' from 'https://packages.vmware.com/photon/4.0/photon_updates_4.0_x86_64'\nopenssl-fips-provider package not found or not installed\nError(1011) : No matching packages", "stderr_lines": ["Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_release_4.0_x86_64'", "Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Extras 4.0 (x86_64)' from 'https://packages.vmware.com/photon/4.0/photon_extras_4.0_x86_64'", "Error(1229) : Timeout was reached", "Error: Failed to synchronize cache for repo 'VMware Photon Linux 4.0 (x86_64) Updates' from 'https://packages.vmware.com/photon/4.0/photon_updates_4.0_x86_64'", "openssl-fips-provider package not found or not installed", "Error(1011) : No matching packages"], "stdout": "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Linux 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Extras 4.0 (x86_64)'\nRefreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'\nretrying 1/10\nretrying 2/10\nretrying 3/10\nretrying 4/10\nretrying 5/10\nretrying 6/10\nretrying 7/10\nretrying 8/10\nretrying 9/10\nretrying 10/10\nDisabling Repo: 'VMware Photon Linux 4.0 (x86_64) Updates'", "stdout_lines": ["Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64)'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Linux 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Extras 4.0 (x86_64)'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Extras 4.0 (x86_64)'", "Refreshing metadata for: 'VMware Photon Linux 4.0 (x86_64) Updates'", "retrying 1/10", "retrying 2/10", "retrying 3/10", "retrying 4/10", "retrying 5/10", "retrying 6/10", "retrying 7/10", "retrying 8/10", "retrying 9/10", "retrying 10/10", "Disabling Repo: 'VMware Photon Linux 4.0 (x86_64) Updates'"]}

PLAY RECAP *********************************************************************
127.0.0.1                  : ok=19   changed=4    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   

Adding default FIPS Off configuration for OpenSSL
Finished installing version xx.xx.xxxxxxx
DD/MM/YYYY 07:01:03 [INFO] Update status: Done post-install scripts
DD/MM/YYYY 07:01:03 [INFO] Update status: Running VMware tools reconfiguration
DD/MM/YYYY 07:01:03 [INFO] Running /opt/vmware/share/vami/vami_reconfigure_tools 
vmware-toolbox-cmd is /bin/vmware-toolbox-cmd
vmtoolsd wrapper not required on this VM with systemd.
DD/MM/YYYY 07:01:03 [INFO] Update status: Done VMware tools reconfiguration
DD/MM/YYYY 07:01:03 [INFO] Update status: Running finalizing installation
DD/MM/YYYY 07:01:03 [INFO] Running /opt/vmware/var/lib/vami/update/data/job/2/manifest_update 
DD/MM/YYYY 07:01:03 [INFO] Update status: Done finalizing installation
DD/MM/YYYY 07:01:03 [INFO] Update status: Update completed successfully
DD/MM/YYYY 07:01:03 [INFO] Install Finished

Environment

VMware Cloud Director 10.4
VMware Cloud Director 10.5
VMware Cloud Director 10.6

Resolution

Note: Apply the below workaround, only if the environment is upgraded to 10.6 GA. Implementing this workaround will not work on lower versions, prior 10.6, as it will not install the RPMs.

Workaround:
Steps below is a workaround to install missing RPMs manually.

Download the archive.tar.gz file from attachments. 

1. Copy the downloaded archive.tar.gz to /tmp
2. Extract the archive file.
# tar -zxvf archive.tar.gz
3. Run the script which will disable default repos in photon-os to allow installation of missing RPMs and enables back the repos
# /tmp/archive/install.sh