VCF 4.5 - SDDC Manager Third_Party Upgrade fails during VIP service installation due to an expired root password
search cancel

VCF 4.5 - SDDC Manager Third_Party Upgrade fails during VIP service installation due to an expired root password

book

Article ID: 345203

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

Symptoms:

When upgrading from VCF 4.X instances to VCF 4.5, the configuration drift bundle application i.e the third party upgrade fails at VIP installation stage.

NOTE : The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment


We see log entries similar to:

/var/log/vmware/vcf/lcm/thirdparty/upgrades/<upgrade_id>/vcf-platform/upgrade:

Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 494, in <module>
    upgrade_workflow()
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 106, in upgrade_workflow
    errmsg=cap_update_status["message"])
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/../../wrapper.py", line 168, in update_status
    raise Exception
..
..
2022-10-27 09:24:23,820: ERROR: Failed to update VCF Services and Photon rpms in SDDC Manager
2022-10-27 09:24:23,893: INFO:
2022-10-27 09:24:23,893: INFO: RC: 1, OUT:
2022-10-27 09:24:23,893: INFO: ERR: Traceback (most recent call last):
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 494, in <module>
    upgrade_workflow()
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/vcf_platform_upgrade.py.copy", line 106, in upgrade_workflow
    errmsg=cap_update_status["message"])
  File "/var/log/vmware/vcf/lcm/thirdparty/bundles/b984c5e4-167a-4886-8f23-be34c9176ca9/thirdparty/vcf-platform-upgrade/bin/../../wrapper.py", line 168, in update_status
    raise Exception


/var/log/vmware/capengine/cap-update/workflow.log: 
2022/10/27 09:21:49.817860 task_progress.go:24: Installing RPMs
2022/10/27 09:21:49.822956 installer.go:123: RPM installation logs will be written to /var/log/vmware/capengine/cap-update/install-20221027092149
2022/10/27 09:21:49.822974 installer.go:124: Rebuilding RPM database
2022/10/27 09:21:50.359787 installer.go:131: Checking RPMs validity
2022/10/27 09:21:51.551537 installer.go:139: Installing RPMs
2022/10/27 09:24:08.549872 installer.go:143: Failed to execute command: chroot /storage/alt_root /bin/bash << EOF
rpm -Uvvh --force --nodeps --nodigest /storage/da1568f5-9581-4e02-b74a-ad47889d1d2a/stage/*.rpm >> /var/log/vmware/capengine/cap-update/install-2022102709214
9 2>&1
EOF. Output: , Error: exit status 251
2022/10/27 09:24:08.549932 install_plugin.go:61: Failed to install RPMs. Error: exit status 251
2022/10/27 09:24:08.549947 installer.go:301: Unmounting devices from /storage/alt_root
2022/10/27 09:24:08.549958 installer.go:305: Executing command: umount -l /storage/alt_root/data
2022/10/27 09:24:08.575901 installer.go:305: Executing command: umount -l /storage/alt_root/opt/vmware/vcf
2022/10/27 09:24:08.595730 installer.go:305: Executing command: umount -l /storage/alt_root/var/log/vmware
2022/10/27 09:24:08.623721 installer.go:305: Executing command: umount -l /storage/alt_root/boot
2022/10/27 09:24:08.651774 installer.go:305: Executing command: umount -l /storage/alt_root/nfs/vmware/vcf/nfs-mount
2022/10/27 09:24:08.679920 installer.go:305: Executing command: umount -l /storage/alt_root/home/backup/nfs/vmware/vcf/nfs-mount/backup
2022/10/27 09:24:08.703801 installer.go:313: Unmounting dev, proc, sys
2022/10/27 09:24:08.771811 installer.go:325: Finished unmounting
2022/10/27 09:24:08.778045 workflow_manager.go:198: Task install failed. Error: failed to install RPM


Environment

Vmware Cloud Foundation 4.5

Cause

If the root or vcf account password expiration is set to never expire, the SDDC Manager upgrade workflow sets it to 365 days.
This results in the SDDC Manager root or vcf account passwords, if they were last set before 365 days, to instantly expire, causing an RPM installation to fail due to this expiration.

Resolution

This issue is resolved in VCF 4.5.1.

Workaround:
- Login to SDDC manager using the existing vcf and root credentials.

- Reset the vcf and root passwords using the following commands respectively: 
passwd vcf
passwd root

- Go to SDDC Manager UI and restart the failed workflow/upgrade.

Additional Information

If the password reset is failing with errors such as:
Password has already been used. Choose another.
passwd: Authentication token manipulation error
passwd: password unchanged

Please refer to the following KB: https://kb.vmware.com/s/article/88226

Impact/Risks:
NONE: The process involves resetting root and vcf account passwords. There is no impact to the SDDC Manager or its workflows.

Powered by Wolken Software