"Timed out while waiting in-place upgrade of vRSLCM to complete", VREALIZE_UPGRADE stage fails while upgrading vRSLCM to 8.x

Products

• VMware Aria Suite
• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Upgrading vRSLCM to 8.x (during VCF 4.x upgrade) fails in the VREALIZE_UPGRADE stage.
• In the /var/log/vmware/vcf/lcm/lcm-debug.log file, you see similar to:

Timed out while waiting in-place upgrade of vRSLCM to complete

Environment

VMware Cloud Foundation 4.3
VMware Cloud Foundation 4.1
VMware Cloud Foundation 4.2
VMware vRealize Suite Lifecycle Manager 8.x
VMware Cloud Foundation 4.4

Cause

This issue is caused by the time period that the SDDC Managers automation waits for the vRSLCM upgrade to complete is 10 mins, on some resource constrained labs this may not be enough, and even though the vRSLCM upgrade in SDDC Manager UI fails after 10-15 minutes the vRSLCM UI is up and the product is successfully upgraded to the target 8.x version.

Resolution

The vRSLCM product is successfully upgraded but the SDDC Manager upgrade workflow is marked as failed, please follow below steps to resolve this issue:

1. Take snapshot of SDDC Manager VM.
2. Open the vRSLCM UI, click on the username (top right corner), open the About page, verify that the version is upgraded to the target version (for example 8.2) and copy the Build number.
3. SSH to SDDC Manager VM, enable the root user and run: 

psql -h localhost -U postgres -d platform -c "update vrslcm set version = '8.x.x-BUILD_NUMBER';"

Note:
Replace 8.x.x with the correct version number, for example 8.2.0
Replace BUILD_NUMBER with the build number copied from Step 2

Following are couple of examples for 8.2.0 and 8.4.1.1 : 

psql -h localhost -U postgres -d platform -c "update vrslcm set version = '8.2.0-17465722';"
psql -h localhost -U postgres -d platform -c "update vrslcm set version = '8.4.1-18067607';"

3. SSH to the vRSLCM VM and run:

eject -F /dev/sr0