A VMware Cloud Foundation upgrade fails at stage 'JRE_UPGRADE_STAGE'

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• When attempting to upgrade SDDC Manager, its fails at stage JRE_UPGRADE_STAGE.
• You see errors similar to the following in the /home/vrack/lcm/logs/lcm.log file on the SDDC Manager Controller VM:

2018-12-20 11:56:46.975 [ThreadPoolTaskExecutor-10] INFO  [com.vmware.evo.sddc.lcm.primitive.evo.impl.LcmThirdPartySWUpgradeService]  upgradeId=5349da85b-4502-4901-ba91-5b237ec160a10,resourceType=LCM,resourceId=a3162928-e654-4953-b4e8-b6cdecda1434,bundleElementId=98babeb6-8b0a-4643-a3b2-268f26358ee8 component upgrade jre failed at stage JRE_UPGRADE_STAGE

2018-12-20 11:56:46.975 [ThreadPoolTaskExecutor-10] ERROR [com.vmware.evo.sddc.lcm.primitive.evo.impl.VcfServicePrimitiveImpl]  upgradeId=5349da85b-4502-4901-ba91-5b237ec160a10,resourceType=LCM,resourceId=a3162928-e654-4953-b4e8-b6cdecda1434,bundleElementId=98babeb6-8b0a-4643-a3b2-268f26358ee8 upgrade failed at stage JRE_UPGRADE_STAGE

• A custom vSphere SSO domain name is in use, not the default vsphere.local.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 

Environment

VMware Cloud Foundation 2.3.x

Cause

This issue occurs when a non-default vSphere SSO domain (vsphere.local) is used in a VMware Cloud foundation 2.3.x environment.

Resolution

This is a known issue affecting VMware Cloud Foundation 2.3.x. There is currently no resolution.

Workaround:

Once you see the message  upgrade failed at stage JRE_UPGRADE_STAGE in /home/vrack/lcm/logs/lcm.log on the SDDC Manager Controller VM, you can use the follwing steps to workaround the issue:

 

1. Log in to the SDDC Manager Controller VM as the root user. 
2. Issue the following command to replace the cacerts file with a backup copy created during the upgrade:

cp /usr/java/jre-vmware/lib/security/cacerts.bak /usr/java/jre-vmware/lib/security/cacerts

2. Remount the NFS mount from the SDDC Manager Utility VM by issuing the following command:

mount -o remount,nosuid,nodev,noexec /home/vrack/VMware/vRack/nfs-mount/lcm/repo

3. Open the /etc/fstab file with a text editor.

vi /etc/fstab

4. Find the line that reads

/home/vrack/VMware/vRack/nfs-mount/lcm/repo nfs defaults

and change defaults to nosuid,nodev,noexec.
 

Note: This change enforces more restrictive NFS mount permissions to make it more secure. Normally, this is done automatically in the third-party upgrade script.

5. Save and close the file by typing :wq.
6. Open the /home/vrack/lcm/lcm-app/conf/application-evo.properties file with a text editor:

vi /home/vrack/lcm/lcm-app/conf/application-evo.properties

7.  Find the line that reads

lcm.upgrade.thirdparty.skip=false

and change false to true.

8. Save and close the file by typing :wq.
9.  Issue the following commands to restart the lcm service:

systemctl stop scs
systemctl stop lcm

systemctl start lcm
systemctl start scs

Note: The upgrade can be re-attempted at this point.

Additional Information

To be alerted when this article is updated, click the Subscribe to Article link in the Actions box.