Upgrading vRSLCM from SDDC Manager fails due to SSH Password expiry

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

•  During vRealize Suite Lifecycle Manager upgrade operation through SDDC Manager, the workflow fails at deploying the vRSLCM appliance.

• You see messages similar to the following in /var/logs/vmware/vcf/lcm/lcm.log file on the SDDC Manager VM.

2020-01-23T10:38:52.309+0000 ERROR [cce734187b390fbf,8c79,upgradeId=5b0fa1b6-b433-438d-8ff4-0b1238e2ab2c,resourceType=VRSLCM,resourceId=d234c21c-723c-4aa6-b7b9-5e18dcbabe0e,bundleElementId=19195ea8-9eca-4981-a872-bc4d342737de] [c.v.e.s.v.v.DeployVrslcmContract,ThreadPoolTaskExecutor-3] Could not SSH to vRSLCM!
com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to vrslcm.vrack.vsphere.local
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:145)
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:88)
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:103)
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuterFactory.createSshCommandExecuter(SshCommandExecuterFactory.java:25)
        at com.vmware.evo.sddc.vrealize.vrslcm.DeployVrslcmContract.checkSshConnection(DeployVrslcmContract.java:233)
        at com.vmware.evo.sddc.vrealize.vrslcm.DeployVrslcmContract.execute(DeployVrslcmContract.java:146)
        at com.vmware.evo.sddc.lcm.primitive.impl.vrslcm.VrslcmActionHelper.install(VrslcmActionHelper.java:143)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.7.x
VMware Cloud Foundation 3.8.x

Cause

This is happening due to an expired password for the 'root' user on vRSLCM VM which prevents the first-boot scripts in vRSLCM to complete successfully.

Resolution

This is a known issue affecting the VMware Cloud Foundation upgrades in 3.x. It is resolved in a new patch bundle containing a patched vRSLCM build. Use the steps below in order to download and implement the latest vRSLCM upgrade bundle. 

1. Navigate to the SDDC Manager UI > Repository > Bundles and check for an available vRSLCM upgrade bundle. Download the vRSLCM upgrade bundle.
2. Open an SSH session to the SDDC Manager VM and switch to root user:

su -

3. Ensure that the latest vRSLCM bundle is downloaded by running the following command and verifying the new bundle is available:

curl '127.0.0.1/lcm/images?productType=VRSLCM&imageType=PATCH' | json_pp

The contents will be similar to the below:
{
    "productType": "VRSLCM",
    "version": "2.0.0-11187327",
    "imageType": "PATCH",
    "imagePath": "/nfs/vmware/vcf/nfs-mount/bundle/c2d9b61a-1695-4a09-8560-a7e22f4deb39/bundle-9924/vrslcm_patch",
    "fromVersion": "2.0.0-10150522",
    "size": 4733030540,
    "checksum": "aee5e713bcc34441af86cf8aa6e9e2e4910264e54c7a2ace7d14421fbe7dff3a"
  },
  {
    "productType": "VRSLCM",
    "version": "2.0.0-18023742", 
    "imageType": "PATCH",
    "imagePath": "/nfs/vmware/vcf/nfs-mount/bundle/f8b074ae-0137-4dcc-8e7a-7c39492a670f/bundle-20101/vrslcm_patch",
    "fromVersion": "2.0.0-18023741", 
    "size": 4700713100,
    "checksum": "b00d0d67f7e93965b7c7f63315eec143d28b275257aa1fc27c5f1708737b048f"
  }

•  For 3.7.x the version should match: "version": "2.0.0-18023742".

•  For 3.8.x and above, the version should match: "version": "2.1.0-19716566"

4. Execute the following command as root on the SDDC Manager shell console for the VCF version in use:

•    For 3.7.x: 
psql -h /home/postgresql/ -U platform -c "update vrslcm set version = '2.0.0-18023741';"

•    For 3.8.x:  
psql -h localhost -U postgres -d platform -c "update vrslcm set version = '2.0.0-15341636';"
 

6. Refresh the Updates / Patches page and ensure the vRealize Suite Lifecycle Manager upgrade bundle details now show the following info: 

•    For 3.7.x: 
Required Version: 2.0.0-18023741
Update to Version: 2.0.0-18023742

•    For 3.8.x and above:
Required Version: 2.0.0-15341636

Update to Version: 2.1.0-19716566

7. Trigger the upgrade through SDDC Manager UI.