SDDC Manager password rotation for vRSLCM fails error - Unable to update vCenter password in vRSLCM.

Products

VMware Aria Suite VMware Cloud Foundation

Issue/Introduction

SDDC Manager password rotation for vrslcm fails with below error.
```
Unable to update vCenter password in vRSLCM.
```
Error in SDDC Manager UI

Error in /var/log/vmware/vcf/operationsmanager/operationsmanager.log

ERROR [vcf_om,e02dbb963005dc7f,b7c1] [c.v.v.p.u.d.VrslcmVcenterPasswordUpdater,om-exec-13] Failed to update sso credentials in vRSLCM
com.vmware.evo.sddc.common.vrealize.vrlcm.VrlcmException: Failed to update vCenter[{"messageId":"UNKNOWN_SYSTEM_ERROR",
"message":"Unknown system error.","eventId":"3e83582a-7ee7-44e7-90d6-fd330e470973","retry":true,
"exceptionMessage":null,"exceptionStackTrace":"java.lang.NullPointerException
at java.base/java.lang.String.contentEquals(Unknown Source)
at com.vmware.vrealize.lcm.drivers.commonplugin.task.UpdateVcenterDetailsTask.execute(UpdateVcenterDetailsTask.java:104)
at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:63)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)","localizedMessageId":null,"parameters":null,"properties":{}}]
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.updateVcenter(VrslcmService.java:881)
        at com.vmware.evo.sddc.common.vrealize.vrslcm.VrslcmService.updateVcenterCredentials(VrslcmService.java:892)
        at com.vmware.vcf.passwordmanager.update.dependents.VrslcmVcenterPasswordUpdater.update(VrslcmVcenterPasswordUpdater.java:139)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdaters(AbstractPasswordChanger.java:993)


ERROR [vcf_om,e02dbb963005dc7f,b7c1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-13] Unable to update vCenter password in vRSLCM
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Unable to update vCenter password in vRSLCM
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateDependantError(AbstractPasswordChanger.java:935)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdaters(AbstractPasswordChanger.java:1007)


Caused by: com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Unable to update vCenter password in vRSLCM
        at com.vmware.vcf.passwordmanager.update.dependents.VrslcmVcenterPasswordUpdater.update(VrslcmVcenterPasswordUpdater.java:180)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdaters(AbstractPasswordChanger.java:993)
        ... 9 common frames omitted

DEBUG [vcf_om,e02dbb963005dc7f,b7c1] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-13] 
Error Message : Unable to update vCenter password in vRSLCM, Error Token : E3T6ST, Error Cause : {}

vRSLCM locker has the below error when attempting to view passwords

Failed to validate root password. Provide correct root password to authorize this action.

Environment

VMware Cloud Foundation 4.x
VMware Cloud Foundation 5.x

Cause

This issue occurs when the vRSLCM upgrade is rolled back from version 8.10 to version 8.8, which causes a discrepancy in the inventory version information between the vRSLCM instance and the SDDC Manager.

Resolution

Take Snapshot of SDDC Manager VM
SSH to SDDC Manager with vcf user and su to root

Connect to platform database

psql -h localhost -U postgres -d platform

for 5.1.1

/usr/pgsql/13/bin/psql -h localhost -U postgres -d platform

Check the vrslcm table

select id,status,version from vrslcm;

Sample output

id 										| status 		| version
f79c67a0-92fb-471e-9a48-172267801431	  ACTIVE		8.8.2-20080494

Update vRSLCM version to the correct version

update vrslcm set version='8.8.2.3-20080494';

Additional Information

Impact/Risks:

HIGH: Since the resolution involves manual edits to the SDDC Manager database entries, a snapshot of the SDDC Manager VM is highly recommended prior to making any changes.