Password Rotation failing in SDDC Manager after manually updating the vRealize Log Insight passwords.

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Password Rotation is failing with the error: FAILED - SsoRotatorTask
• The certificate in use for vRealize Log Insight contains the fully qualified domain name for the cluster address but not the IP address.
• You see messages similar to the following in the /home/vrack/vrm/logs/vrack-vrm.log file on the SDDC Manager Controller VM:

 

2019-01-18 11:14:15.345 [pool-2-thread-10] INFO  [com.vmware.vrack.vrm.core.util.HttpClientService] Initializing Http client hosted with 1 threads.

2019-01-18 11:14:15.772 [pool-2-thread-10] INFO  [com.vmware.vrack.vrm.cms.updaters.LogInsightSsoUpdater] vRLI logging is disabled, updating only management vCenter in vRLI

2019-01-18 11:14:15.790 [pool-2-thread-10] INFO  [com.vmware.vrack.vrm.core.util.HttpClientService] Initializing Http client hosted with 1 threads.

2019-01-18 11:14:16.059 [pool-2-thread-10] ERROR [com.vmware.vrack.vrm.cms.updaters.LogInsightApiClient] LogInsight login API failed after 6 retries

2019-01-18 11:14:16.060 [pool-2-thread-10] ERROR [com.vmware.vrack.vrm.cms.workflows.rotators.SsoRotatorTask] ########-####-####-####-########bbe9: SsoRotatorTask:192.168.1.23: failed java.lang.IllegalStateException: could not get credential

        at com.vmware.vrack.vrm.cms.updaters.LogInsightApiClient.getCredential(LogInsightApiClient.java:76)

        at com.vmware.vrack.vrm.cms.updaters.LogInsightSsoUpdater.test(LogInsightSsoUpdater.java:87)

        at com.vmware.vrack.vrm.cms.workflows.rotators.AbstractRotatorTask.doUpdatersTest(AbstractRotatorTask.java:311)

        at com.vmware.vrack.vrm.cms.workflows.rotators.AbstractRotatorTask.run(AbstractRotatorTask.java:229)

        at com.vmware.vrack.vrm.core.engine.SimpleWorkflowEngine.executeTask(SimpleWorkflowEngine.java:550)

        at com.vmware.vrack.vrm.core.engine.SimpleWorkflowEngine.runSingleTask(SimpleWorkflowEngine.java:501)

        at com.vmware.vrack.vrm.core.engine.SimpleWorkflowEngine.access$1000(SimpleWorkflowEngine.java:47)

        at com.vmware.vrack.vrm.core.engine.SimpleWorkflowEngine$2.run(SimpleWorkflowEngine.java:311)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.lang.IllegalStateException: could not get sessionId

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 2.x

Cause

In SDDC Manager, both the Fully Qualified Domain Name and the IP address are required in the Subject Alternative Name for the vRealize Log Insight certificate.

Resolution

Update the SSL certificate for vRealize Log Insight to include both the IP address and the fully qualified domain name in the Subject Alternative Name.

Workaround:
You can work around this issue by implementing the following steps.

Note: Take a snapshot of the SDDC Manager Controller VM prior to making any changes.

1. SSH to the SDDC Manager Controller VM.
2. Issue the following command to display the relevant Log Insight data in the Cassandra database:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "select id,hostname,ipaddress,name,vipaddress,vipfqdn from vrmkeyspace.loginsight"

 

Note: You will see output similar to the following:

 

 id                                   | hostname                         | ipaddress   | name            | vipaddress  | vipfqdn
--------------------------------------+----------------------------------+-------------+-----------------+-------------+-------------------------------------
 ########-####-####-####-########9cdd | loginsight-vm-3.vcf.vxrack.local | 192.168.1.27 | loginsight-vm-3 |        null |                                null
 ########-####-####-####-########9cdd | loginsight-vm-1.vcf.vxrack.local | 192.168.1.25 | loginsight-vm-1 | 192.168.1.28 | loginsight-cluster.vcf.vxrack.local
 ########-####-####-####-########9cdd | loginsight-vm-2.vcf.vxrack.local | 192.168.1.26 | loginsight-vm-2 |        null |                                null

 

Note: Make a note of the id, vipaddress and vipfqdn values for the entry where the vipfqdn value is populated. In this example, the id value is ########-####-####-####-########9cdd, the vipaddress is 192.168.1.25 and vipfqdn value is loginsight-cluster.vcf.vxrack.local.

 

3. Issue a command similar to the following to temporarily change the vipaddress value in the vrmkeyspace.loginsight table to the fully qualified domain name instead of the IP address:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "update vrmkeyspace.loginsight set vipaddress='loginsight-cluster.vcf.vxrack.local' where id=########-####-####-####-########9cdd"

Note: Replace loginsight-cluster.vcf.vxrack.local and ########-####-####-####-########9cdd with the values noted for vipfqdn and id (respectively) in Step 2.

4. Run the password rotation again.
5. Issue a command similar to the following to change the vipaddress value in the vrmkeyspace.loginsight table back to the original IP address:

/opt/vmware/cassandra/apache-cassandra-2.2.4/bin/cqlsh -e "update vrmkeyspace.loginsight set vipaddress='192.168.1.26' where id=########-####-####-####-########9cdd"

 

Note: Replace 192.168.1.26 and ########-####-####-####-########9cdd with the values noted for vipaddress and id (respectively) in Step 2.

 

Note: The snapshot taken at the beginning of this process can be removed.