Password Management operations fail for Aria Operations for Logs (Formerly VRLI)

Products

• VMware SDDC Manager

Issue/Introduction

• SDDC Manager password remediation or any password management operations fail for vRLI with "Certificate for <FQDN> doesn't match any of the subject alternative names"
• Inventory sync completes successfully
• Passwords set on the appliance match the passwords stored in lookup_passwords
• SDDC Manager can successfully SSH to Aria Operations for Logs
• Operationsmanager.log shows the following:

2024-05-31T14:24:02.333+0000 ERROR [vcf_om,051ac695ea4569f9,4033] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-29] Certificate for <FQDN> doesn't match any of the subject alternative names: [<IP,FQDN>]
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Certificate for <FQDN> doesn't match any of the subject alternative names: [<IP,FQDN>]
	at com.vmware.vcf.passwordmanager.update.changers.VrliApiChanger.loginTest(VrliApiChanger.java:159)
	at com.vmware.vcf.passwordmanager.update.changers.VrliApiChanger.doTest(VrliApiChanger.java:71)
	at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:429)
	at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:198)
	at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
	at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
	at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <FQDN> doesn't match any of the subject alternative names: <IP,FQDN>]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.ava:236)
	at brave.httpclient.TracingMainExec.execute(TracingMainExec.java:65)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at brave.httpclient.TracingProtocolExec.execute(TracingProtocolExec.java66)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at com.vmware.vcf.secure.http.HttpClientService.getHTTPResponse(HttpClientService.java:1073)
	at com.vmware.vcf.secure.http.HttpClientService.getHTTPResponse(HttpClientService.java:972)
	at com.vmware.vcf.secure.http.HttpClientService.getHTTPResponseStatus(HttpClientService.java:741)
	at com.vmware.vcf.secure.http.HttpClientService.getHTTPResponseCode(HttpClientService.java:538)
	at com.vmware.vcf.secure.http.HttpClientService.getHTTPResponseCode(HttpClientService.java:560)
	at com.vmware.vcf.passwordmanager.update.changers.VrliApiChanger.loginTest(VrliApiChanger.java:140)
	. . . 10 common frames omitted
2024-05-31T14:24:02.334+0000 DEBUG [vcf_om,051ac695ea4569f9,4033] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-29] Error Message : Certificate for <FQDN> doesn't match any of the subject alternative names: [<IP,FQDN>], Error Token : UMLTQN, Error Cause : {}
2024-05-31T14:24:02.520+0000 DEBUG [vcf_om,051ac695ea4569f9,4033] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-29] About to mark resource state as error. . . 
2024-05-31T14:24:02.535+0000 DEBUG [vcf_om,051ac695ea4569f9,4033] [c.v.v.p.r.AbstractPasswordTransactionExecutor,om-exec-2] Password operations failed for admin

Environment

VMware Cloud Foundation 5.x
Aria Operations for Logs 8.12 and later

Cause

The Aria Operations for Logs certificate does not contain the FQDNs and IPs for the VIP and all three vRLI nodes.

Resolution

1. Take a snapshot of vRLI and vRSLCM
2. Verify that the hostnames on all 3 vRLI nodes are set to the FQDN. You can do this by running the following command on each node: hostname. If the hostname does not show the FQDN, set it to the FQDN by running this command: hostname <FQDN>
3. Configure a certificate by following the steps here: https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/8.6/com.vmware.vrsuite.lcm.8.6.doc/GUID-C1FE1310-1D8B-425C-9B45-F1307A55CBAF.html. Ensure that the Subject Alternative Names include the FQDNs and IPs for the vRLI VIP and all 3 vRLI nodes.
4. Assign the new certificate by following the steps here: https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/8.10/com.vmware.vrsuite.lcm.8.10.doc/GUID-66D54179-D9E1-414C-B956-B21F5B0B463C.html
5. Run an inventory sync for vRLI by following the steps here: https://docs.vmware.com/en/VMware-vRealize-Suite-Lifecycle-Manager/8.10/com.vmware.vrsuite.lcm.8.10.doc/GUID-070226B9-02E1-4879-9471-F677EE0F3323.html
6. Reattempt the password management workflow from SDDC Manager