Error: Failed/Unable to trust load balancer's certificate for VIDM

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

The VIDM certificate shows up to date; however, while trying to re-trust the vIDM with the load balancer, it fails with the following error:

Error Code: LCMVIDM71092
Failed to trust load balancer's certificate. Ensure load balancer has proper root certificate or provide the root certificate chain as retry param 'vidmLBRootCertificateChain' and try again.
Unable to fetch root/intermediate CA certificates from the certificate chain provided. 
Failed to trust vIDM load balancer certificate. 
Retry by providing the root or intermediate CA certificate chain

Inventory sync from Aria Suite Lifecycle Manager to vIDM fails with error LCMVIDM71077 after replacing certificates on vIDM node(s):

Error Code LCMVIDM71077
Unable to trust load balancer's certificate. Refer to the log for additional details and retry. 
Error while getting login token from vIDM

VMware Aria Suite Lifecycle engine logs (/var/log/vrlcm/vmware_vrlcm.log) have the following error:

YYYY-MM-DD HH:MM:SS.### ERROR [####-#-######-##] c.v.v.l.v.d.h.VidmInstallHelper -  -- Exception occured while trusting certificate
com.vmware.vrealize.lcm.common.exception.LcmException: Error while trusting LB's certificate on the host <VIDMNODEFQDN>, failed with message : {"message":"Error installing custom certificate, refer logs for more details.","code":2,"success":false,"results":null,"resultObj":null,"fieldMessages":null,"redirectUrl":null}
    at com.vmware.vrealize.lcm.vidm.driver.helpers.VidmInstallHelper.trustCertificate(VidmInstallHelper.java:867) [#####-##########-######-#.12.#-########.jar!/:?]
    at com.vmware.vrealize.lcm.vidm.driver.helpers.VidmInstallHelper.trustCertificate(VidmInstallHelper.java:824) [#####-##########-######-#.12.#-########.jar!/:?]
    at com.vmware.vrealize.lcm.vidm.core.task.VidmTrustLBCertificateTask.execute(VidmTrustLBCertificateTask.java:139) [#####-##########-####-#.12.#-########.jar!/:?]
    at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:63) [#####-#############-####-#.12.#-########.jar!/:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]

Aria Suite Lifecycle Manager /var/log/vrlcm/vmware_vrlcm.log shows the following error related to Failed to establish ssl handshake with server VIDMNODEFQDN:8443:

YYYY-MM-DDTHH:MM:SS.###Z INFO vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  --  Sleeping and retrying after 10000 milliseconds...
YYYY-MM-DDTHH:MM:SS.###Z ERROR vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  -- Exception caught while handshake : java.net.SocketTimeoutException: Read timed out
YYYY-MM-DDTHH:MM:SS.###Z ERROR vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  -- Failed to establish ssl handshake with server VIDMNODEFQDN:8443. Retrying for 59 minutes...
YYYY-MM-DDTHH:MM:SS.###Z INFO vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  --  Sleeping and retrying after 10000 milliseconds...
YYYY-MM-DDTHH:MM:SS.###Z INFO vrlcm[1171] [http-nio-8080-exec-7] [c.v.v.l.s.n.s.NotificationServiceImpl]  -- Authentication object is not null org.springframework.security.authentication.UsernamePasswordAuthenticationToken@: YXYXYXYX org.springframework.security.core.userdetails.User########: Username: ########; Password: YXYXYXYX Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: LCM_ADMIN; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@957e: RemoteIpAddress: ###.#.#.#; SessionId: null; Granted Authorities: LCM_ADMIN
YYYY-MM-DDTHH:MM:SS.###Z ERROR vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  -- Exception caught while handshake : java.net.SocketTimeoutException: Read timed out
YYYY-MM-DDTHH:MM:SS.###Z ERROR vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  -- Failed to establish ssl handshake with server VIDMNODEFQDN:8443. Retrying for 59 minutes...
YYYY-MM-DDTHH:MM:SS.###Z INFO vrlcm[1171] [pool-3-thread-12] [c.v.v.l.v.c.t.u.VidmInstallTaskUtil]  --  Sleeping and retrying after 10000 milliseconds...

Environment

VMware Identity Manager 3.3.x

Cause

This issue is likely caused by one of the following scenarios:

Network connectivity problems between the load balancer
The certificate’s Signature Algorithm is not set to SHA-256. Using RSASSA-PSS as the algorithm can lead to this issue
The CA certificate or vIDM server certificate is not properly configured on the load balancer
A known issue in version 3.3.7

Resolution

There are multiple possible causes, please see resolutions below.

Connectivity issue:

SSH to the VMware Aria Suite Lifecycle appliance (formerly known as vRealize Suite Lifecycle Manager).
Validate you can ping the hostnames of the Load Balancers and vIDM nodes.
If there is no connectivity, work with your Networking team to solve this issue.

Certificate Signature Algorithm

Create new certificates using SHA 256 as Signature Algorithm instead of rsassaPss.

Certificate missing in the load balancer

The vIDM cluster is SSL terminated, and you need to import the CA and vIDM server certificates in the load balancer.

For NSX-T Manager, follow the steps below:

Add server and CA certificates to NSX-T

Log in to the NSX-T Manager.
Go to System > Certificates > Import, and import the vIDM Server certificate.
Repeat step 2 above for the root CA certificate.

Apply the certificate(s) to the Virtual Server

Go to Networking > Load Balancing > Virtual Servers.
Select the vertical ellipsis (⋮) for the virtual server used for vIDM, and select Edit.
Click on Configure in the SSL Configuration field.
Select Client SSL tab, and update the certificate in the field Default Certificate.
Expand Advanced Properties, in the Trusted CA Certificates field add the CA (root) certificate.
Select Server SSL tab and replace the certificate in the Default Certificate field.
Expand Advanced Properties, in the Trusted CA Certificates field add the CA (root) certificate.
Save the SSL configuration.

Apply the certificate to the active monitor for the server pool

Go to Networking > Load Balancing > Server Pools.
Select the desired server pool, then click on the hyperlink after Active Monitor.
Select the vertical ellipsis (⋮), and select Edit, then go to SSL Configuration and click Configure.
Update the certificate in the Client Certificate field with the new certificate.
Expand Advanced Properties, in the Trusted CA Certificates field add the CA (root) certificate.
Save the SSL configuration.

Note: If you are not replacing the CA certificate you can leave the Trusted Root CA Certificate configuration with the default setting.

vIDM 3.3.7 known issue

VMware is aware of a known issue in versions 3.3.7. Please see the Workaround section for additional information.

Workaround for versions 3.3.7

Prerequisites

You have snapshots of all the nodes in the cluster.
You have access to root username and password.

Procedure

SSH to each node in the cluster as root user.

Run the following command on each node:

chmod 660 /opt/vmware/horizon/workspace/webapps/ROOT/lb_rootca.pem

Log in to VMware Aria Suite Lifecycle and trigger an inventory sync: Lifecycle Operations > Environments > Global Environment > Trigger Inventory sync.
Once the inventory sync is completed, request to re-trust the load balancer.

Additional Information

Note: When the certificate changes (e.g., due to renewal or re-issuance), the thumbprint also changes. An outdated thumbprint prevents NSX-T from establishing a secure connection with vIDM, leading to authentication failures. To fix this issue, follow - Resolving NSX-T and VMware Identity Manager (vIDM) Connectivity Issues due to SSL Thumbprint Mismatch.