Troubleshooting LCMVIDM71077: Unable to trust the load balancer certificate assigned to vIDM

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• Replacing the certificates on vIDM cluster fails with the error LCMVIDM71007 Unable to trust load balancer's certificate
• Inventory sync from vRLCM to vIDM fails with error LCMVIDM70000

• VMware Aria Suite Lifecycle engine logs var/log/vrlcm/vmware_vrlcm.log have the following error:

2023-08-08 19:03:08.778 ERROR [pool-3-thread-16] c.v.v.l.v.d.h.VidmInstallHelper -  -- Exception occured while trusting certificate
com.vmware.vrealize.lcm.common.exception.LcmException: Error while trusting LB's certificate on the host <VIDMNODEFQDN>, failed with message : {"message":"Error installing custom certificate, refer logs for more details.","code":2,"success":false,"results":null,"resultObj":null,"fieldMessages":null,"redirectUrl":null}
	at com.vmware.vrealize.lcm.vidm.driver.helpers.VidmInstallHelper.trustCertificate(VidmInstallHelper.java:867) [vmlcm-vidmplugin-driver-8.12.0-SNAPSHOT.jar!/:?]
	at com.vmware.vrealize.lcm.vidm.driver.helpers.VidmInstallHelper.trustCertificate(VidmInstallHelper.java:824) [vmlcm-vidmplugin-driver-8.12.0-SNAPSHOT.jar!/:?]
	at com.vmware.vrealize.lcm.vidm.core.task.VidmTrustLBCertificateTask.execute(VidmTrustLBCertificateTask.java:139) [vmlcm-vidmplugin-core-8.12.0-SNAPSHOT.jar!/:?]
	at com.vmware.vrealize.lcm.automata.core.TaskThread.run(TaskThread.java:63) [vmlcm-engineservice-core-8.12.0-SNAPSHOT.jar!/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]

Environment

VMware Identity Manager 3.3.x

Cause

This issue is likely caused by one of the following scenarios:

1. Network connectivity issues with the load balancer.
2. The Signature Algorithm of the certificate is not SHA 256. Using rsassaPss as Signature Algorithm will cause this problem.
3. CA certificate or vIDM Server certificate is not configured on the load balancer.
4. Known issue in versions 3.3.7

Resolution

1. Connectivity issue:

1. SSH to the VMware Aria Suite Lifecycle appliance (formerly known as vRealize Suite Lifecycle Manager)
2. Validate you can ping the hostnames of the Load Balancers and vIDM nodes.
3. If there is no connectivity, work with your Networking team to solve this issue.

2. Certificate Signature Algorithm

1. Create new certificates using SHA 256 as Signature Algorithm instead of rsassaPss.

3. Certificate missing in the load balancer.

1. vIDM cluster is SSL terminated, therefore, it is needed to import the CA and vIDM Server in the load balancer.
2. In case of NSX-T Manager follow the next steps:
   1. Login to the NSX-T Manager.
   2. Select System > Certificates > Import. and import the vIDM Server certitficate.
   3. Repeat the same steps for the root CA certificate. 
3. Then apply the certificate to the Virtual Server:
   1. Select Networking > Load Balacing > Virtual Servers in the Virtual Server of interest select the virtual ellipsis (⋮) and select Edit
   2. Click on Configure in the SSL Configuration field.
   3. Select Client SSL and update the certificate in the field Default Certificate.  Then, expand Advanced Properties, in the field Trusted CA Certificates a ssociate the CA certificate. 
   4. Select Server SSL and replace the certificate in the field Default Certificate. Then, expand Advanced Properties, in the field Trusted CA Certificates a ssociate the CA certificate. 

4. vIDM 3.3.7 known issue.

1. VMware is aware of a known issue in versions 3.3.7. Please see the Workaround section for additional information.

Workaround:

Workaround for versions 3.3.7

Prerequisites

• You have snapshots of all the nodes in the cluster.
• You have access to root username and password.

Procedure

1. SSH to each node in the cluster as root user.
2. Run the following command on each node:

   chmod 660 /opt/vmware/horizon/workspace/webapps/ROOT/lb_rootca.pem

3. Login to VMware Aria Suite Lifecycle and trigger an inventory sync: Lifecycle Operations > Environments > globalenvironment > Trigger Inventory sync
4. Once the inventory sync is completed, request to re-trust the load balancer.