Workaround for 'javax.net.ssl.SSLException: org.bouncycastle.tls.TlsFatalAlert' exception in vRealize Suite Lifecycle Manager 8.x versions
search cancel

Workaround for 'javax.net.ssl.SSLException: org.bouncycastle.tls.TlsFatalAlert' exception in vRealize Suite Lifecycle Manager 8.x versions

book

Article ID: 312292

calendar_today

Updated On:

Products

VMware Aria Suite

Issue/Introduction

This KB provides a workaround for the exception 'javax.net.ssl.SSLException: org.bouncycastle.tls.TlsFatalAlert' seen in vRealize Suite Lifecycle Manager application log (log file name is 'vmware_vrlcm.log, located at /var/log/vrlcm inside vRealize Suite Lifecycle Manager VM). The issues mentioned here can be observed in vRealize Suite Lifecycle Manager 8.x versions where connecting vRealize Cloud Universal to vRealize Cloud Subscription Manager is allowed.

Symptoms:
The exception is seen in case when no usage is sent from vRealize Suite Lifecycle Manager to vRealize Cloud Subscription Manager(vRCSM) for a vRealize Cloud Universal(vRCU) license key, even when the vRCU keys are connected to vRCSM.

Environment

VMware vRealize Suite Lifecycle Manager 8.x

Cause

If VMware's self signed certificate is child of customer's internal certificate, which is not applied on LCM server and hence the traffic is blocked from vRSLCM to vRCU. For confirmation of above statement, perform the troubleshooting steps mentioned below.

1) Verify vRSLCM Logs from "/var/log/vrlcm/vmware_vrlcm.log"- if following is seen in exception for SSL Certificate
javax.net.ssl.SSLException: org.bouncycastle.tls.TlsFatalAlert: certificate_unknown(46)

2) Login to postgres inside vRSLCM and validate the URL it is routed from DB using below command
su postgres // command to login to postgress
select * from vm_engine_property where key='lcm.flex.lemans.gateway.uri' // command to get the URL

The value should be https://data.mgmt.cloud.vmware.com

3) curl -i https://data.mgmt.cloud.vmware.com/le-mans/health - shall with below error
~[ /opt/vmware/vpostgres/current/bin ]# curl -i https://data.mgmt.cloud.vmware.com/le-mans/health
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

4) curl -i https://data.mgmt.cloud.vmware.com/le-mans/health Or curl https://data.mgmt.cloud.vmware.com/le-mans/health fails in customers environment hinting that the url 'data.mgmt.cloud.vmware.com' is not white listed in their organisations firewall or there is some other issue hindering the call to complete successfully.
A sample of a successful call is shared as attachment 'curl-success-example.txt'. If the actual result deviates from it, then customer needs to re-visit their network settings to ensure this curl commands succeeds.

If all the steps produce results as shown above, follow below steps to resolve it:
 

Resolution

We need to add root CA (of the customer's internal proxy/gateway server) in the Aria Suite Lifecycle JVM keystore. This solution is only required when we want to connect an Aria Universal or 

1) The user needs to check their firewall configuration and confirm API calls to the url 'data.mgmt.cloud.vmware.com' can go through without any hinderences. This can be confirmed via point number (4) mentioned under troubleshooting. 

2) Take a snapshot of Aria Suite Lifecycle before ding the required changes. Use this snapshot as restore point if the changes are not effective and the appliance runs into some problems after the changes mentioned in the following steps.

3) Take Backup of existing certificate from "/etc/pki/tls/certs" OR "/etc/ssl/certs" in LCM server. These certificates can be backed up under a  user created directory in '/data', example :
[~]# mkdir /data/cert-backup
[~]# mkdir /data/cert-backup/tls
[~]# mkdir /data/cert-backup/ssl
[~]# cp -r /etc/pki/tls/certs /data/cert-backup/tls
[~]# cp -r /etc/ssl/certs /data/cert-backup/ssl

4) In a browser, open this url - https://data.mgmt.cloud.vmware.com. Once the page loads, grab the certificate chain from the browser. Every node of the certificate chain should be saved (including the root node and last leaf).
For example, if Aria Suite Lifecycle UI is accessed from a VM/machine running on Windows OS then, save all the certificate nodes in a directory like c:\Downloads\cert-chain-nodes and then use any SFTP/FTP client like WinSCP to put the directory 'cert-chain-nodes' and its contents into Aria Suite Lifecycle appliance under '/data/' directory.

5) SSH into LCM VM and go to the directory copied in the last step. Example -
[~]# cd /data/cert-chain-nodes

6) Import the certificate to JVM Keystore using below command. For each of the certificates present in the folder run the following (Default password for this is “changeit”) –

[~]# keytool -importcert -file <cert-name>.crt -keystore /usr/java/jre-vmware/lib/security/cacerts -alias “<cert-name>”


7) Validate the connectivity using curl --insecure -i https://data.mgmt.cloud.vmware.com/le-mans/health.
   Output should look like -
   HTTP/1.1 200 OK
   Date: Fri, 13 Jan 2023 11:25:40 GMT
   Content-Type: application/json
   Content-Length: 0
   Connection: keep-alive

8) Refresh the sync usage from LCM by executing the following -
   ~[/root]# curl --location --request GET ‘https://<lcm-hostname-or-IPv4>/lcm/flex/api/v1/rawusage/<license-id>’\ --header ‘Authorization: Basic <base64-encoded-string-for-vRSLCM-uilogin>’
   where -
   (a) 'license-id' is the trailing string found when viewing details of a license key in vRSLCM locker. This is found by clicking on the
       license key alias in the locker, example : https://192.168.10.1/lcm/locker/license/566233d2-0e99-412b-84fc-4646586afcc3, here the
       license id is '566233d2-0e99-412b-84fc-4646586afcc3'.
   (b) 'base64-encoded-string-for-vRSLCM-uilogin' is base64-encoded value for 'admin@local:<vrslcm-ui-password>' string.

Check vRSLCM log immediately after running this command : vi /var/log/vrlcm/vmware_vrlcm.log to see if the previous exception is visible. If the workaround has worked then the exception shall not be there.

NOTE: The changes specified in this document does not survive across patching or upgrade of Lifecycle Manager VM (unless this issue is mentioned in the patch/upgrade release note as fixed). Hence after every patching/upgrade we need to redo these steps