Internal Certificates Not Updating After Custom CA certificate Replacement in Aria Operations for Logs 8.18.x

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Internal certificates are not being updated after replacing the VMware Aria Operations for Logs certificates with custom CA-signed certificates.
  
  
• When attempting to reset or unlock the admin account for Aria Operations for Logs using the script "li-reset-admin-passwd.sh", the operation fails with the error message: "Unable to get user data. Possible Cassandra is down"
  
  
• Running the nodetool-no-pass status command reports that the Cassandra service is UP and Normal for all nodes.
  
  
• The following error in /storage/core/loginsight/var/cassandra.log reports SSL handshake exception errors, indicating disrupted SSL communication between system components:
  
  

  WARN [nioEventLoopGroup-5-11] AbstractChannelHandlerContext.java:311 - An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptio nCaught() method while handling the following exception:
  io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca
  
  Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca

   

• Validating web certificate shows its updated from browser but internal certs are not getting updated
  
  
  
  
• Validating the expiry date by running the below command shows that current internal certificate is expired
  

  echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

  
  

Environment

VMware Aria Operations for Logs 8.18.x

Cause

This issue occurs when the CA certificate lacks the "SSL Client" purpose, causing the internal certificate replacement operation to fail.

Resolution

To work around this issue, perform the following steps:

1. Take non-memory snapshots of the Aria Operations for Logs cluster by following the KB : 383191
   
   
2. Download the existing CA certificate that is currently installed on the Aria Operations for Logs cluster. If Aria Operations for Logs is integrated with Aria Suite Lifecycle, download the CA certificate from the Locker:
   
   Steps to get custom certificates from VMware Aria Suite Lifecycle:
   
   
   1. Log in to the VMware Aria Suite Lifecycle user interface.
   2. Select Locker from the main dashboard.
   3. In the left-hand navigation pane, select Certificates.
   4. Locate the Aria Operations for Logs certificate, click the vertical ellipses (options/action menu), and select Download.
      
      
3. Generate and install a self-signed SSL certificate by following Broadcom KB 315949
   
   (Optional) After the self-signed certificate has been installed successfully, take another non-memory snapshot of the Aria Operations for Logs cluster
   
   
4. Replace default.pem with the newly generated certificate:
   
   cp /tmp/cert.pem /usr/lib/loginsight/application/etc/certs/default.pem
   
   
5. Copy the existing CA certificate to custom.pem:
   
   cp /tmp/existing_AriaOpSforLogs_CA_cert.pem /usr/lib/loginsight/application/etc/certs/custom.pem
   
   
6. Run the custom SSL certificate script:
   
   /usr/lib/loginsight/application/sbin/custom-ssl-cerf
   
   
7. Copy default.pem to cacert.pem:
   
   cp /usr/lib/loginsight/application/etc/certs/default.pem /storage/core/loginsight/cidata/cassandra/config/cacert.pem
   
   
8. Restart the loginsight service on all nodes, one node at a time:
   
   systemctl restart loginsight
   
   
9. Perform step 4 to 8 on all the nodes.
   
   
10. If the admin account remains locked even after completing the above steps, follow the KB 339878 to unlock the account.