Admin fails to load due to expired keystore certificates

Products

• VMware Aria Operations (formerly vRealize Operations) 8.x
• VMware Telco Cloud Infrastructure

Issue/Introduction

In VMware Aria Operations for Logs (formerly vRealize Log Insight), the cluster user interface (UI) fails to load, throwing connection or loading errors when accessed via the primary node IP address or FQDN.

Running the following validation commands over SSH reveals that the internal keystore validity period has expired:

1. Non-FIPS enabled cluster:
   
   

   # echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

   
   
   
2. FIPS enabled cluster:
   
   

   # /opt/vmware/bin/keytool-no-provider -list -keystore /usr/lib/loginsight/application/3rd_party/apache-tomcat/conf/keystore.bcfks -storepass $(/usr/lib/loginsight/application/sbin/fips.sh --keystore_password) -rfc 2>/dev/null | openssl x509 -noout -enddate

Environment

TCI: 2.2

vRealize Operations for Logs: 8.1.1

Cause

The expiration of the internal SSL application certificates stored within the keystore breaks the HTTPS binding for Apache Tomcat and invalidates mutual trust configuration requirements for the Cassandra database services.

Resolution

Disclaimer: Take snapshots following How to take a Snapshot of Operations for Logs before starting

Generate and install an updated self-signed certificate across all affected cluster nodes by performing the following steps:

1. Log into the Primary node as root via SSH or Console.
   
   
2. Generate a new self-signed certificate by executing the following command.
   
   

   # openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes

   
   
   
3. Concatenate the key and certificate into a unified PEM file
   
   

   # cat domain.key domain.crt > /tmp/cert.pem

   
   
   
4. Copy the generated /tmp/cert.pem file to all remaining nodes in the cluster using a secure transfer protocol.
   
   
5. Back up any existing custom.pem file and copy the new certificate payload to the runtime configuration directory
   
   

   # cp /tmp/cert.pem /storage/var/loginsight/apache-tomcat/conf/custom.pem

   
   
   
6. Execute the built-in script to apply the SSL modifications
   
   

   # /usr/lib/loginsight/application/sbin/custom-ssl-cerf

   
   
   
7. Relocate the updated keystore file to the application configuration directory
   
   

   # cp /usr/lib/loginsight/application/3rd_party/apache-tomcat-#.#.##/conf/keystore /usr/lib/loginsight/application/etc/3rd_config/

   
   
   
8. Validate the updated expiration date based on the cluster type:
   
   8.1. Non-FIPS enabled cluster:
   
   

   # echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

   8.2. FIPS enabled cluster:
   
   

   # /opt/vmware/bin/keytool-no-provider -list -keystore /usr/lib/loginsight/application/3rd_party/apache-tomcat/conf/keystore.bcfks -storepass $(/usr/lib/loginsight/application/sbin/fips.sh --keystore_password) -rfc 2>/dev/null | openssl x509 -noout -enddate

   
   
   
9. Repeat steps 5 through 8 on all remaining nodes in the cluster.
   
   
10. Perform a rolling restart of the loginsight daemon across all nodes, one at a time.
    
    

    # systemctl restart loginsight

Note: Wait a few minutes after each restart and confirm operational status with service loginsight status before proceeding to the next node.

# systemctl status loginsight

Additional Information

Install a self-signed certificate in VMware Aria Operations for Logs 8.12 and Later