Renew the expired internal certificate

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

 

1. Updating the internal communication certificate maintains internode connectivity.
   
   
2. An expired SSL certificate can affect multiple services and cause issues such as:
   
   2.1. UI fails to load
   
   2.2. Upgrade does not complete
   
   2.3. Worker nodes are missing in the UI

 

NOTE: Do not execute on version 8.12 or later. Refer to the Additional Information section.

Environment

vRealize Log Insight (vRLI) versions 8.6.x, 8.8.x, and 8.10.x

Cause

The internal communication (SSL) certificate expired, disrupting secure communication between nodes. This resulted in loss of internode connectivity and led to issues.

Resolution

Set up a new internode communication certificate.

Prerequisites:

 

1. Create a self-signed certificate to serve as the default certificate.
   
   
2. For using a custom certificate:
   
   2.1. Verify that Client Auth is enabled by executing the following command on a node where the custom certificate is installed:
   
   

   # echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -purpose | grep 'SSL client :'

   
   2.2. If the command above shows "SSL client: Yes", you can proceed to the Install certification section.
   
   2.3. If it shows "SSL client: No", you need to regenerate the custom certificate with Client Auth enabled.

3. After confirmation, copy the .pem file with Client Auth enabled to the /tmp directory on all cluster nodes.

 

Create a self-signed certificate:

 

1. Access the Primary node via SSH or console as root.
   
   
2.  Create a self-signed certificate
   
   

   # openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes

    

   Note:
   
   1. This command generates a self-signed certificate valid for 3650 days (10 years). You can adjust the -days value according to your organization's security policies.

    2. When prompted by OpenSSL, enter the values required for your company. The default certificate options are as follows:

   Prompt	Value
   Country	US
   State Or Province	California
   Locality	Palo Alto
   Organization	VMware, Inc.
   Organization Unit	vCenter Log Insight
   Common Name	VMware vCenter Log Insight

   
   
   
3. Combine the key and certificate into a single .pem file.
   
   

   # cat domain.key domain.crt > /tmp/cert.pem

   
   
   
4. Transfer the /tmp/cert.pem file from the Primary node to the /tmp directory on the other Worker nodes in the cluster.

 

Install Certificate:

Note: The following steps need to be performed on every node in the vRealize Log Insight cluster. This can be done either sequentially or in parallel.

1. Download the attached fix_expired_cert_v2.tar file.

   Note: The file may download with an alphanumeric prefix, such as 0685G00001AQoFaQal_fix_expired_cert_v2.tar. You can either rename it to fix_expired_cert_v2.tar before uploading or use the full filename with the prefix in the commands below.

2. Transfer the fix_expired_cert_v2.tar file to the /tmp directory on the Primary node.

3. SSH or access the Primary node via console as root, then navigate to the /tmp directory using cd /tmp.

5. Extract the script from fix_expired_cert_v2.tar and update its permissions.

    

# tar xvf fix_expired_cert_v2.tar

# chmod 744 fix_expired_cert.sh

6. Execute the `fix_expired_cert.sh script:
   
   

   # ./fix_expired_cert.sh <pemLocation/pemName>

   
   Note: Replace <pemLocation/pemName> with the full path of the certificate file from the Prerequisites section that you intend to use.

Example: ./fix_expired_cert.sh /tmp/cert.pem

8. Verify that the script has completed successfully:
   
   

   # ./fix_expired_cert.sh --verify

    

   Example:
   
   Cluster certificate fingerprint is: <ID_1>
   Inter-node certificate fingerprint is: <ID_1>
   Root certificate fingerprint is: <ID_2>
   Cluster and inter-node certificates are same
   Found alias with matching root fingerprint
   loginsight2:##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##.##

Optional: Verify the expiration date of the newly applied certificate to ensure it is set in the future:

# echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Example:

10. Restart the Log Insight service:
    
    

    # systemctl restart loginsight

     

11. Perform steps 1–10 on all other nodes in the cluster.

Additional Information

Important: This article does not apply to VMware Aria Operations 8.12. Running these steps on version 8.12 may cause issues in your cluster. To install a custom certificate in VMware Aria Operations for Logs 8.12 and later, see Install a custom certificate in VMware Aria Operations for Logs 8.12 and later.

You can also update the SSL certificate if it is managed by vRealize Lifecycle Manager. See  Generate and replace Log Insight Certificates