Replace expired internal certificate in Aria Operations for Logs 8.10 and lower

Products

• VMware Aria Suite

Issue/Introduction

The internal communication certificate used for internode communication in Aria Operations for Logs 8.10.x (formerly known as vRealize Log Insight) or below is expired.

UI, API, syslog and internal certificate functionalities are affected.

Few examples below :

 

• UI is not loading
• Upgrade is failing
• Worker nodes not showing in the UI

Environment

VMware vRealize Log Insight 8.6.x
VMware vRealize Log Insight 8.8.x
VMware vRealize Log Insight 8.10.x

Resolution

To resolve the issue, follow the steps below to install a new internode communication certificate.
Quick Links:

• Prerequisites
  • Generate a self-signed certificate
• Install Certificate

Prerequisites

• This workaround requires a .pem certificate file to be uploaded.
  • If you have a custom certificate, using an SCP utility like WinSCP, upload the .pem file to the /tmp directory on all nodes in the vRealize Log Insight cluster.

Note: VMware Aria Operations 8.12 and later requires Client Auth usage.  Ensure your custom certificate meets this requirement.  You can run the echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -purpose | grep 'SSL client :' command on a node with your custom certificate installed to confirm.
If the above command returns SSL client : Yes, then you can proceed to the Install Certificate section.
If the above command returns SSL client : No, then you must regenerate your custom certificate with Client Auth enabled.

• If you want to use the default certificate, you will need to Generate a self-signed certificate for use.

Generate a self-signed certificate

1. Log into the Primary node as root via SSH or Console.
2. Run the following command to generate a self-signed certificate:

openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes

Note: This command will generate a self-signed certificate that is valid for 3650 days (10 years). You may alter the -days value as needed per your organization's security requirements.

Note: When prompted by openssl, provide the required values for your company.  If you want to use the default certificate options, enter the following values:

Prompt	Value
Country	US
State Or Province	California
Locality	Palo Alto
Organization	VMware, Inc.
Organization Unit	vCenter Log Insight
Common Name	VMware vCenter Log Insight

3. Run the following command to concatenate the key and certificate into a .pem file

cat domain.key domain.crt > /tmp/cert.pem

4. Using an SCP utility like WinSCP, copy the /tmp/cert.pem file from the Primary node to the /tmp directory on the other Worker nodes in the cluster.

 

Install Certificate

Note: The following steps must be completed on each node in the vRealize Log Insight cluster.  This can be done sequentially or in parallel.

1. Download the fix_expired_cert_v2.tar file attached to this article form the Attachments section on the bottom of the article.

Note: The file will download with an alphanumeric prefix similar to 0685G00001AQoFaQal_fix_expired_cert_v2.tar. Either rename it to fix_expired_cert_v2.tar prior to uploading, or use the full filename with the prefix in the below commands.

1. Using an SCP utility like WinSCP, copy the fix_expired_cert_v2.tar file to the /tmp directory on the Primary node.
2. Log into the Primary node as root via SSH or Console.
3. Run the following command to change to the /tmp directory:

cd /tmp

5. Run the following command to extract the script from fix_expired_cert_v2.tar:

tar xf fix_expired_cert_v2.tar

6. Run the following command to update the permissions of the script:

chmod 744 fix_expired_cert.sh

7. Run the following command to run the fix_expired_cert.sh script:

./fix_expired_cert.sh pem_location

Note: Replace pem_location with the name of the certificate file from the Prerequisites that you plan to use.
Example: ./fix_expired_cert.sh /tmp/cert.pem

8. Run the following command to verify that the script completed successfully:

./fix_expired_cert.sh --verify

Example: The output will be similar to:
Cluster certificate fingerprint is: <ID_1>
Inter-node certificate fingerprint is: <ID_1>
Root certificate fingerprint is: <ID_2>
Cluster and inter-node certificates are same
Found alias with matching root fingerprint
loginsight2:04:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:34

9. Optional: Run the following command check the expiration date of the newly applied certificate to ensure the expiration date is in the future:

echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Example:

10. Run the following command to restart the loginsight service:

systemctl restart loginsight

11. Repeat steps 1-10 on all nodes in the cluster, if they have not already been run.

Additional Information

Impact/Risks:
Important: This article is not applicable to VMware Aria Operations 8.12.  Attempting to run these steps on VMware Aria Operations 8.12 may cause further harm to your cluster.  To install a custom certificate in VMware Aria Operations for Logs 8.12 and later, see Install a custom certificate in VMware Aria Operations for Logs 8.12 and Later (93768).

If a new web certificate is applied to VMware Aria Operations for Logs (formerly known as vRealize Log Insight), the agents may disconnect.  See Main SSL Functions for more information.  It is recommended to Configure the vRealize Log Insight Agent SSL Parameters.