Expired certificate in VMware Aria Operations for Logs 8.12 and Later

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Starting in VMware Aria Operations for Logs 8.12 there is a new script (/usr/lib/loginsight/application/sbin/custom-ssl-cerf) shipped with the appliance which can be used to change the UI, API, syslog and internal certificate.

Following common symptoms :

UI is unavailable after rebooting the cluster or upgrading from 8.12 to a later version
Running the below command on each cluster node, the certificate shows expired :

echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Cassandra service is down on a node and the node shows disconnected in cluster.
An error as below could be seen in the /storage/core/loginsight/var/cassandra.log:

Caused by: java.security.cert.CertificateExpiredException: NotAfter: {Date/Time in the past}

Receiving an alert from our Aria Ops for Logs saying the certificate expiring soon.

Environment

VMware Aria Operations for Logs 8.12.x and later.

Resolution

Prerequisites

This workaround requires a .pem certificate file to be uploaded.
If you have a custom certificate, using an SCP utility like WinSCP, upload the .pem file to the /tmp directory on all nodes in the VMware Aria Operations for Logs cluster.

Note: VMware Aria Operations for Logs 8.12 build 21618456 requires Client Auth usage. Build 21696970 and later drop this requirement. If you are on build 21618456, ensure your custom certificate meets this requirement.

You can run echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -purpose | grep 'SSL client :' on a node with your custom certificate installed to confirm.

- - If the above command returns SSL client : Yes, then you can proceed to the Install Certificate section.
  - If the above command returns SSL client : No, then you must regenerate your custom certificate with Client Auth enabled. Client Auth should be set in extendedKeyUsage on the Certificate Authority (CA) side. So please contact CA when signing a custom certificate to make sure "Client Auth" is enabled.
See Install a Custom SSL Certificate for more details on certificate requirements, or follow the below steps to generate a self-signed certificate.

Disclaimer: Creating custom certificates for use in VCF Operations for Logs is out of scope for support. For additional support in creating a custom certificate for use in VCF Operations for Logs, contact your Account manager or Technical account manager.

Generate a self-signed certificate

Log into the Primary node as root via SSH or Console.
Run the following command to generate a self-signed certificate:
```
openssl req -newkey rsa:2048 -keyout domain.key -x509 -days 3650 -out domain.crt -nodes
```
Note: This command will generate a self-signed certificate that is valid for 3650 days (10 years). You may alter the -days value as needed per your organization's security requirements.

Note: When prompted by openssl, provide the required values for your company. If you want to use the default certificate options, enter the following values exactly:

Prompt	Value
Country	US
State Or Province	California
Locality	Palo Alto
Organization	VMware, Inc.
Organization Unit	vCenter Log Insight
Common Name	VMware vCenter Log Insight

Run the following command to concatenate the key and certificate into a .pem file, which you can then use in the next Prerequisites section:
```
cat domain.key domain.crt > /tmp/cert.pem
```
Using an SCP utility like WinSCP, copy the /tmp/cert.pem file to the /tmp directory on all other nodes in the cluster.

Install Certificate

Log into the Primary node as root via SSH or Console.
Run the following command to copy the newly generated or uploaded certificate to the following location:
```
cp /tmp/cert.pem /usr/lib/loginsight/application/etc/certs/custom.pem
```
Note: Replace /tmp/cert.pem with the path and file name of the custom certificate you uploaded in the section.
Run the following command to use the custom-ssl-cerf script:
```
/usr/lib/loginsight/application/sbin/custom-ssl-cerf
```
Note: In version 8.18.5, you may encounter a 'sed' error stating it cannot read custom.pem. This is a known cosmetic issue; you can proceed to the next step to verify that the certificate change was actually successful. ( sed: can't read /usr/lib/loginsight/application/etc/certs/custom.pem: No such file or directory )

Validate the certificate will expire in the future date:

echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate

Repeat steps 2 to 4 on all remaining nodes in the cluster.
Run the following command to restart the loginsight service on all nodes, one at a time:
```
systemctl restart loginsight
```
Note: Once the service has restarted, wait a few minutes, check that the loginsight service is running by running the command service loginsight status, then proceed to restart the service on the next node.

Note: If the environment is running in FIPS mode, additional steps are needed to complete the certificate replacement: Aria Operations for Logs UI running in FIPS mode is not accessible after SSL certificate expired.

Additional Information

Openssl is installed on default VMware virtual appliance, please make sure you generate CSR and key based on your requirement.

Generate a CSR and key

VMware Aria Operations for Logs 8.12 以降における期限切れ証明書