Error: "The Site cannot be reached , Connections to port 443 refused." when logging into Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Unable to log in using the IPs & FQDN of the nodes or the Internal Load Balancer address (VIP)
  
  
• Network connectivity (ICMP ping/SSH) is functional, but nodes are not listening on TCP port 443.
  
  
• /storage/core/loginsight/var/runtime.log shows the following errors:

["s3-admin-0"/<node_IP> INFO] [com.datastax.oss.driver.internal.core.time.Clock] [Could not access native clock (see debug logs for details), falling back to Java system clock]
["s3-io-0"/<node_IP> INFO] [com.datastax.oss.driver.internal.core.protocol.Lz4Compressor] [[s3] Using LZ4Factory:JavaUnsafe]
["s3-admin-1"/<node_IP> WARN] [com.datastax.oss.driver.internal.core.control.ControlConnection] [[s3] Error connecting to Node(endPoint=<node_ip>:9042, hostId=null, hashCode=799e25fd), trying next node (AnnotatedConnectException: Connection refused: /<node_IP>:9042)]
["DaemonCommands-thread-1"/<node_IP>WARN] [org.apache.thrift.server.TThreadPoolServer] [SocketException occurred during processing of message.]
java.net.SocketException: Broken pipe (Write failed)
   at sun.security.ssl.SSLSocketOutputRecord.flush(Unknown Source) ~[?:?]
        at sun.security.ssl.HandshakeOutStream.flush(Unknown Source) ~[?:?]
        at sun.security.ssl.CertificateRequest$T12CertificateRequestProducer.produce(Unknown Source) ~[?:?]
        at sun.security.ssl.SSLHandshake.produce(Unknown Source) ~[?:?]

or

["DaemonCommands-thread-1"/#.#.#.# ERROR] [org.apache.thrift.server.TThreadPoolServer] [Thrift Error occurred during processing of message.]
org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_expired

• In /storage/core/loginsight/var/cassandra.log

ERROR [Messaging-EventLoop-3-3]  OutboundConnectionInitiator.java:394 - Failed to handshake with peer /<node_ip>:7000( /<node_ip>:7000)
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
Caused by: java.security.cert.CertificateExpiredException: NotAfter: <END_Date>

• The following command shows that the certificate has expired:
  
  Non-FIPS enabled cluster:
  
  echo "" | keytool -list -keystore /usr/lib/loginsight/application/etc/3rd_config/keystore -rfc 2> /dev/null | openssl x509 -noout -enddate
  
  FIPS enabled cluster:
  
  /opt/vmware/bin/keytool-no-provider -list -keystore /usr/lib/loginsight/application/3rd_party/apache-tomcat/conf/keystore.bcfks -storepass $(/usr/lib/loginsight/application/sbin/fips.sh --keystore_password) -rfc 2>/dev/null | openssl x509 -noout -enddate

Environment

Aria Operations for logs / vRealize Log Insight (vRLI) 8.18.x

Cause

Aria Operations for logs certificate has expired.

Resolution

To resolve this issue, install a valid certificate by following the steps outlined in: Install a custom certificate in VMware Aria Operations for Logs 8.12 and Later.

Additional Information

 Please follow the steps below to verify your environment's FIPS status

1.Log in as root via SSH to any node in the cluster.

2.Run the following command to determine the FIPS status:

/usr/lib/loginsight/application/sbin/fips.sh --all --status

Output when FIPS is enabled:

FIPS mode check for all components.
Photon FIPS mode: activated
BouncyCastleFipsProvider in FIPS mode: on
BouncyCastelJsseProvider in FIPS mode: on
java.security keystore.type FIPS mode: on
java.security KeyManagerFactory.algorithm type FIPS mode: on
loginsight approved_only FIPS mode: activated
Apache Tomcat approved_only FIPS mode: activated
Apache Tomcat FIPS mode: on
Cassandra FIPS node: on
Internal config. fips-enabled = true