After changing Syslog protocol from UDP/TCP to SSL, hosts no longer send logs to Aria Operations for Logs

Products

• VMware Aria Suite

Issue/Introduction

There is a noticeable drop in the number of log events received in Aria Operations for Logs from any host which has been switched to use SSL for its syslog protocol.

This issue may also occur if the Aria Operations for Logs certificate has been changed and is not present in the ESXi host's SSL truststore.

Symptoms may also present as the following error messages in log files detailed below:

 

Aria Operations for logs:

Filename: /storage/core/loginsight/var/runtime.log

[####-##-## ##:##:##.###+####] ["Thread-5231777"/##.#.##.## ERROR] [com.vmware.loginsight.ingestion.syslog.SyslogEventHandler] [Syslog ssltcp exception by
esxi.domain.local from port 37593]
javax.net.ssl.SSLException: readHandshakeRecord
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1117)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:398)
        at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:726)
        at sun.security.ssl.SSLSocketImpl.access$200(SSLSocketImpl.java:72)
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:809)
        at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:768)
        at org.productivity.java.syslog4j.server.impl.net.tcp.SyslogMessageReader.nextOctetMessage(SyslogMessageReader.java:91)
        at org.productivity.java.syslog4j.server.impl.net.tcp.SyslogMessageReader.nextMessage(SyslogMessageReader.java:182)

 

ESXi: 

Filename: /var/log/.vmsyslogd.err:

####-##-## ##:##:##.###+#### vmsyslog : CRITICAL] vmsyslogd daemon starting (2097817)
####-##-## ##:##:##.###+#### vmsyslog.main : ERROR ] switching to normal mode from init mode
####-##-## ##:##:##.###+#### vmsyslog.loggers.network : ERROR ] SSL certificate verification failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1108)
####-##-## ##:##:##.###+#### vmsyslog.loggers.network : ERROR ] failed to establish connection to remote syslog server OpsForLogs.domain.local:1514

Environment

VMware vRealize Log Insight 8.x

VMware Aria Operations for logs 8.x

Cause

ESXi hosts are unaware of the CA which signed the certificate presented by Operations for Logs and requires that this CA cert is added to its truststore

Resolution

1. SSH to the relevant ESXi host as the root user.
2. Run the following command to extract the Ops for Logs cert into individual cert files, replacing <OpsForLogsFQDN>:
   
   openssl s_client -connect <OpsForLogsFQDN>:1514 -showcerts 2>&1 < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'

NOTE: The previous command will generate a number of .pem files depending on the configuration of the cert presented by Aria Ops for Logs, for example: cert1.pem, cert2.pem, etc.
   If Operations for Logs is using the default self signed cert then only a single file will be generated.
   
   
3. Use the following command to find which .pem file is the valid CA certificate to import, repeat for each generated .pem file:
   
   openssl x509 -in cert1.pem -text

NOTE: The Issuer: and Subject: fields will be the same for the valid CA certificate and the X509v3 extensions: will contain CA:TRUE
   
   
4. Once you have found which file is the valid CA certificate, copy the file to the /etc/vmware/ssl/ directory, using the following command as an example, replacing <n>:
   
   cp cert<n>.pem /etc/vmware/ssl/OpsForLogsCA.pem
   
   
5. Then run the following commands to import the CA cert to ESXi's truststore and reload the syslog service

esxcli system security certificatestore add -f /etc/vmware/ssl/OpsForLogsCA.pem

esxcli system syslog reload


6. Repeat steps 1 to 5 on all ESXi hosts which are required to send syslog over SSL to this Aria Operations Deployment.

Additional Information

It is possible to add the required CA cert to the ESXi hosts using the Add-VITrustedCertificate for PowerCLI

For further information please see: Add-VITrustedCertificate