Aggregator logs shows javax.net.ssl.SSLException: Received fatal alert: certificate_expired for IP that belongs to Load Balancer
search cancel

Aggregator logs shows javax.net.ssl.SSLException: Received fatal alert: certificate_expired for IP that belongs to Load Balancer

book

Article ID: 261982

calendar_today

Updated On:

Products

Data Loss Prevention Core Package Data Loss Prevention Data Loss Prevention Endpoint Prevent

Issue/Introduction

Suddenly many DLP agents are not reporting, in Aggregator logs you observed multiple entries javax.net.ssl.SSLException: Received fatal alert: certificate_expired for IP that belongs to Load Balancer [RemoteHostAndPort=/###.###.###.###:61309]

File: EndpointPrevent\logs\debug\Aggregator16.log
Date: 14.03.2023 21:44:48
Class: com.symantec.dlp.communications.common.activitylogging.JavaLoggerImpl
Method: log
Level: WARNING
Message:  
javax.net.ssl.SSLException: Received fatal alert: certificate_expired
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:214)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1667)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1635)
    at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1801)
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1090)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:913)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:783)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1285)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:917)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.SimpleChannelHandler.messageReceived(SimpleChannelHandler.java:142)
    at com.symantec.dlp.communications.transportlayer.impl.NettyChannelEventCaptureConnectionHandler.messageReceived(NettyChannelEventCaptureConnectionHandler.java:62)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
TC - SSL handshake failed  for connection number 1515347 at 2023-03-14 09:44:48.Connection statistics:
Connection Number = 1515347
    PeerId = null
    StartTime = 2023-03-14 21:44:48.034
    Disconnected Time = Not Yet Disconnected
    Duration Of Connection In Millis = 47
                        Bytes Dequeued  Bytes Enqueued
    HTTP                             0               0
    SSL                            524           1,753
Connection specific high frequency logs for connection number = 1515347. There is no peerId information for this connection.
DateTime                 Event                                                   ReplicatorId     Num Bytes  AdditionalInformation
-----------------------  ------------------------------------------------------  ---------------  ---------  --------------------
2023-03-14 21:44:48.034  DC - Scheduling succeeded                                                        0  ScheduledToServiceInNanos=59999996439             
2023-03-14 21:44:48.034  NCE - Connected                                                                  0  
2023-03-14 21:44:48.034  TC - Connection opened                                                           0  RemoteHostAndPort=/###.###.###.###:61309              
2023-03-14 21:44:48.034  TC - Connection accepted by connection acceptor                                  0  RemoteHostAndPort=/###.###.###.###:61309              

Environment

Release: 15.8

Cause

The Detection Server certificate expired whereas it should be recreated automatically every 5 years

Resolution

Check if the Load balancer has disabled the TLS interception. If it is now enabled, should be disabled.

The direct solution was certificate renewal, shared in Article Id: 249203 How to manually renew the Endpoint Prevent server certificate used for Agent communication?

1) Run the below SQL query to pull which monitor keystore .jks files are used by which EP server:

SELECT im.monitorname, mck.keystorefilename
FROM InformationMonitor im
JOIN EndpointChannel ec ON im.informationmonitorid = ec.informationmonitorid
JOIN MonitorChannelKeystore mck ON mck.monitorchannelkeystoreid = ec.monitorchannelkeystoreid;


2) Note down the name of the monitor keystore filename for the EP detection servers for which you would like to renew the certificate. 
3) On Enforce, go to the folder where the keystores are located - on Windows, by default, this is C:\ProgramData\Symantec\DataLossPrevention\EnforceServer\<version>\keystore.
4) Stop the Symantec DLP Detection Server Controller service on Enforce.
5) Rename and backup the monitorXX_keystore_vXX.jks files which have been listed by the SQL query from point 1 as belonging to specific Endpoint detection servers. XX's here are internal ID numbers of the detection servers and versions of their keystores. Rename the file to i.e. samefilename.jks.bak. 
6) Start Symantec DLP Detection Server Controller service and wait for a couple of minutes. Enforce should create new versions of the monitor .jks files for the Endpoint detection servers whose .jks files have been backed up. I.e. if the original filename was monitor6_keystore_v3.jks, then the new file will be named monitor6_keystore_v4.jks. 
7) Once the files have been created, go to the Endpoint servers and recycle the service Symantec DLP Detection Server on each server whose monitor keystore file has been regenerated. This is required so that the detectors receive the renewed certificate. 
8) Check again the website https://EP_detection_server_FQDN_or_IP:10443 - it should now present a new certificate with new validity/expiration dates. 

Powered by Wolken Software