How to fix javax.net.ssl.SSLHandshakeException: ssl_protocol_version is disabled issue in VMware Tanzu GemFire upgrade from 9.10.1/9.10.2/9.10.3/9.10.4 to 9.10.6
search cancel

How to fix javax.net.ssl.SSLHandshakeException: ssl_protocol_version is disabled issue in VMware Tanzu GemFire upgrade from 9.10.1/9.10.2/9.10.3/9.10.4 to 9.10.6

book

Article ID: 294437

calendar_today

Updated On:

Products

VMware Tanzu Gemfire

Issue/Introduction

During the upgrade process for VMware Tanzu GemFire 9.10.1 / 9.10.2 / 9.10.3 / 9.10.4 to 9.10.6, GemFire may throw this exception when incompatible SSL configurations are found between any members. Many times, the old gfsecurity.properties files are used as-is assuming that the same SSL keystores, truststores, protocols, etc. should be used on the upgraded cluster. Due to a resolved issue, this can cause issues like the ones below:

This is strictly in the context of P2P communication (between cluster members) when upgraded nodes (locators or cache servers) are joining the cluster.

Note: This exception is thrown from Java when it tries to match the specified SSL version between members, and when not found, throws the error below: 
[warn 2021/04/26 15:44:51.080 EDT kbc000100.rw.example.com <main> tid=0x1] Connection: Attempting reconnect to peer 192.0.2.0(kbc000101.rw.example.com:2258:locator)<ec><v0>:41000

[warn 2021/04/26 15:44:52.418 EDT kbc000100.rw.example.com <P2P message reader@388969b8> tid=0x3a] SSL handshake exception
javax.net.ssl.SSLHandshakeException: <<ssl_version>> is disabled
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:637)
        at sun.security.ssl.InputRecord.read(InputRecord.java:527)
        at sun.security.ssl.EngineInputRecord.read(EngineInputRecord.java:382)
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:951)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
        at org.apache.geode.internal.net.NioSslEngine.handshake(NioSslEngine.java:148)
        at org.apache.geode.internal.net.SocketCreator.handshakeSSLSocketChannel(SocketCreator.java:840)
        at org.apache.geode.internal.tcp.Connection.createIoFilter(Connection.java:1747)
        at org.apache.geode.internal.tcp.Connection.readMessages(Connection.java:1548)
        at org.apache.geode.internal.tcp.Connection.run(Connection.java:1472)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[info 2021/04/26 15:44:52.418 EDT kbc000100.rw.example.com <P2P message reader@388969b8> tid=0x3a] Failed initializing socket for message receiver: SSLv2Hello is disabled

[info 2021/04/26 15:45:51.082 EDT kbc000100.rw.example.com <main> tid=0x1] TLS handshake is timing out


Environment

Product Version: 9.10
OS: NA

Resolution

The simplest way to resolve this issue is to either comment out the ssl-protocols property in your gfsecurity.properties file, or to simply provide the value as "any" as a workaround. Then restart that member. Repeat the same process if you are getting this error for multiple members. For example:

#ssl-protocols=TLSv1.2

or

ssl-protocols=any


Important point to note

Using the value as "any" for "ssl-protocols" or for "ssl-ciphers" is not required by VMware Tanzu GemFire and you should follow the guidance of your security engineering team. The important part is that the allowed protocols must be compatible and should be consistent. Using "any" means that GemFire will let the Java SSL implementation decide the best possible matching protocol and cypher to use, using its own internal mechanism outside the control of GemFire, which is often sufficient and is less complicated.


Background

In GemFire versions 9.10.0 to 9.10.4, the ssl-protocols and ssl-ciphers properties were silently ignored for some types of communication and would behave as if the value "any" had been configured. Additionally, some other SSL/TLS related issues were found that could cause performance issues in peer-to-peer TLS communication. Hence it was recommended that all v9.10 customers update to at least to 9.10.5.
To read more, click on the following links: 
 


Additional Details (SSL debugging)

If you are interested in debugging the SSL/TLS related system settings and their impact on GemFire, please run your locator/server with the attributes below:
 

  • --J=-Djavax.net.debug=ssl:handshake (this will print SSL/TLS version getting used as well as other SSL negotiation details)
  • --redirect-output (this is required so that GemFire can capture stdout and stderr in the locator/server log files)


​​​For example:

gfsh> start server --name=server --security-properties-file=example_security.properties --locators="127.0.0.1[10334]" --J=-Djavax.net.debug=ssl:handshake --redirect-output



Powered by Wolken Software