Rotate the expiring or expired certificates in VMware GemFire
Article ID: 294355
Updated On:
Products
VMware Tanzu Gemfire
Issue/Introduction
How can an expired/expiring certificate be replaced by a new certificate in VMware GemFire?
One of the many problems that can result from an expired certificate is that peers or client apps will not able to communicate with VMware GemFire servers due to the expired certificate, in which case you may see logs like the following (Note the bad_certificate exception):
One of the many problems that can result from an expired certificate is that peers or client apps will not able to communicate with VMware GemFire servers due to the expired certificate, in which case you may see logs like the following (Note the bad_certificate exception):
“[error 2020/06/01 12:45:33.924 EDT <main> tid=0x1] Unexpected problem starting up membership services org.apache.geode.distributed.internal.tcpserver.LocatorCancelException: Unable to form SSL connection, caused by javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:293) at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:184) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave$TcpClientWrapper.sendCoordinatorFindRequest(GMSJoinLeave.java:1284) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.findCoordinator(GMSJoinLeave.java:1176) at org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.join(GMSJoinLeave.java:324) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.join(GMSMembershipManager.java:654) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.joinDistributedSystem(GMSMembershipManager.java:737) at org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:155) at org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:108) at org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:95) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:780) at org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:901) at org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:540) at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:756) at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135) at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3000) at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:251) at org.apache.geode.internal.cache.InternalCacheBuilder.createInternalDistributedSystem(InternalCacheBuilder.java:372) at org.apache.geode.internal.cache.InternalCacheBuilder.lambda$create$1(InternalCacheBuilder.java:170) at java.util.Optional.orElseGet(Optional.java:267) at org.apache.geode.internal.cache.InternalCacheBuilder.create(InternalCacheBuilder.java:170) at org.apache.geode.cache.CacheFactory.create(CacheFactory.java:142) at org.apache.geode.distributed.internal.DefaultServerLauncherCacheProvider.createCache(DefaultServerLauncherCacheProvider.java:52) at org.apache.geode.distributed.ServerLauncher.createCache(ServerLauncher.java:887) at org.apache.geode.distributed.ServerLauncher.start(ServerLauncher.java:803) at org.apache.geode.distributed.ServerLauncher.run(ServerLauncher.java:732) at org.apache.geode.distributed.ServerLauncher.main(ServerLauncher.java:251) Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127) at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761) at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124) at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1152) at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1280) at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1190) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:369) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) at sun.security.ssl.Handshaker.process_record(Handshaker.java:965) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379) at org.apache.geode.internal.net.SocketCreator.configureClientSSLSocket(SocketCreator.java:1096) at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:877) at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:839) at org.apache.geode.internal.net.SocketCreator.connect(SocketCreator.java:828) at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:290) ... 26 more ”
Environment
Product Version: 9.9
Resolution
There are four possible cases depending upon whether you need to update the trust-stores as well, and whether the certificate has already expired or not.
| Keep Existing Trust-store | Needs New Trust-store | |
|---|---|---|
| Expiring Certificate | 1 or 2 | 1 + trust-store or 3 |
| Expired Certificate | 1 or 2 | 1 + trust-store |
Below are the details for each option:
- Shutdown entire cluster and replace the certificates/key-store, and trust-store if necessary, on every member including locators and servers. Then restart the cluster.
- Bring one member down preferably a locator first, replace the certificates/key-store and restart the member. Repeat this process until all members of the cluster are updated with the new certificate.
- First, bring one member down preferably a locator, replace the trust-store with a trust-store containing both the old and new public certificates and restart the member and repeat until all members are updated with the new trust-store. Next, bring one member down preferably a locator, replace the certificate/key-store and restart the member and repeat until all members are updated with the new certificate. Finally and optionally, do another rolling restart to deploy a trust-store with only the new certificate.
Feedback
Yes
No
Powered by
