"Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub" error when entering one of the Cloud Director cell in maintenance mode from the UI
search cancel

"Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub" error when entering one of the Cloud Director cell in maintenance mode from the UI

book

Article ID: 325656

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

Symptoms:
  • When entering one of the cells in maintenance mode from the UI, the task fails with similar entry:
Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub:
javax.naming_CommunicationException [Root exception is java.rmiConnectlOException:
error during JRMP connection establishment; nested exception is:
javax.netssLSSLHandshakeException: Received fatal alert: handshake_failure]
  • JMX certificates currently used from the cells are valid and in the Trusted Certificates available on the UI under Administration> Certificate Management > Trusted Certificates
  • Connectivity between cells is working as expected checking using OpenSSL as below: 
openssl s_client -connect Cell_Primary_IP:8998  
openssl s_client -connect Cell_Primary_IP:8999
  • ​​​From /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, we have an entry similar to the following:
ERROR | ForkJoinPool.commonPool-worker-31 | CellServiceImpl | Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
        javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure] |  
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
        javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure]  
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)  
        at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)  
        at com.vmware.ssdc.backend.services.impl.CellServiceImpl.createMBeanServerConnection(CellServiceImpl.java:203)  
        at com.vmware.ssdc.backend.services.impl.CellServiceImpl.getApplicationSelectorMBean(CellServiceImpl.java:212)  
        at com.vmware.ssdc.backend.services.impl.CellServiceImpl.doUpdateStatus(CellServiceImpl.java:158)  
        at com.vmware.ssdc.backend.services.impl.CellServiceImpl.updateStatus(CellServiceImpl.java:132)at com.vmware.vcloud.api.rest.openapi.impl.cell.CellsApiHandler.lambda$updateCell$10(CellsApiHandler.java:262)  
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)  
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1728)  
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)  
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)  
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)  
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)  
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)  
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
        javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure]  
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)  
        at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)  
        at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)  
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)  
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)  
        at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)  
        ... 13 more  
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
        javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:300)          at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)  
        at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)  
        at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)  
        at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)  
        ... 18 more  
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)  
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)  
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)  
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)  
        at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)          at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)          at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)  
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)          at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:922)          at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:1292)  
        at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)          at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142)  
        at java.base/java.io.DataOutputStream.flush(DataOutputStream.java:123)  
        at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:222)          ... 22 more






Environment

VMware Cloud Director 10.x

Cause

The issue occurs when the SSL ciphers configuration is not the same on all cells.

Resolution

This issue is resolved in a future release of Cloud Director. 
To be alerted when this article is updated, click Subscribe.​

To resolve the issue without upgrading, follow the actions below:
  • To check the SSL cipher enabled on each cell run the cell-management-tool as below:
/opt/vmware/vcloud-director/bin/cell-management-tool ciphers -l
  • To confirm the SSL ciphers configuration on each cell,it is possible to check the /opt/vmware/vcloud-director/etc/global.properties file.
For example: ssl.ciphers.disallowed = TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
  • Factory default ciphers are: 
* TLS_AES_256_GCM_SHA384  
* TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384  
* TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
The ciphers can be reset to the factory default settings on each cell by running the cell-management-tool as below:
/opt/vmware/vcloud-director/bin/cell-management-tool ciphers -r
 


Additional Information

Impact/Risks:
Entering into maintenance mode fails to carry out any upgrade or downtime tasks.

Powered by Wolken Software