"Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub" error when entering one of the Cloud Director cell in maintenance mode from the UI
search cancel

"Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub" error when entering one of the Cloud Director cell in maintenance mode from the UI

book

Article ID: 325656

calendar_today

Updated On:

Products

VMware Cloud Director

Issue/Introduction

  • When entering one of the cells in maintenance mode from the UI, the task fails with similar entry:

    Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub:
    javax.naming_CommunicationException [Root exception is java.rmiConnectlOException:
    error during JRMP connection establishment; nested exception is:
    javax.netssLSSLHandshakeException: Received fatal alert: handshake_failure]

  • JMX certificates currently used from the cells are valid and in the Trusted Certificates available on the UI under Administration> Certificate Management > Trusted Certificates

  • Connectivity between cells is working as expected checking using OpenSSL as below: 

    openssl s_client -connect Cell_Primary_IP:8998  
    openssl s_client -connect Cell_Primary_IP:8999

  • From /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, we have an entry similar to the following:

    ERROR | ForkJoinPool.commonPool-worker-31 | CellServiceImpl | Failed to transition cell status to: MAINTENANCE. Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
            javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure] |  
    java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
            ......
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)  
    Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
            javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure]  
            at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)  
            at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:220)  
            at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)  
            at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)  
            at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)  
            at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)  
            ... 13 more  
    Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:  
            javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
            at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:300)          
    at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196)  
            at java.rmi/sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)  
            at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)  
            at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)  
            ... 18 more  
    Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure  
            at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)  
            at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)  
            at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:347)  
            at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)  
            at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:186)
            at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)        
    at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1507)  
            at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1417)     
    at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)... 22 more
     
     
  • The CORS filtering list is populated and updated during the cell configuration. It may be missing HTTP and HTTPS entries with IPs and DNS names for all cells in the server group.

Environment

VMware Cloud Director 10.x

Cause

  • The issue occurs when the SSL ciphers configuration is not the same on all cells. 
  • The issue occurs even when "webapp.allowed.origins" are not properly configured and CORS do not include all the endpoints. 

Resolution

To resolve the issue please follow the below steps:

  • To check the SSL cipher enabled on each cell run the cell-management-tool as below:

    /opt/vmware/vcloud-director/bin/cell-management-tool ciphers -l

  • To confirm the SSL ciphers configuration on each cell,it is possible to check the /opt/vmware/vcloud-director/etc/global.properties file.

    For example: ssl.ciphers.disallowed = TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA

  • Factory default ciphers are:

    * TLS_AES_256_GCM_SHA384 
    * TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 
    * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

  • The ciphers can be reset to the factory default settings on each cell by running the cell-management-tool as below:

    /opt/vmware/vcloud-director/bin/cell-management-tool ciphers -r

  • If the CORS do not include all the endpoints please refer to the following documents: Configure CORS for VMware Cloud Director OR the kb: Modifying the "webapp.allowed.origins" configuration on version 10.3 and later

Additional Information

Entering into maintenance mode fails to carry out any upgrade or downtime tasks.

Powered by Wolken Software