VMware Cloud Director is not accessible via the load balanced address after an upgrade

Products

• VMware Cloud Director

Issue/Introduction

• After the upgrade, VMware Cloud Director is not accessible via the load-balanced address which is load-balanced through NSX, it is failing with the error "502 HTTP error".
• Accessing the VMware Cloud Director Provider UI can result in error messages like "An error occurred during the initialization. Accessing the application through an unsupported public url or poor connectivity might cause this error."
• In /opt/vmware/vcloud-director/logs/cell-runtime.log on the Cells errors are seen of the form:

DEBUG    | pool-jetty-37             | HttpEngineStartupAction        | Handshake failed | 
javax.net.ssl.SSLHandshakeException: no cipher suites in common
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
    at java.base/java.lang.Thread.run(Thread.java:829)

Environment

• VMware Cloud Director 10.6.x
• VMware Cloud Director 10.5.x

Cause

The issue is related to SSL cipher suite compatibility between the upgraded nodes and the NSX Load Balancer. The SSL handshake failure suggests that there are no common cipher suites between the Load Balancer and the upgraded nodes, causing the connection to fail.

Resolution

To resolve this issue ensure the load balancer supports the cipher suites enabled on the Cloud Director Cells.

The currently allowed ciphers can be listed on the Cells using the "cell-management-tool ciphers" command:

/opt/vmware/vcloud-director/bin/cell-management-tool ciphers -l

As per the documentation, Managing the List of Allowed SSL Ciphers in VMware Cloud Director, several ciphers are not on this list, either because they are not strong enough to secure the connection, or because they are known to contribute to SSL connection failures.

As a workaround, additional ciphers can be enabled on the Cells using the "cell-management-tool ciphers" command, see the documentation for more details.

WARNING: Running the "ciphers -d" option without any values activates all ciphers, some ciphers may be considered insecure. Running the "ciphers -d <disallowed_cipher_list>" option to allow additional ciphers may also result in activating some ciphers which may be considered insecure. Broadcom recommends running Cloud Director Cells using the default ciphers, use the "ciphers --reset" option from the documentation to return the Cell to the default settings.