Error "ERR_SSL_PROTOCOL_ERROR" when accessing Cloud Director portal

Products

• VMware Cloud Director

Issue/Introduction

• VCD(VMware Cloud Director) is successfully upgraded from 10.4.x to 10.5.1 or above(including 10.6.1.2) and cell startup is succesfully completed on all the cells.
• Accessing the VCD provider portal fails with the following error:
  
  <https://vcd.example.com> sent an invalid response.
 ERR_SSL_PROTOCOL_ERROR
  
  
  
  
• Attempting to access the VCD portal using cell's FQDN/IP directly (bypassing the load balancer, per KB/374695) continues to fail with the same error.
• In /opt/vmware/vcloud-director/logs/cell-runtime.log of VCD cell, you see entries similar to following:
  
  | DEBUG    | pool-jetty-47             | HttpEngineStartupAction        | Handshake failed |
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:360)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:303)

Environment

Cloud Director 10.5.1.x
Cloud Director 10.6.x

Cause

This issue occurs when the migrated web server certificate is corrupted during the VCD upgrade process.

Resolution

To resolve this issue, 

1. Temporarily point the web server certificate of primary VCD cell to it's JMX certificate from the database.
   
   Note: Step.1 requires Database modifications in Cloud Director. Please open a Support Request with Broadcom Technical Support and note this Article ID (430478) in the problem description. For more information, see Creating and managing Broadcom support cases.
   
   
2. Restart vcd service on primary cell: service vmware-vcd stop && service vmware-vcd start . Once the cell startup is completed, VCD portal will be accessible.
3. If using a load balancer, bypass the same following KB/374695 and access the portal using it's primary VCD cell's FQDN/IP address.
4. From VCD provider portal, edit the 'web server' certificate of other cells and select JMX certificate from certificate library - Change the Certificates of a Cell .
5. Ensure there are no consumers listed under the 'Migrated/Web server certificate' and delete the certificate from certificate library 
6. Import the required Web certificates(CA/self-signed) to certificate library following Import Certificates and assign it to the cells - Change the Certificates of a Cell .

Once the certificates are re-assigned, discard actions performed in step.3 to ensure traffic is redirected to load balancer.