This issue is caused by the communication issue between nova-mks and ESXi vmkernel port because MKS communication is used for displaying VM console.
For example, if MTU of vmkernel port is set to "9,000", nova-mks and vmkernel port tries to use "MSS=8960" for TCP communition. However, MTU configurations of Openstack controller nodes are "1,500" by default. As a result, packet with larger size than 1500 are dropped at vNIC of an Openstack controller node running nova-mks. In the above situation, TCP 3-way handshake can be established, but following server hello packet for SSL handshake is dropped at vNIC.
Resolution
This issue can be fixed by troubleshooting network issues. For example, this issue can be fixed by matching MTU among vmkernel port and Openstack contoller nodes.