VM Console is not displayed due to "Failed to connect to server (code: 1006)"

Products

• VMware Integrated OpenStack

Issue/Introduction

• Trying to open a VM console via Openstack Horizon, this console is NOT displayed with an error "Failed to connect to server (code: 1006)".
• In nova-mksproxy log, you can see entries similar to the followings:

2020-04-16T08:18:57Z 2020-04-16 08:18:57.986 109 DEBUG nova.objects.console_auth_token [req-########-####-####-####-############ - - - - -] Validated token - console connection is ConsoleAuthToken(access_url_base='https://10.13.0.96:6090/vnc_auto.html',console_type='novnc',created_at=2020-04-16T08:18:57Z,host='ESXi.acme.local',id=147,instance_uuid=########-####-####-####-############,internal_access_path='{"ticket": "########-####-####-####-############", "thumbprint": "####", "cfgFile": "/vmfs/volumes/########-####-####-####.vmx"}',port=902,token='***',updated_at=None) validate /usr/lib/python2.7/site-packages/nova/objects/console_auth_token.py:158

2020-04-16T08:18:58Z 2020-04-16 08:18:58.128 109 INFO nova.compute.rpcapi [req-########-####-####-####-############ - - - - -] Automatically selected compute RPC version 5.1 from minimum service version 37
2020-04-16T08:20:17Z 2020-04-16 08:20:17.767 109 INFO nova.console.websocketproxy [req-########-####-####-####-############ - - - - -] handler exception: [Errno 104] Connection reset by peer

• In vmauthd.log, you can see entries similar to followings:

2020-04-16T08:18:58Z vmauthd[4203806]: Connect from remote socket (10.13.0.98:41926).
2020-04-16T08:18:58Z vmauthd[4203806]: Connect from 10.13.0.98
2020-04-16T08:20:17Z vmauthd[4203806]: SSL: syscall error 110: Connection timed out
2020-04-16T08:20:17Z vmauthd[4203806]: recv() FAIL: 110.
2020-04-16T08:20:17Z vmauthd[4203806]: VMAuthdSocketRead: read failed.  Closing socket for reading.
2020-04-16T08:20:17Z vmauthd[4203806]: Read failed.

Environment

6.x, 7.x

Cause

This issue is caused by the communication issue between nova-mks and ESXi vmkernel port because MKS communication is used for displaying VM console.

For example, if MTU of vmkernel port is set to "9,000", nova-mks and vmkernel port tries to use "MSS=8960" for TCP communication. However, MTU configurations of Openstack controller nodes are "1,500" by default. As a result, packet with larger size than 1500 are dropped at vNIC of an Openstack controller node running nova-mks. In the above situation, TCP 3-way handshake can be established, but following server hello packet for SSL handshake is dropped at vNIC.

Resolution

This issue can be fixed by troubleshooting network issues.
For example, this issue can be fixed by matching MTU among vmkernel port and Openstack controller nodes.

Additional Information

Failed to download a Glance image due to "500 Internal Server Error"