Error: "Couldn't establish a connection to the VM web console" on Windows Guest OS
search cancel

Error: "Couldn't establish a connection to the VM web console" on Windows Guest OS

book

Article ID: 429790

calendar_today

Updated On:

Products

VMware vSphere ESX 8.x VMware vSphere ESXi 8.0

Issue/Introduction

After upgrading VMware Tools on a Windows Guest OS, the following symptoms occur.

  • The virtual machine (VM) console in the vSphere Client displays a black screen with the error: 
    "Couldn't establish a connection to the VM web console."



  • RDP (Remote Desktop Protocol) connections fail to establish or disconnect immediately, resulting in no display.
  • The Guest OS itself is running. Ping responses and other network services are functioning normally.

The following task failure is triggered in the vCenter Server UI:

The task VirtualMachine.setScreenResolution failed with A general system error occurred: Undeclare

The following errors are recorded in the virtual machine's /vmfs/volumes/<Datastore>/vmware.log:

YYYY-MM-DDTHH:MM:SSZ Wa(03) mks - SOCKET 2988 (106) unable to determine remote IP address
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MKSRemoteMgr: AddRemoteConsole (numConsoles=0)
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2987 (106) AsyncTCPSocketSetOption: sendLowLatencyMode set to [1].
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2988 (106) Creating VNC remote connection.
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MKSControlMgr: New VNC connection 21
YYYY-MM-DDTHH:MM:SSZIn(05) svga - MVNCEncode: MVNCEncode_ServerSetTopology - original root: (0, 0) size: (1580, 606)
YYYY-MM-DDTHH:MM:SSZ In(05) svga - MVNCEncode: Number of screens changed from 0 to 1
YYYY-MM-DDTHH:MM:SSZ In(05) svga - MVNCEncode: screen: 0 BoundingBox: (1580x606) Screen (1580x606) @ (0,0) bytesPerLine: 6320
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MVNCRegionEncoder_Create: region encoder static. Screen 0/1 @ Resolution: 1580 x 606
YYYY-MM-DDTHH:MM:SSZ In(05) mks - JPEG quality levels (min, mid, max). Input: (25, 35, 90) Clamped: (25, 35, 90)
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MVNCRegionEncoder_Create: region encoder adaptive. Screen 0/1 @ Resolution: 1580 x 606
YYYY-MM-DDTHH:MM:SSZ In(05) mks - JPEG quality levels (min, mid, max). Input: (25, 35, 90) Clamped: (25, 35, 90)
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MVNCRegionEncoder_Create: region encoder adaptive. Screen 0/1 @ Resolution: 1580 x 606
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - Tools_SetGuestResolution: Sending rpcMsg = Resolution_Set 2341 1110
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-1 - Guest: DXUM_devapi: 2156: Application requested VM3DDevApi_Version3 (0x2)
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-1 - Guest: vm3d: Memory required for bounding box of all screens exceeds maximum (required=10394040 max=4194304)
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-1 - Guest: DXUM_devapi: 2156: Topology validation failed, screen memory limits exceeded.
YYYY-MM-DDTHH:MM:SSZ In(05) vcpu-1 - TOOLS call to Resolution_Set failed.
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2987 (106) recv detected remote shutdown
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2988 (106) VNC Remote Disconnect: socket closed.
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MKSControlMgr: Remove VNC connection 21
YYYY-MM-DDTHH:MM:SSZ In(05) vmx - Tools_LastRemoteConsoleDisconnected: Guest Tools are not able to autolock desktop.
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2988 (106) Sending websocket close frame, status code = 1000
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2987 (106) send error 32: Broken pipe
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2988 (106) VNC Remote Disconnect: socket closed.
YYYY-MM-DDTHH:MM:SSZ In(05) mks - SOCKET 2988 (106) MVNCBackend: Spurious socket error?
YYYY-MM-DDTHH:MM:SSZ In(05) mks - MKSRemoteMgr: CleanupConnection (numConsoles=0)

Note: The values shown in the log excerpt, such as resolution (Resolution_Set) and memory values (required/max), are examples. Actual values may vary depending on the environment and configuration.

Environment

VMware vSphere 8.x

VMware ESXi 8.x

Cause

This issue is caused by insufficient video memory allocated to the virtual machine.

Following a VMware Tools upgrade, the SVGA display driver requires an increased threshold of video memory overhead to support active resolution and topology settings. The validation failure occurs when the driver's requested allocation size (required=...) exceeds the virtual machine's maximum hardware ceiling (max=...).

Resolution

To resolve this issue, increase the video memory allocation threshold within the virtual machine configuration:

  1. Gracefully shut down the Guest Operating System and power off the virtual machine.

  2. Right-click the affected virtual machine in the inventory pane and select Edit Settings.

  3. Expand Video card.
  4. Apply one of the following settings:
  • Method 1: Use Auto-detect
    Select Auto-detect settings. This automatically calculates and allocates the necessary video memory based on the Guest OS requirements.
  • Method 2: Specify a custom value
    Select Specify custom settings. Enter a value of 16 MB or higher in Total video memory (Recommended value: 16 MB or 32 MB).
  1. Click OK to save the settings.
  2. Power on the virtual machine and verify that the screen is displayed correctly.
Powered by Wolken Software