Windows Guest OS console displays a black screen or RDP fails after upgrading VMware Tools

Products

• VMware vSphere ESXi

Issue/Introduction

After upgrading VMware Tools on a Windows Guest OS, the following symptoms occur.

• The virtual machine console in the vSphere Client displays a black screen and the login prompt does not appear.
• RDP (Remote Desktop Protocol) connections fail to establish or disconnect immediately, resulting in no display.
• The Guest OS itself is running. Ping responses and other network services are functioning normally.

The following errors are recorded in the virtual machine's vmware.log:

MMMM-DD-HHThh:mm:ss.sssZ In(05) vmx - Tools_SetGuestResolution: Sending rpcMsg = Resolution_Set 3046 1056
MMMM-DD-HHThh:mm:ss.sssZ In(05) vcpu-0 - Guest: vm3d: Memory required for all screens exceeds maximum (required=12866304 max=8388608)
MMMM-DD-HHThh:mm:ss.sssZ In(05) vcpu-0 - Guest: DXUM_devapi: ####: Topology validation failed, screen memory limits exceeded.
MMMM-DD-HHThh:mm:ss.sssZ In(05) vcpu-0 - TOOLS call to Resolution_Set failed.

Note: The values shown in the log excerpt, such as resolution (Resolution_Set) and memory values (required/max), are examples. Actual values may vary depending on the environment and configuration.

Environment

VMware vSphere ESXi

Cause

This issue is caused by insufficient video memory allocated to the virtual machine.
With the upgrade of VMware Tools, the amount of video memory required to maintain the current resolution and display settings has increased, exceeding the virtual machine's maximum allocation limit.
The log entry required=... max=... indicates that the memory amount requested by the driver exceeds the configured limit of the virtual machine.

Resolution

To resolve this issue, modify the video memory allocation settings for the virtual machine.

1. Power off the virtual machine.
2. Right-click the virtual machine in the vSphere Client and select Edit Settings.
3. Expand Video card.
4. Apply one of the following settings:

• Method 1: Use Auto-detect
  Select Auto-detect settings. This automatically calculates and allocates the necessary video memory based on the Guest OS requirements.
• Method 2: Specify a custom value
  Select Specify custom settings. Enter a value of 16 MB or higher in Total video memory (Recommended value: 16 MB or 32 MB).

5. Click OK to save the settings.
6. Power on the virtual machine and verify that the screen is displayed correctly.