VMware Tools service crashes upon boot of a VM or attempts to update/install/remove with "VMware Tools unrecoverable error: Exception 0xc0000005 (access violation)"

Products

• VMware vCenter Server

Issue/Introduction

When attempting to install or update the VMware Tools running in a Windows Server 2012R2 VM, the following errors occur.

Environment

vSphere ESXi 7.x
vSphere ESXi 8.x
vSphere vCenter Server 7.x
vSphere vCenter Server 8.x
VMware Tools 11.x
VMware Tools 12.x

Cause

An application sets a system-wide environment variable for the Guest OS to a character set (CHARSET) unknown to VMware Tools. This causes the VMware Tools GLib to fail strings/error conversions and causes VMware Tools to crash due to NULL pointer access or uninitialized variables.

Resolution

VMware by Broadcom is aware of the issue and working on fix to be released in a future VMware Tools update.

Workaround:

• Download the attached script and upload it to storage that is accessible by the Virtual Machine

• In the Windows Server 2012R2 VMs

  • Copy the vmtoolsd.cmd file into the VMware Tools installation folder; "C:\Program Files\VMware\VMware Tools\"

    • NOTE: The account used to perform this, as well as the Registry entry edits below, must be a part of the Administrator’s group for the Guest OS.  Further, ensure that the vmtoolsd.cmd file is not saved in any encoding other that ANSI, or you may experience failures with this recommended process.

    • The vmtoolsd.cmd script contains the following lines:

      @echo OFF
      setlocal
      rem prevent wrong environment charset from being used
      set CHARSET=
      if not "%1" == "" goto vmusr
      call "%~dpn0.exe"
      goto exit
      
      :vmusr
      start "" /B "%~dpn0.exe" %*
      
      :exit

       

  •  

• Then change 2 registry entries:

  • HKLM\SYSTEM\CurrentControlSet\Services\VMTools\ImagePath

    • From
      "C:\Program Files\VMware\Vmware Tools\vmtoolsd.exe" 
      
      To
      "C:\Program Files\VMware\Vmware Tools\vmtoolsd.cmd"

  • HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run\Vmware User Process

    • From  
      "C:\Program Files\VMware\Vmware Tools\vmtoolsd.exe" -n vmusr"
      
      To
      "C:\Program Files\VMware\Vmware Tools\vmtoolsd.cmd" -n vmusr"

• Restarting the Windows system is recommended in case of registry caching.  You should now see that the VMware Tools service can run without crashing after the reboot.
• Verify, using taskmgr or tasklist.exe, that 2 vmtoolsd.exe processes are running.  One as SYSTEM and one as the logged-on user account.

 

NOTE: This is only a temporary workaround until a version of VMware Tools can be installed that handles the environment variable correctly.
Any upgrade to Tools versions without a fix would require reapplying the above changes.

Additional Information

 

Process Monitor logs are helpful in diagnosing the cause of the crash, download the latest version from Microsoft SysInternals

Run the VMware Tools installer and let us know if it crashes.  If it crashes:

• Stop existing the vmtoolsd.exe processes
  • Open command prompt in "Run as Administrator", run:
    • sc stop VMTools
• Run the Process Monitor as administrator
  • Set the Filter for Process Name as vmtoolsd.exe, add and apply the filter
• Click Menu>Filter>Drop Filtered Events
• In the Administrator command prompt, run:
  • sc start VMTools
  • sc stop VMTools <=== Wait 3 seconds after the previous command
• Save the Process Monitor events to a file, with the default settings, you can change the file path to which ever folder you choose.
• Upload this file to your support case in the Broadcom Portal
  • Uploading files to cases on the Broadcom Support Portal

Enabling debug logging for VMware Tools within a guest operating system

Another utility from Microsoft SysInternals is Process Explorer, which can also help investigate the VMware Tools processes. Run the Procexp[64].exe as Administrator. Once running, you can find the VMware Tools processes by selecting the ribbon filter box “<Filter by name>” and typing the name of the VMware Process to view. This can be:

• Vmtoolsd.exe
• Vgauthservice.exe

Once filtered, if nothing is shown, the processes are not running. 

If found, you select the process shown and right-click properties. This will show a dialog with many information tabs. Here, the Environment tab is useful, and it should show CHARSET is used or set for the process. If missing, then the variable has been cleared for the process.

Furthermore, in the main application window, select the “View” menu option and “Show Lower Pane”. This will show open handles, loaded DLLs, or threads for the selected process.