VMware Tools upgrade fails and previous version is uninstalled after virtual machine reboot

Products

• VMware vCenter Server

Issue/Introduction

During a virtual machine reboot, the VMware Tools upgrade fails, resulting in the previous version of VMware Tools being uninstalled.

The vminst.log file indicates that the installer attempted to terminate the running vmtoolsd.exe process before replacing files and drivers, but failed after multiple consecutive retries:

YYYY-MM-DDTHH:MM:SS| #####-#####-########| E1: Util_IsProcessRunning: Found running process: 3372:C:\Program Files\VMware\VMware Tools\vmtoolsd.exe
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I1: Util_FreeLibrary: Freeing library: 1471873024
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I1: Util_FreeLibrary: Freeing library: 1459224576
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I1: Util_IsProcessRunning: Found running process: vmtoolsd.exe
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I2: Util_FileExists: Found "C:\Program Files\VMware\VMware Tools\vmtoolsd.exe"
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I1: Util_Launch: Attempting to launch ""C:\Program Files\VMware\VMware Tools\vmtoolsd.exe" -n vmusr --kill"
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I1: Util_Launch: Child exited with code 1
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I0: VMTools user process 'vmtoolsd.exe' not terminiated, retry #1
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I0: VMTools user process 'vmtoolsd.exe' not terminiated, retry #2
YYYY-MM-DDTHH:MM:SS| #####-#####-########| I0: VMTools user process 'vmtoolsd.exe' not terminiated, retry #3

Additionally, the vmupgrader_msi.log file records file lock warnings. Because the existing processes could not be terminated, MSI reported multiple File In Use warnings during file verification. It identified several processes locking VMware Tools files, including third-party software such as Trend Micro Deep Security Agent (dsa.exe):

Info 1603. The file C:\Program Files\VMware\VMware Tools\vm3dservice.exe is being held in use by the following process: Name: vm3dservice, Id: 3360, Window Title: '(not determined yet)'. Close that application and retry.
Info 1603. The file C:\Program Files\VMware\VMware Tools\vm3dservice.exe is being held in use by the following process: Name: vm3dservice, Id: 3896, Window Title: '(not determined yet)'. Close that application and retry.
Info 1603. The file C:\Program Files\VMware\VMware Tools\VMware VGAuth\####-#.#-#.dll is being held in use by the following process: Name: vmtoolsd, Id: 3372, Window Title: '(not determined yet)'. Close that application and retry.
Info 1603. The file C:\Program Files\VMware\VMware Tools\VMware VGAuth\####-#.#-#.dll is being held in use by the following process: Name: dllhost, Id: 4752, Window Title: '(not determined 
Info 1603. The file C:\Program Files\VMware\VMware Tools\VMware VGAuth\######-#-###.dll is being held in use by the following process: Name: dsa, Id: 4876, Window Title: '(not determined yet)'. Close that application and retry.
Info 1603. The file C:\Program Files\VMware\VMware Tools\VMware VGAuth\libxmlsec.dll is being held in use by the following process: Name: VGAuthService, Id: 3332, Window Title: '(not determined yet)'. Close that application and retry.
MSI (s) (CC:3C) [15:31:06:469]: Note: 1: 2727 2:  
MSI (c) (70:74) [15:31:06:463]: File In Use: -vmtoolsd- Window could not be found. Process ID: 3372
MSI (c) (70:74) [15:31:06:463]: File In Use: -VGAuthService- Window could not be found. Process ID: 3332
MSI (c) (70:74) [15:31:06:463]: File In Use: -dllhost- Window could not be found. Process ID: 4752
MSI (c) (70:74) [15:31:06:463]: File In Use: -vm3dservice- Window could not be found. Process ID: 3360
MSI (c) (70:74) [15:31:06:463]: File In Use: -vm3dservice- Window could not be found. Process ID: 3896
MSI (c) (70:74) [15:31:06:463]: File In Use: -dsa- Window could not be found. Process ID: 4876
MSI (c) (70:74) [15:31:06:463]: No window with title could be found for FilesInUse

Environment

VMware ESXi 8.0

Cause

Third-party applications (such as Trend Micro Deep Security Agent dsa.exe) or other system processes held file locks on VMware Tools components and plugin DLLs, preventing the installer from replacing them.

Resolution

Workaround / Manual Installation:

Mount the VMware Tools ISO image to the virtual machine.

Log into the VM guest OS and execute setup64.exe Run as Administrator to complete the installation manually.

Preventative Action:

Contact the third-party software vendor (e.g., Trend Micro) for further investigation and consider configuring process or folder exclusions for VMware Tools during upgrades.