Error: "Setup wizard ended prematurely" during VMware Tools installation on Windows virtual machines

Products

• VMware vCenter Server

Issue/Introduction

When attempting to install or upgrade VMware Tools, the process begins but fails shortly after the "Rolling Back Action" message appears. The installer indicates "VMware Tools setup wizard ended prematurely" without a specific error code in the UI. This is commonly observed on Windows Server 2016 virtual machines (VMs) following Windows Updates.

Symptoms:

• VMware Tools status shows "Not running, not installed".
  
  
• In %temp%/vminst.log, the following error appears:
  
  CustomAction VM_SecureVGAuthDirs returned actual error code 1603
Action ended: InstallFinalize. Return value 3.
  
  
• In %temp%/vmmsi.log, the following error appears:
  
  ERROR: C:\ProgramData\VMware\VMware VGAuth\logfile.txt.0's file SDDL string lacks ACLs (80070057)

Environment

• VMware Tools 13.x
• VMware ESXi Server 7.x
• VMware ESXi Server 8.x
• VMware vCenter Server 7.x
• VMware vCenter Server 8.x

Cause

The VMware Tools installer requires access to the VGAuth directory to secure the Alias Manager. If the logfile.txt.0 file is locked by the system or lacks proper Access Control Lists (ACLs), the MSI installer encounters an access violation (Error 1603) and rolls back the installation.

Resolution

1. Log into the affected Windows VM with Administrator privileges.
2. Stop the VMware Alias Manager and Ticket Service (VGAuthService) from the Services console (services.msc).
3. Navigate to C:\ProgramData\VMware\VMware VGAuth\.
4. Locate the file logfile.txt.0.
5. Rename the file to logfile.txt.0.old or move it to the Desktop.
6. Rerun the VMware Tools installer.
7. If the issue persists, ensure the C:\ProgramData\VMware\VMware VGAuth\ folder has appropriate system permissions.