"Setup failed to generate the SSL keys necessary to run VMware Server" when installing VMware Workstation

Products

• VMware Desktop Hypervisor

Issue/Introduction

This article discusses identifying and resolving various issues that affect VMware Workstation from install or upgrade.

Symptoms:

• Workstation fails to install or upgrade on a computer running Windows
• The installation fails with an error message:
  
  Setup failed to generate the SSL keys necessary to run VMware Server.

Environment

VMware Workstation Player 14.x (Windows)
VMware Workstation Pro 16.x (Windows)
VMware Workstation Player 16.x (Windows)
VMware Workstation Player 15.x (Windows)
VMware Workstation Pro 15.x (Windows)
VMware Workstation Pro 14.x (for Windows)

Resolution

During installation, the setup program invokes a copy of openssl.exe on the command line. If a copy of openssl.exe is already present on the computer and in the PATH variable, the wrong version of openssl.exe may be executed and cause the Workstation installation to fail.

To workaround this issue:

1. Locate the duplicate copy of openssl.exe that is located in the folder listed within the %PATH% environment variable.

        Note: You can also search the computer's hard drive to locate the file.

2. Rename the duplicate file to something else.
3. Install VMware Workstation. 
4. When the installation is complete, change the name of the duplicate file back to openssl.exe

If the above steps do not work, try the below-listed troubleshooting steps.
Gather vminst.log as well as vmmsi.log from the %TEMP% directory for the latest install attempt & look for similar lines as listed below.

vminst.log
**********
2021-07-15T08:35:51.858+02:00| inst-build-17966106| I2: Util_FileExists: Found "C:\Program Files (x86)\VMware\VMware Workstation\openssl.exe"
2021-07-15T08:35:51.858+02:00| inst-build-17966106| I1: Util_Launch: Attempting to launch ""C:\Program Files (x86)\VMware\VMware Workstation\openssl.exe" req -x509 -sha256 -days 365 -newkey rsa:2048 -keyout "C:\ProgramData\VMware\SSL\rui.key" -out "C:\ProgramData\VMware\SSL\rui.crt" -config "C:\ProgramData\VMware\SSL\hostd.ssl.config""
2021-07-15T08:35:51.968+02:00| inst-build-17966106| I1: Util_Launch: Child exited with code 3221225775
2021-07-15T08:35:51.968+02:00| inst-build-17966106| I1: Util_Launch: Process returned -1073741521
2021-07-15T08:35:51.968+02:00| inst-build-17966106| I0: ERROR: Failed to generate SSL keys [3221225775]
2021-07-15T08:35:51.968+02:00| inst-build-17966106| E1: Util_GetKeyValueString: Cannot query key value HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\Wnd| winerror code 2 (The system cannot find the file specified.)

vmmsi.log
*********
MSI (s) (E4:94) [08:35:51:828]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSIBE5F.tmp, Entrypoint: VMGenerateSSLKeys
CustomAction VM_GenerateSSLKeys.9F27F70C_3D9B_4082_8801_88387445C782 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)Action ended 08:35:56: InstallFinalize. Return value 3.

Please follow the steps listed below to isolate the cause. Can be a DLL File as in this case.

1. Run the VMware Workstation installer to the point where it fails with the error message but this time, DO NOT click OK.
2. Keeping the installer at that error dialog box open, open an elevated command prompt, and run the below command which can be copied from the vminst.log as seen above.

• "C:\Program Files (x86)\VMware\VMware Workstation\openssl.exe" req -x509 -sha256 -days 365 -newkey rsa:2048 -keyout
• "C:\ProgramData\VMware\SSL\rui.key" -out
• "C:\ProgramData\VMware\SSL\rui.crt" -config
• "C:\ProgramData\VMware\SSL\hostd.ssl.config"

3. If installing Workstation to a different Path, you may edit the path in the first segment of the command.  

        The output of the command which is run above will let us know the cause