VMware Tools installation fails due to a VMCI driver error on a Windows Server guest OS

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• When attempting to install or upgrade VMware Tools on a Windows Server guest operating system, the installation fails with the following error:

"Setup failed to install the VMCI Virtual Machine Communication driver automatically."

• Attempting a manual installation of the VMCI driver via Device Manager fails with the following error:

Windows encountered a problem installing the driver software for your device

Windows found driver software for your device but encountered an error while attempting to install it.

VMware VMCI Bus Device

The service has not been started.

• Analysis of the C:\Windows\system32\setupapi.dev.log file reveals the following failure error during the driver staging process: 

!!! sig: Failed to install catalog 'vmci.cat' as 'oem3.cat'. Error = 0x00000426

!!! sto: Failed to import driver package into Driver Store. Error = 0x00000426

• Run the following command in a Command Prompt with Administrative privileges to check if the system recognizes the service name:

sc query cryptsvc

If the service has been deleted from the registry, the following message will be displayed: [SC] EnumQueryServicesStatus:OpenService FAILED 1060: The specified service does not exist.

Cause

The issue is caused by the absence or corruption of the Cryptographic Services (CryptSvc) in the Windows registry.

Windows cannot verify the digital signature (catalog file) of the driver due to the missing Cryptographic Service; it rolls back the installation and applies a "NULL driver" placeholder.

Common reasons for the disappearance of this service include:

• Registry Corruption: Damage to the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CryptSvc key.
• Security Software Interference: Antivirus tools may have quarantined cryptsvc.dll or blocked registry access.
• System File Integrity Issues: Potential malware or manual deletion of core service components.

Resolution

• The Cryptographic Services component is a critical part of the Windows operating system and is required for file integrity validation and driver signature verification.
• If this service is not functioning correctly, engage the operating system vendor to perform further investigation and remediation.

Note: VMware Tools installation cannot be completed successfully until the underlying Windows Cryptographic Services are restored to a functional state.