VMware Tools showing 'Upgrade Available' in the VM Summary Tab when running on certain hosts and 'Current' when running on different hosts on the same cluster in vCenter

Products

• VMware vSphere ESXi

Issue/Introduction

Clustered ESXi hosts that are on the exact same ESXi build and contain the same VMware Tools Async release version, VM(s) showing in VMware Tools section in the Summary Tab 'Upgrade available' on a host and when the VM is vMotioned/Migrated to another host in the same cluster it shows the VMware Tools version to be 'Current'.

 

Environment

ESXi 8.0.x

ESXi 7.0.x

vCenter 7.0.x

vCenter 8.0.x

Cause

The VMware Tools manifest file that the VM(s) check against on the ESXi host, when the VM Summary Tab is NOT showing 'Upgrade available' in the VMware Tools section. It is likely retaining stale data in the manifest file residing on the ESXi host compared to the other hosts that show  in the VMware Tools section 'Upgrade available' when all the hosts in the cluster have a newer VMware Tools version installed.

In the vmware.log of the VM we are seeing error toolsInstallManager.lastInstallError = "21009"

Along with similar log entries below in the vmware.log

 

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Updated cached value for imageName to 'windows.iso'.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Selected Tools ISO 'windows.iso' for 'windows2019srvNext-x4' guest.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Item 'qpair.context' [0, -1] not found.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Item 'timeoutRemaining' [0, -1] not found.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Item 'timeoutRemaining' [1, -1] not found.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring GuestMsg

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring GuestRpc

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Item 'AsyncVmciSocket.numSendBuf' [-1, -1] not found.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring Tools

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - Tools: Changing running status: 0 => 1.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - Tools: [RunningStatus] Last heartbeat value 0 (last received 2579984s ago)

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring Tools Install

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - TOOLS INSTALL setting state to 0 on restore.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring GuestAppMonitor

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Requested 10 bytes, found 5 bytes.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DUMPER: Requested 20 bytes, found 5 bytes.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring MKSVMX

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring pvnvram

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - GetHostManifests: Extracting /tmp/vmware-root/manifest.txt.xxxxxxxx.iso.shipped manifest file.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx -   restoring ToolsDeployPkg

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DEPLOYPKG: ToolsDeployPkgCptRestore: state=0 err=0 (null msg)

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - DEPLOYPKG: ToolsDeployPkgRestoreSuccessTasks: state=0 err=0, msg=null

 

In the vmware.log it presents the error below: 

ToolsISO: Error failed to verify signature using default cert: (null)

 

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vcpu-0 - Preparing for SPEC_CTRL Guest MSR write (0x48) passthrough.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Error failed to verify signature using default cert: (null)

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Signature verification failed for '/vmfs/volumes/xxxxxxxxxx-xxxxxxxxxxx-xxxx-xxxxxxxxxxxxxx/locker/packages/vmtoolsRepo/vmtools/windows.iso'.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Failed to verify signature for Tools ISO image.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Error getting host tools manifest files.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Refreshing imageName for 'windows2019srvNext-x4' (refreshCount=1, lastCount=1).

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Updated cached value for imageName to 'windows.iso'.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Selected Tools ISO 'windows.iso' for 'windows2019srvNext-x4' guest.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - GetHostManifests: Extracting /tmp/vmware-root/manifest.txt.xxxxxxxx.iso.shipped manifest file.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vcpu-0 - CPT: vmstart

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vcpu-1 - CPT: vmstart

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Error failed to verify signature using default cert: (null)

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsISO: Signature verification failed for '/vmfs/volumes/xxxxxxxxx-xxxxxxxxxx-xxxx-xxxxxxxxxxxxxxx/locker/packages/vmtoolsRepo/vmtools/windows.iso'.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Failed to verify signature for Tools ISO image.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Error getting host tools manifest files.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Using ToolsMinVersion = 8384

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - Using ToolsMaxVersion = 12448

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) worker-xxxxxxxx - ToolsVersionGetStatusWorkerThread: Tools status 3 derived from environment

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - TOOLS refreshing value for isoImageExists (refreshCount=1, lastCount=0).

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - ToolsISO: Refreshing imageName for 'windows2019srvNext-x4' (refreshCount=1, lastCount=1).

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - ToolsISO: Updated cached value for imageName to 'windows.iso'.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - ToolsISO: Selected Tools ISO 'windows.iso' for 'windows2019srvNext-x4' guest.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - TOOLS updated cached value for isoImageExists to 1.

xxxx-xx-xxxxx:xx:xx.xxxZ In(xx) vmx - VMXVmdb_SetToolsVersionStatus: status value set to 'ok', 'current', install possible

 

 

Resolution

To address this issue when using Cluster Single Image Baseline, Navigate to Cluster > Updates > Image > In the Image section press the EDIT button

Under Components select Show Details and ADD COMPONENTS

Under Component Name tab select the Filter icon and enter tool and you will see VMware Tools Async Release show up.

Add the VMware Tools Async Release that is needed from the dropdown and press Select > Next press Save and then Validate

 

Check the current location of /productlocker with the command below

ls -l /productLocker

Ex Output:

lrwxrwxrwx    1 root     root            xx Month xx xx:xx /productLocker -> /locker/packages/vmtoolsRepo

 

Next we need to remove the tools-light vib from the ESXi host that is not showing the VMware Tools 'Upgrade is available'.

1. Confirm if SSH is enabled, if not start the SSH service.

2. Next SSH into the ESXi host in question and Run the command below

esxcli software vib remove -n tools-light 

3. Then remove the floppies and vmtools folders under the directory below with the command

rm -rf /vmfs/volumes/<datastore-name-or-volume-id>/locker/packages/vmtoolsRepo/*

4. Confirm floppies and vmtools folders are no longer present by running the command below.

ls /vmfs/volumes/<datastore-name-or-volume-id>/locker/packages/vmtoolsRepo/

5. Run Remediate in the vSphere Client > Cluster > Updates > Select the host that you removed the tools-light vib and the folders floppies and vmtools

6. Once the Remediate completes validate that the folders were recreated on the host  floppies and vmtools by running the command below

ls /vmfs/volumes/<datastore-name-or-volume-id>/locker/packages/vmtoolsRepo/

7. Move the VM that was showing VMware Tools as 'Current' and NOT 'Upgrade available' to the host that tools-light was installed on again, in order to confirm that it is now showing 'Upgrade available' for VMware Tools for the VM in the vSphere Client VM Summary Tab.