Unable to remediate ESXi 8.0U2 host with using desired image with VMware VMTools as removed component

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

This article provides steps to remediate a host running ESXi 8.0U2 using a desired image to remove a package 

• Scan is failing on hosts with version 80U2 and vCenter on version 80U3, when desired image is configured with removed components (for example: VMware-VM-Tools).
  
  
• Attempting to remediate the host in Life Cycle manager fails with the following error
  
• In the vCenter server /var/log/vmware/vmware-updatemgr/vum-server/vum-server.log :
  
  YYYY-MM-DD HH:MM:SS.854Z info vmware-vum-server[11371] [Originator@6876 sub=ScanUtil] [ScanUtil 667] Calling host esxi.mgmt.com (host-xxxx) scan API
YYYY-MM-DD HH:MM:SS.861Z error vmware-vum-server[11371] [Originator@6876 sub=ScanUtil] [ScanUtil 121] Failed to scan host - (host = host-xxxx, host name = esxi.mgmt.com) - (error =  com.vmware.vapi.std.errors.invalid_argument) - (details = Error:
-->    com.vmware.vapi.std.errors.invalid_argument
--> Messages:
-->    vapi.invoke.invalid.input<Invalid input for method: ::Com::Vmware::Esx::SettingsDaemon::SoftwareSvc::Scan>
-->    vapi.data.compound.field.extra<Extra field in compound: [::Com::Vmware::Esx::SettingsDaemon::SoftwareSpec] removed_components: >
--> )
YYYY-MM-DD HH:MM:SS.861Z info vmware-vum-server[11371] [Originator@6876 sub=InventoryTree] [InventoryTree 404] [ReleaseWriteLock] Released write lock on node host-xxxx. Post unlock, notified all the waiting readers and writers.
YYYY-MM-DD HH:MM:SS.861Z error vmware-vum-server[11371] [Originator@6876 sub=com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask] [Task, 524] Task:com.vmware.vcIntegrity.lifecycle.DesiredScanClusterTask ID:52b7b722-xxxx-9c79-xxxx-xxxxxxxxxx. Calling host-xxxx host Image Scan request failed. Non Retriable error :  Error:
-->    com.vmware.vapi.std.errors.error
--> Messages:
-->    com.vmware.vcIntegrity.lifecycle.HostScan.TaskApiError<A failure occurred when starting a host compliance check operation on host 'esxi.mgmt.com' : Error:
-->    com.vmware.vapi.std.errors.invalid_argument
--> Messages:
-->    vapi.invoke.invalid.input<Invalid input for method: ::Com::Vmware::Esx::SettingsDaemon::SoftwareSvc::Scan>
-->    vapi.data.compound.field.extra<Extra field in compound: [::Com::Vmware::Esx::SettingsDaemon::SoftwareSpec] removed_components: >

Environment

VMware vSphere ESXi 8.0

Cause

• Using Removed components support in desired image is available on vCenter/ESX from 80U3 version onwards.
• This is an expected behavior since hosts running version 8.0 U2 or lower cannot interpret a desired spec with removed components, as support for removed components is unavailable on these versions.
• This results in scan API to fail with com.vmware.vapi.std.errors.invalid_argument.

Resolution

A workaround to this error would be to perform a double remediation

1. Delete Removed components from the Desired Image.
2. Remediate Cluster with Desired Image (without any Removed Components configured).
3. Once hosts are remediated to 8.0 U3, configure VMware-VM-Tools as removed components in Desired Image. After hosts are upgraded to 8.0U3, they can understand the new software spec with removed_components as a field
4. Perform a second remediation (Apply).

Additional Information

Steps to delete component from desired image 

1. In the vSphere Client, navigate to a cluster or host that you manage with a vSphere Lifecycle Manager image.
2. On the Updates tab, select Hosts Image.
3. In the Image card, click the Edit button.
4. In the Edit Image card, modify the image set-up.
5. To delete a component from the image, click the trash icon in the table with components.