SDDC NSX upgrade fails in with error regarding incompatible image

Products

• VMware SDDC Manager

Issue/Introduction

Error in step 2/4 upgrading NSX-T components in SDDC:

Host group upgrade status is FAILED for group ########-####-####-####-############:domain-c# 
<ESXi_Hostname> : [Upgrade failed: Host '<ESXi_Hostname>' is incompatible with the overall desired image. Please fix desired image issues, Upgrade failed: Solution apply failed on host: '<ESXi_Hostname>']. 

Environment

5.x

Cause

Issue can be caused by inconsistent image status between the ESXi host and the last applied baseline image in VCLM inside of vCenter
Within the vCenter update manager logs shows an issue with the HPE Addons packages are causing an inconsistent image error:

In the below example we see an issue due to two particular components were updated higher than the last HPE Add On Package that is currently applied to the host under the VCLM image settings

/var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log

ERRORS:  "id": "com.vmware.vcIntegrity.lifecycle.HostScan.ComponentDowngrade",
"default_message": "Downgrades of manually added Components Marvell Technology Network/iSCSI/FCoE/RDMA E4 drivers(6.0.368.0-1OEM) in the desired Addon HPE-Custom-AddOn are not supported.",
"default_message": "Version 800.4.6.0.7 - Build 0 of the manually added Component Integrated Smart Update Tool is an unsupported downgrade from version 800.4.7.0.6 - Build 0 on the host.",

Resolution

The image has to be brought back into compliance manually to allow SDDC to proceed, this is accomplished by remediating the affected cluster manually within the vCenter using VCLM.

1. Access the vCenter directly and switch to the "Updates" tab for the affected cluster
2. Set the image baseline and add on package to the same in use currently, set the ESXi image to the same ESXi build currently installed
3. Run a validation and it will outline the existing issues
4. Remove any vibs currently causing an issue if they are a higher release than the current one in the set image baseline, if none are a higher version than the image then this step can be skipped
5. Remediate the cluster to bring it into compliance

Rerun the SDDC upgrade within SDDC after the above to complete the previously failed step