vSphere Lifecycle Manager can show non-compliant status after rollup Remediation, new cluster creation or transition from Baseline-managed cluster to Image-managed cluster for few upgrade paths due to i40en/i40enu driver name change.
search cancel

vSphere Lifecycle Manager can show non-compliant status after rollup Remediation, new cluster creation or transition from Baseline-managed cluster to Image-managed cluster for few upgrade paths due to i40en/i40enu driver name change.

book

Article ID: 318834

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • This is an awareness article where upgrading from 7.0U2a to 7.0U3 using rollup patch baseline will fail
    Environments 
  • Environments upgraded host to 7.0 U3 from 7.0U2 (using rollup patch baseline), will not be able to import image (HostSeeding) from that host to desired image based cluster.
  • During Baseline-Image managed cluster (where hosts are upgraded from 70u2 to 70u3 using rollup baseline upgrade) to Desired-Image managed cluster transition, there is one additional remediation to be performed



Symptoms:

  • Rollup upgrade of ESXi hosts with patch baselines from ESXi 7.0 Update 2a to ESXi 7.0 Update 3 fails
  • "VMware vSphere Update Manager had an unknown error"
  • On vCenter UI the error message reported :- "host returned esxupdate code -1"

    /var/run/log/esxupdate.log reported this exception:
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2102510
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR: An esxupdate error exception was caught:
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR: Traceback (most recent call last):
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 216, in main
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR:     cmd.Run()
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR:   File /lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py, line 153, in Run
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR:   File /tmp/esx-update-2102510/lib64/python3.8/site-packages/vmware/esximage/Transaction.py, line 962, in InstallVibsFromSources
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR:   File /tmp/esx-update-2102510/lib64/python3.8/site-packages/vmware/esximage/Transaction.py, line 928, in checkBaseImageUpgrade
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: ERROR: esximage.Errors.InstallationError: This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40en_1.11.1.31-1vmw.703.0.0.49537716, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade.
    YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2102510: esxupdate: DEBUG: <<<
  • Import Image from an upgraded host (where upgrade was done from 70u2, using 70u3 rollup patch baseline) may fail with the following error
  • "An error occurred while extracting image info on the host" ESXi  /var/log/lifecycle.log will show below message 
YYYY-MM-DDTHH:MM:SS.Z lifecycle: 2101447: HostSeeding:767 ERROR Software info extract errors: The following Components have been removed or downgraded on the host: Intel-i40en                                      
YYYY-MM-DDTHH:MM:SS.Z lifecycle: 2101447: imagemanagerctl:554 ERROR Get Software Info Failed: The following Components have been removed or downgraded on the host: Intel-i40en
  • Baseline-Image managed cluster to Desired-Image managed cluster transition would result in a non-compliant state.
  • ISO-based upgrade to ESXi 7.0 Update 3 fails
/var/log/vua-*.log reported this exception:
ERROR:root:Failed to perform image profile update: Expected 1 component, found 2
...
ValueError: Expected 1 component, found 2
  • On vCenter UI error is reported: "Cannot run upgrade script on host". 
  • One of /var/log/vua-*.log on host shows error: 
--> ERROR:root:Failed to perform image profile update: Expected 1 component, found 2 
--> Traceback (most recent call last): 
--> File "/tmp/vuaScript-QtZ4DU/prep.py", line 145, in <module> 
--> sys.exit(main()) 
--> File "/tmp/vuaScript-QtZ4DU/prep.py", line 135, in main 
--> res = t.InstallVibsFromDeployDir(pathToISO) 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1703, in InstallVibsFromDeployDir 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1207, in _installVibs 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo 
--> File "/tmp/vuaScript-QtZ4DU/esximage.zip/vmware/esximage/Bulletin.py", line 1276, in GetComponent 
--> ValueError: Expected 1 component, found 2 
  • On 7.0 Update 2c patched via patch baseline from vendor Image 7.0 Update 2a, HA cannot be enabled/configured 
    • On vCenter UI error is reported: "Sphere HA agent for this host has an error: The vSphere HA agent is not reachable from the vCenter Server" 
    • /var/run/log/esxupdate.log of impacted host shows: 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: An unexpected exception was caught: 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: Traceback (most recent call last): 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: cmd.Run() 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 965, in InstallVibsFromSources 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: inst, removed, exitstate = self._installVibs(curprofile, 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1207, in _installVibs 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: hasConfigDowngrade = checkFdmConfigDowngrade(curProfile, newProfile) 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: compDowngrades = curProfile.GetCompsDowngradeInfo(newProfile) 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: curComp = self.components.GetComponent(name) 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1276, in GetComponent 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: raise ValueError('Expected 1 component, found %u' 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2110025: esxupdate: ERROR: ValueError: Expected 1 component, found 2 
  • Remediate patch baseline to 7.0 Update 2c/2d or subsequent Update 2 patch failed 
    • On vCenter UI the error message reported is: "host returned esxupdate code -1" 
    • /var/run/log/esxupdate.log of impacted host reported this exception: 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: An esxupdate error exception was caught: 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: Traceback (most recent call last): 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 238, in main 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: cmd.Run() 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: File "/tmp/esx-update-2106354/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 957, in InstallVibsFromSources 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: File "/tmp/esx-update-2106354/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 923, in checkBaseImageUpgrade 
YYYY-MM-DDTHH:MM:SS.Z esxupdate: 2106354: esxupdate: ERROR: esximage.Errors.InstallationError: This upgrade transaction would skip ESXi Base Image VIB(s) VMW_bootbank_i40enu_1.8.1.137-1vmw.702.0.20.18426014, which could cause failures post upgrade. Please use an alternative upgrade method that would install the above VIB(s). For example, use an image profile instead of VIBs to perform the upgrade



Environment

VMware vSphere 7.0.x

Cause

  • Starting with vSphere 7.0 Update 3, the inbox i40enu network driver for ESXi changes name back to i40en.
  • The i40en driver was renamed to i40enu in vSphere 7.0 Update 2
  • The name change impacted rollup upgrade path, where on impacted workflow i40enu would not be replaced by i40en, which causes errors for import image (host seeding) and Baseline Image managed cluster to Desired Image managed cluster transition workflows

Resolution

This issue is resolved in ESXi 7.0 Update 3c.
The issue was also resolved in ESXi 7.0 U3b (No longer available).

Workaround:
Please follow the steps mentioned in any of below methods to resolve the issue :

Pre-Check:

  • Prior to the upgrade, check if i40enu vib is present on the host

# esxcli software vib list | grep i40enu 

  • If i40enu vib is present, check if there are two Intel-i40en components exist on the host  

# esxcli software component get | grep Intel-i40en -A15


Workaround - 1:

  • Run both the steps from pre-check section above,
  • If two Intel-i40en components are found on ESXi host, one with i40enu vib, use the following workaround steps on the impacted hosts
    • Remove i40enu driver vib with the below command 
      • #esxcli software vib remove -n i40enu 
    • Reboot the host executing below command 
      • #reboot
    • Above workaround will help in resolving: Enabling HA issue explained in the KB :- After update to ESXi 7.0 Update 3 vSphere HA fails to enable
    • Now you can attempt enabling HA and it should succeed
    • Upgrade issue to 7.0 Update 3
      • Now you can proceed with upgrade to 7.0 Update 3 and it should succeed
    • Reboot the host once the upgrade to 7.0 U3 is completed with the below command  
      • #reboot

 Workaround - 2:

  • Run both the steps from the pre-check section above,
  • If only one Intel-i40en component found with i40enu vib and ESXi host is currently running ESXi 7.0 Update 2a build, use one of the following alternative ways to upgrade to U3, such as
    • ISO upgrade baseline,  
    • esxcli profile, or  
    • Migrate to image-managed cluster and perform upgrade
  • Alternately 
    • Use patch baseline to apply 7.0 Update 2c first, before upgrading to 7.0 Update 3

 Workaround - 3:

  • Run both the steps from pre-check section above,
  • If you are running into upgrade failures during upgrade attempt to 7.0 Update 2c/ 7.0 Update 2d, it is recommended to upgrade to 7.0 Update 3 directly using ISO upgrade baseline
    • If you are attempting to upgrade using patching baseline/rollup upgrade, you need to apply workaround-1 or workaround-2 based on the i40enu vib presence and number of Intel-i40en components found with pre-check steps

If are running into upgrade issues due to intel-nvme-vmd/iavmd drivers, please refer KB After successfully remediating an ESXi 7.0 against the "Non-critical Host Patches (Predefined)" baseline in VLCM, it is still reported as non-compliant due to missing intel-nvme-vmd VIBs for the relevant resolution/workaround 

IMPORTANT NOTE:
Post upgrade to 7.0 Update 3, do not apply 7.0U2x based baselines or the critical host patches baseline until fix for this issue is made available in future patches. Otherwise, you may hit a similar issue again

Powered by Wolken Software