Upgrading vCenter Server 7.0 fails during precheck with "Host(s) were found in the vCenter Inventory, that are potentially problematic for a vCenter upgrade"

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Upgrading to vCenter Server 7.0U3c+ now requires an additional precheck. This precheck ensures that the ESXi hosts managed by the vCenter Server are at the correct ESXi versions and will not have any issues related to driver conflicts. There are two situations that can cause the vCenter precheck to fail:

• ESXi hosts running 7.0 U3/U3a exist anywhere in the VC inventory

• ESXi hosts running 7.0 U2c/U2d exists in a vSphere Lifecycle Manager (vLCM) image-enabled cluster

• ESXi hosts running 7.0 U2c/U2d outside of a vLCM image-enabled cluster are not impacted.

If the precheck detects versions of ESXi that could potentially have dual i40en driver conflicts, a more detailed check of these hosts is automatically triggered to detect if the dual-driver condition truly exists on the hosts flagged by the VC precheck.

Note that the precheck will fail and an error similar to the below will be reported in the VMware vCenter VAMI.
(Please see KB 87319 for details on the errors reported by the CLI upgrade option)


This error will be reported based on the presence of ESXi hosts running the versions detailed above and the results from the more detailed check will need to be examined to determine if any further action is required. It is possible that the more granular scan will not return any problematic host(s) and the upgrade can be restarted without any remediation efforts.

 

Reading the Results

Using an SSH client such as Putty, log in to the vCenter Server CLI. SSH may need to be enabled first.

Open the dual_driver_check_<timestamp>.log file reported in the error message to determine when the script has finished running. It can take up to 5 seconds per ESXi host to complete. The image below shows what to expect when the script has finished.

The contents of the "dual_driver_check_faulty_hosts_<timestamp>.txt", also contained in the "/var/log/vmware/applmgmt/" directory, will list any confirmed hosts that are impacted by the dual driver issues and will need manual remediation.

The image below is an example of the output generated in the "dual_driver_check_faulty_hosts_<timestamp>.txt". In this case, there are 3 hosts confirmed as having both the i40en and i40enu driver installed. These hosts need remediation using one of the options listed in the Solution section below.

However, as detailed above, it is possible that the results of the more detailed scan will not list any hosts. In this case, please restart the upgrade and the precheck will pass on the 2nd attempt.
 

Output file descriptions

• dual_driver_check_faulty_hosts_<timestamp>.txt: Contains the hosts that were found to have the i40en driver conflict. These need to be addressed before restarting the upgrade.
• dual_driver_check_skipped_hosts_<timestamp>.txt: Contains any hosts that were skipped because they were disconnected or not responding at the time. These are ignored.
• dual_driver_check_errored_hosts_<timestamp>.txt: Contains any hosts that the script was unable to process for any other reason than being disconnected. Use the dual_driver_check_<timestamp>.log to further investigate why these hosts were unable to be scanned. These need to be addressed before restarting the upgrade.
• dual_driver_check_<timestamp>.log: Primary logging for the script process.

Environment

VMware vCenter Server 7.0.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 7.0.3
VMware vCenter Server 7.0.3

Resolution

 Option 1 (Recommended) 

Upgrade the hosts listed in dual_driver_check_faulty_hosts_<timestamp>.txt to 7.0 U3c+ with an ISO upgrade baseline (refer to Related information section of the Knowledge Base article - Post vSphere 7.0 Update 3c Notes for ESXi releases with no ISO) or vLCM image before upgrading the vCenter Server. Although this is not the normally recommended order, it's necessary for this circumstance and fully supported by VMware. The hosts must be upgraded using an ISO image instead of a rollup patch baseline in vSphere Lifecycle Manager.Also note that ISO import depend on respective VC version.

  Upgrading ESXi using an ISO Upgrade Baseline

1. Download the ISO image via the VMware Downloads Portal in Customer Connect.
2. Upload the ISO into the Lifecycle Manager Depot.
   • See Import an ISO Image to the vSphere Lifecycle Manager Depot
3. Create a baseline that uses the imported ISO image.
   • See Create a Host Upgrade Baseline
4. Upgrade the host using the ISO baseline.
5. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
6. Once all affected hosts are corrected, restart the upgrade process.

   Upgrading ESXi as part of a vLCM image-enabled cluster

1. Please review the documentation for Using vSphere Lifecycle Manager Images to find instructions on upgrading the hosts via a vLCM desired-state image.
2. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
3. Once all affected hosts are corrected, restart the upgrade process.

 

Option 2

Alternatively, you can manually correct the i40en driver conflict on each host before restarting the upgrade. This isn’t the recommended option because the precheck will fail one more time before being able to upgrade. To understand why, see the precheck flowchart in Related Information.

1. Put the ESXi host in maintenance mode, then run the following command from an SSH session to remove the obsolete i40enu VIB :

esxcli software vib remove --vibname=i40enu

2. Once the VIB has been removed, reboot the ESXi host.
3. Repeat this for every host listed in the /var/log/vmware/applmgmt/dual_driver_check_faulty_hosts_<timestamp>.txt file generated by the script.
4. Once all affected hosts are corrected, restart the upgrade process

 

 

Important Upgrade Notes

  1. Once the vCenter precheck is clean, it is recommended to proceed to upgrade immediately. Prior to the upgrade, do not introduce additional hosts with potential driver conflicts or transition any clusters to be vLCM image-managed.

  2. Once vCenter has been upgraded, it's important NOT to use the non-critical patch baseline to upgrade any hosts that are 7.0 GA - 7.0 U3a, as it has the potential of re-introducing the issue or fail during remediation with an esxupdate error. Ensure that an image-based upgrade baseline or the predefined critical host patch baseline is used to upgrade them. A banner has been added on the Updates tab as a reminder of this.

Additional Information

Precheck FlowChart


Background
This issue originated when the Intel driver was renamed to i40enu in ESXi 7.0u2 and back to i40en in 7.0u3. During those renames, the relationship between the drivers was not created correctly, which led to both drivers being present on the same system. These conflicting drivers would cause solutions that installed agent vibs to fail, such as HA, WCP, and NSX on the hosts. The fix for this problem is included in ESXi 7.0 U3c. However, because these agent vibs can be pushed to the ESXi hosts as soon as vCenter gets upgraded, this issue would still manifest during the normal upgrade order (vCenter -> ESXi), as the ESXi hosts may still be running affected versions.
 
The vCenter precheck was created to notify users whether potentially affected ESXi versions existed in the vCenter inventory pre-upgrade. The automated script was added to help users narrow down the scope of truly affected hosts, so they can remediate the issue by either upgrading them or manually removing the duplicate i40enu driver.


For other information related to this issue see:

• Upgrading to vCenter Server 7.0 U3c using the CLI fails during precheck with "Installation failed. Retry to resume from the current state. Or please collect the VC support bundle"'
• Using the dual_driver_check.py script
• Converting an ESXi cluster to be vLCM image managed fails in vCenter Server 7.0u3c with "The following host(s) have an ESXi version higher than ESXi 7.0 U2a and lower than ESXi 7.0 U3c"
• Enabling vSphere HA might fail or never complete on hosts with ESXi 7.0u2c/u2d and 7.0u3/u3a

 
Post vSphere 7.0 Update 3c Notes –

The scenario mentioned in the Details section of this KB, is applicable to future vSphere 7.0 patch releases post 7.0 Update 3c.

If the source system is of version  between ESXi 7.0 Update 2 and Update 3c, before upgrading to future vSphere patch release, ensure VMware vCenter server upgraded to 7.0 Update 3c or later to avoid potential vSphere HA failure due to dual i40en driver conflicts condition.

The above solution section is applicable for all 7.x future patch releases regardless of patch being ESXi-only or VC-only or both, but with additional instructions to Solution 1 as applicable:

In case of *no* ISO deliverable in the target ESXi patch post 7.0 Update 3c, then follow either of the options below:

1. Create custom/stock ESXi ISO using the target ESXi patch ZIP bundle referring to the steps mentioned in the Create a Custom ESXi ISO Image with PowerCLI Cmdlets documentation.
2. In vSphere Lifecycle Manager, use Baseline Group upgrade method to create and combine the baselines as described below:
   1. Create baseline say “baseline-A” of ESXi ISO (custom/stock) from the last release (example: ESXi 7.0 Update 3d) 
   2. Create baseline say “baseline-B” of target ESXi patch using rollup 
   3. Create baseline Group say “ baselineGroup-C” to combine baselines – “baseline-A” and “baseline-B” 
   4. Remediate the baseline group “ baselineGroup-C” against the intended ESXi hosts or Cluster.

For the steps to use Baseline Group upgrade method refer to Creating and Working with Baselines and Baseline Groups documentation.